You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2021/03/11 00:39:00 UTC
[jira] [Created] (HDDS-4961) Disk space leak after deleting Ozone
files
Wei-Chiu Chuang created HDDS-4961:
-------------------------------------
Summary: Disk space leak after deleting Ozone files
Key: HDDS-4961
URL: https://issues.apache.org/jira/browse/HDDS-4961
Project: Apache Ozone
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Wei-Chiu Chuang
I'm finding cases where deleting Ozone files does not reclaim disk space, even if the corresponding containers are empty and in CLOSED state.
In some cases, blocks are removed from container, resulting in empty and closed container. However, the container metadata is not removed, and SCM still records the container. No way to remove them. In this case, each dangling container occupies ~140mb of space. Overtime, tens or even hundreds of GB is wasted on each DN.
Because of HDDS-2534, there is no way to manually delete the containers.
Steps to reproduce:
{code}
ozone sh volume create o3://ozone1/vol1
ozone sh bucket create o3://ozone1/vol1/bucket1
# Generated data.
export EXAMPLES_PATH=
yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce//hadoop-mapreduce-examples.jar teragen -Dmapreduce.job.maps=10 -DmDmapreduce.map.memory.mb=4096 -Dmapreduce.reduce.memory.mb=4096 1000000000 o3fs://bucket1.vol1.ozone1/teragen11
Delete skip trash:
hdfs dfs -rm -r -skipTrash "o3fs://bucket1.vol1.ozone1/*”
(run "ozone admin container close ${containerid} to force closing the containers)
# check disk usage:
du -h /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/
# the container directory is not removed
...
142M /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/34-dn-container.db
4.0K /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/db.checkpoints
142M /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata
4.0K /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/chunks
142M /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34
13G /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org