[jira] [Created] (HDDS-4960) Disk space leak after deleting Ozone files

["Wei-Chiu Chuang (Jira)" <ji...@apache.org>]

Wei-Chiu Chuang created HDDS-4960:
-------------------------------------

             Summary: Disk space leak after deleting Ozone files
                 Key: HDDS-4960
                 URL: https://issues.apache.org/jira/browse/HDDS-4960
             Project: Apache Ozone
          Issue Type: Bug
    Affects Versions: 1.0.0
            Reporter: Wei-Chiu Chuang


I'm finding cases where deleting Ozone files does not reclaim disk space, even if the corresponding containers are empty and in CLOSED state.

In some cases, blocks are removed from container, resulting in empty and closed container. However, the container metadata is not removed, and SCM still records the container. No way to remove them. In this case, each dangling container occupies ~140mb of space. Overtime, tens or even hundreds of GB is wasted on each DN.

Because of HDDS-2534, there is no way to manually delete the containers.

Steps to reproduce:

{code}
 ozone sh volume create o3://ozone1/vol1
 ozone sh bucket create o3://ozone1/vol1/bucket1

# Generated data.
export EXAMPLES_PATH=
yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce//hadoop-mapreduce-examples.jar teragen -Dmapreduce.job.maps=10 -DmDmapreduce.map.memory.mb=4096 -Dmapreduce.reduce.memory.mb=4096 1000000000 o3fs://bucket1.vol1.ozone1/teragen11

Delete skip trash:

hdfs dfs -rm -r -skipTrash "o3fs://bucket1.vol1.ozone1/*”

(run "ozone admin container close ${containerid} to force closing the containers)

# check disk usage:
du -h /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/
# the container directory is not removed
...
142M    /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/34-dn-container.db
4.0K    /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/db.checkpoints
142M    /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata
4.0K    /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/chunks
142M    /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34
13G     /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org