[jira] [Created] (HDDS-4964) Disk space leak after deleting Ozone files

["Wei-Chiu Chuang (Jira)" <ji...@apache.org>]

Wei-Chiu Chuang created HDDS-4964:
-------------------------------------

             Summary: Disk space leak after deleting Ozone files
                 Key: HDDS-4964
                 URL: https://issues.apache.org/jira/browse/HDDS-4964
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Wei-Chiu Chuang


Sometimes deleting Ozone files does not remove containers and block files. Even removing the bucket doesn't reclaim the space. I even forcefully closed all the containers.

Due to HDDS-2534, there is no way to delete the dangling containers manually.

Looking at OM's rocksdb instance, keyTable is empty, bucketTable is empty. prefixKeyTable 
 empty. But openKeyTable is not.

The application that generated the data (teragen) was killed. I don't think the clients are still connected and the openKeyTable is not pruned after a day.

Maybe we shouldn't prune openKeyTable when deleting the corresponding volume/bucket?

====
Container metadata not removed after deleting Ozone files

Steps to repro
{code}
 ozone sh volume create o3://ozone1/vol1
 ozone sh bucket create o3://ozone1/vol1/bucket1

# run teragen to generate data
yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen -Dmapreduce.job.maps=10 -DmDmapreduce.map.memory.mb=4096 -Dmapreduce.reduce.memory.mb=4096 1000000000 o3fs://bucket1.vol1.ozone1/teragen1

# delete, skip trash
hdfs dfs -rm -r -skipTrash o3fs://bucket1.vol1.ozone1/teragen1

# forcefully close containers

for i in {1..1000}; do sudo -u hdfs ozone admin container close ${i}; done
{code}

Files are deleted from namespace, but space not released.
All containers are CLOSED.

The metadata of each leftover container occupies ~140mb in space. Over time, a DN can accumulate tens or even hundreds of GB of wasted space.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org