You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Franklinsam Paul (Jira)" <ji...@apache.org> on 2022/09/01 16:28:00 UTC
[jira] [Commented] (HDDS-7196) Disk space used by failed job(teragen here) is not reclaimable
[ https://issues.apache.org/jira/browse/HDDS-7196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599059#comment-17599059 ]
Franklinsam Paul commented on HDDS-7196:
----------------------------------------
[~erose] thanks for the update and help, is there a way to clean up the stale data in my current ozone version 1.0.0, other than manually cleanup the blocks in each DN. ( even in manually cleaning how to find the blocks related to that issue bucket/key.- is there any option similar to "hdfs fsck -blockId"
> Disk space used by failed job(teragen here) is not reclaimable
> --------------------------------------------------------------
>
> Key: HDDS-7196
> URL: https://issues.apache.org/jira/browse/HDDS-7196
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Datanode
> Environment: |Apache Ozone|1.0.0|
> Reporter: Franklinsam Paul
> Priority: Major
> Attachments: Ozone usage_ after_failing_cleanup.png, Ozone usage_ fresh_install.png
>
>
> On Fresh ozone cluster, ran a tergane job and killed it around 25% completion. this left ozone used about 74.4GB but none of the files written is listing.
> Issue can be reproducible with below steps. ( snapshots from the recon UI will be attached for usage reference)
> {code:java}
> ozone sh volume create o3://ozonefrankserviceid/testvol/
> ozone sh bucket create o3://ozonefrankserviceid/testvol/testbucketyarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen -Dmapreduce.job.maps=2 1000000000 ofs://ozonefrankserviceid/testvol/testbucket
> ozone sh volume create o3://ozonefrankserviceid/testvol/
> ozone sh bucket create o3://ozonefrankserviceid/testvol/testbucket
>
> yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen -Dmapreduce.job.maps=2 1000000000 ofs://ozonefrankserviceid/testvol/testbucket/teragentest1
>
> ozone fs -ls ofs://ozonefrankserviceid/testvol/testbucket/teragentest1
> ozone fs -ls ofs://ozonefrankserviceid/testvol/testbucket/teragentest1/_temporary
> ozone fs -ls ofs://ozonefrankserviceid/testvol/testbucket/teragentest1/_temporary/1
> ozone fs -ls ofs://ozonefrankserviceid/testvol/testbucket/teragentest1/_temporary/1/_temporary
> ozone fs -du -s -h ofs://ozonefrankserviceid/testvol/testbucket/teragentest1/_temporary/1/_temporary
> ozone fs -ls ofs://ozonefrankserviceid/testvol/testbucket/teragentest1/_temporary/1/_temporary/attempt_1661777485132_0001_m_000000_2 --> no files/bject
> ozone fs -ls ofs://ozonefrankserviceid/testvol/testbucket/teragentest1/_temporary/1/_temporary/attempt_1661777485132_0001_m_000001_2 --> no files/object
>
> Ozone usage is increased in the recon UI as 75GB
>
> hdfs dfs -rm -r -skipTrash ofs://ozonefrankserviceid/testvol/testbucket/teragentest1
> ozone sh bucket delete o3://ozonefrankserviceid/testvol/testbucket
>
> [root@DNHOST1 ozone-conf]# grep -A1 'hdds.datanode.dir' ozone-site.xml
> <name>hdds.datanode.dir</name>
> <value>/var/lib/hadoop-ozone/datanode/data</value>
> [root@DNHOST1 ozone-conf]#[root@DNHOST1 containerDir0]# du -sh /var/lib/hadoop-ozone/datanode/data/hdds/a9461a7f-ef81-4942-a278-15ff7602df14/current/containerDir0/
> 26G /var/lib/hadoop-ozone/datanode/data/hdds/a9461a7f-ef81-4942-a278-15ff7602df14/current/containerDir0/
> [root@DNHOST1 containerDir0]#
> [root@DNHOST1 chunks]# ozone sh volume list o3://ozonefrankserviceid/ -a |egrep 'name|usedNamespace'
> "name" : "s3v",
> "usedNamespace" : 0,
> "name" : "om",
> "name" : "testvol",
> "usedNamespace" : 0,
> "name" : "hive/HMSHOST.example.com@SUPPORT.COM",
> "name" : "hive",
> [root@DNHOST1 chunks]# {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org