You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ethan Rose (Jira)" <ji...@apache.org> on 2022/02/28 22:42:00 UTC
[jira] [Commented] (HDDS-6301) containerDir left over after volume/bucket/key deletion
[ https://issues.apache.org/jira/browse/HDDS-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499209#comment-17499209 ]
Ethan Rose commented on HDDS-6301:
----------------------------------
This turned out to be a byproduct of the existing missing chunks dir bug HDDS-6235. The block deleting service on the datanode is running and deleting blocks, but it is operating very slowly. Each run it picks containers to delete from until the total pending delete block counts meet its threshold, then it tries to delete them. Some of these empty chunks directory containers have pending delete blocks (but no actual block files), so they are being included in the batch. Since there's no chunks dir, deletion fails and the pending delete block count never gets updated, leading to these same containers to be picked on the next run and sucking up the block threshold. Here's an extract from the datanode log that showed this happening:
{code:java}
Plan to choose 5000 blocks for block deletion, actually deleting 3141 blocks.
...
Container: 266899, deleted blocks: 7, task elapsed time: 8ms
{code}
As a result block deletion on this datanode is effectively operating at 7 deletions per run on this cluster, when ideally it would be hitting 5000 per run.
> containerDir left over after volume/bucket/key deletion
> -------------------------------------------------------
>
> Key: HDDS-6301
> URL: https://issues.apache.org/jira/browse/HDDS-6301
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager, SCM
> Reporter: George Huang
> Assignee: Ethan Rose
> Priority: Major
>
> Ozone volume/bucket/key were deleted via command line using command like:
> ozone fs -rm -skipTrash -r ofs://ozone1/vol-1-49431
> Now check data nodes, many containerDir### were left over on the data node:
> [root@ve1135 current]# pwd
> /data/1/hadoop-ozone/datanode/data8/hdds/CID-f72ad861-4e70-4fe5-982b-79ce831c6d31/current
> [root@ve1135 current]# ls -al ./containerDir1/262719/chunks/107544261459853384.block
> -rw-r--r-- 1 hdfs hdfs 10240 Jan 31 22:23 ./containerDir1/262719/chunks/107544261459853384.block
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org