[jira] [Commented] (HDDS-6449) Failed container delete can leave artifacts on disk

["Neil Joshi (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541764#comment-17541764 ] 

Neil Joshi commented on HDDS-6449:
----------------------------------

In the keyValueHandler.handleDeleteContainer datanode delete path, the 4 items are deleted including DB, metadata and chunks.  During the delete process an IOException can be raised by the deletion of the metadata, chunks or container directory. 

Should an exception be raised, it looks to result to the condition this Jira was opened to handle, unaccounted artifacts left on the disk.  A way of handling this is that on handling the exception in the delete path, to enqueue the failed container delete transaction to be handled by a dedicated thread service for removing artifacts on datanodes (failed container deletions).

> Failed container delete can leave artifacts on disk
> ---------------------------------------------------
>
>                 Key: HDDS-6449
>                 URL: https://issues.apache.org/jira/browse/HDDS-6449
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: Ozone Datanode
>    Affects Versions: 1.0.0, 1.1.0, 1.2.0
>            Reporter: Ethan Rose
>            Priority: Major
>
> When SCM issues a delete command to a datanode, the datanode does the following steps:
> writeLock()
>     1. The container is removed from the in memory container set.
> writeUnlock()
>     2. The container metadata directory recursively deleted.
>     3. The container chunks directory recursively deleted.
>     4. The datanode sets the container's in memory state to DELETED
>         - This is purely for the ICR as the container is not present in the container set anymore.
>     5. Datanode sends incremental container report to SCM with the new state.
>         - The container has been removed from the in-memory set at this point, so once the ICR is sent the container is unreachable.
> In HDDS-6441, A failure in step 2 removed the .container file and db.checkpoints directory (unused) from the metadata directory, and the rest of the steps were not done after the IO exception was thrown during the delete. This caused an error to be logged when the partial state was read on datanode restart.
> This current method of deleting containers provides no way to recover from or retry a failed delete, because the container is removed from the in-memory set as the first step. This Jira aims to change the datanode delete steps so that if a delete fails, the existing SCM container delete retry logic or the datanode itself can eventually get the lingering state off the disk.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org