[jira] [Comment Edited] (HDFS-12866) Recursive delete of a large directory or snapshot makes namenode unresponsive

["Yongjun Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271732#comment-16271732 ] 

Yongjun Zhang edited comment on HDFS-12866 at 11/29/17 11:04 PM:
-----------------------------------------------------------------

Thanks much for the feedback [~kihwal] and [~daryn].

Yes, with snapshot, an INode won't be disconnected when it's in previous snapshot, it will be repaced with INodeRefenrence instead. So we can use a bit in the INode to indicate whether it's "disconnected" instead of physical disconnection from the parent.

Indeed I was thinking traversing to the root to check, like done in {{FSNamesystem#isFileDeleted}}, it cost some time, but we can find if an INode is disconnected, right?

Optimizing the permission checking etc would help, however, without postponing the deletion work to later, if the tree is large enough, we can still hit this problem.  Quota  computation will still be done, just not done right away. Hopefully that won't cause too much problem?

So the main issue of this approach is the cost of traversing to the root to check if any ancestor is disconnected? I wonder how bad it is. In IBR and FBR, can we assume the file exists if the INode is there? The block deletion step (step 2) will get them removed later anyways.

Thanks again.







was (Author: yzhangal):
Thanks much for the feedback [~kihwal] and [~daryn].

Yes, with snapshot, an INode won't be disconnected when it's in previous snapshot, it will be repaced with INodeRefenrence instead. So we can use a bit in the INode to indicate whether it's "disconnected" instead of physical disconnection from the parent.

Indeed I was thinking traversing to the root to check, like done in {{FSNamesystem#isFileDeleted}}, it cost some time, but we can find if an INode is disconnected, right?

Optimizing the permission checking etc would help, however, without postponing the deletion work to later, if the tree is large enough, we can still hit this problem.  Quota  computation will still be done, just not done right away. Hopefully that won't cause too much problem?

So the main issue of this approach is the cost of traversing to the root to check if any ancestor is disconnected? I wonder how bad it is. In IBR and FBR, can we assume the file exists in the INode is there? The block deletion step (step 2) will get them removed later anyways.

Thanks again.






> Recursive delete of a large directory or snapshot makes namenode unresponsive
> -----------------------------------------------------------------------------
>
>                 Key: HDFS-12866
>                 URL: https://issues.apache.org/jira/browse/HDFS-12866
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>            Reporter: Yongjun Zhang
>
> Currently file/directory deletion happens in two steps (see {{FSNamesystem#delete(String src, boolean recursive, boolean logRetryCache)}}:
> # Do the following under fsn write lock and release the lock afterwards
> ** 1.1  recursively traverse the target, collect INodes and all blocks to be deleted
> ** 1.2  delete all INodes
> # Delete the blocks to be deleted incrementally, chunk by chunk. That is, in a loop, do:   
> ** acquire fsn write lock,
> ** delete chunk of blocks
> ** release fsn write lock
> Breaking the deletion to two steps is to not hold the fsn write lock for too long thus making NN not responsive. However, even with this, for deleting large directory, or deleting snapshot that has a lot of contents, step 1 itself would takes long time thus still hold the fsn write lock for too long and make NN not responsive.
> A possible solution would be to add one more sub step in step 1, and only hold fsn write lock in sub step 1.1:
> * 1.1. hold the fsn write lock, disconnect the target to be deleted from its parent dir, release the lock
> * 1.2 recursively traverse the target, collect INodes and all blocks to be deleted
> * 1.3  delete all INodes
> Then do step 2.
> This means, any operations on any file/dir need to check if its ancestor is deleted (ancestor is disconnected), similar to what's done in FSNamesystem#isFileDeleted method.
> I'm throwing the thought here for further discussion. Welcome comments and inputs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org