[jira] [Commented] (HADOOP-13308) S3A delete may fail to preserve parent directory.

["Chris Nauroth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343787#comment-15343787 ] 

Chris Nauroth commented on HADOOP-13308:
----------------------------------------

I see a potential weakness in the S3A logic that preserves empty directories after file deletions.  If we consider an S3A instance containing a single file, /dir1/file1, and a user deletes /dir1/file1, then the sequence of the current logic is:

# DELETE(/dir1/file1)
# if (!exists(/dir1)) then PUT(/dir1)

We lack atomicity across those 2 steps, so if there is a process death between those 2 steps, then /dir1 will mysteriously vanish.  Rename has a similar problem too.

I'd like to propose that those 2 steps ought to be reversed, so that the fake directory gets created first before the delete.  Then, /dir1 can't vanish.  It would cause a different side effect: if the process dies in between, then we'd have a fake directory where we don't really need one.  I don't think this impacts correctness, because listStatus always does a listing, even if the requested directory is a fake.  There would probably be a very negligible hit on the bill paid to Amazon for that extra object.  This would require a listing call on the parent to check existence, but that listing call already happens today via createFakeDirectoryIfNecessary -> exists -> getFileStatus.  We're just reordering that call, so I don't think this is going to be a performance regression.

What are your thoughts?  If it makes sense, this is something I'd file in Apache under the phase 3 (not urgent).

--Chris Nauroth

> S3A delete may fail to preserve parent directory.
> -------------------------------------------------
>
>                 Key: HADOOP-13308
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13308
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>
> When a file or directory is deleted in S3A, and the result of that deletion makes the parent empty, S3A must store a fake directory (a pure metadata object) at the parent to indicate that the directory still exists.  The logic for restoring fake directories is not resilient to a process death.  This may cause a directory to vanish unexpectedly after a deletion of its last child.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org