[jira] [Commented] (HADOOP-15183) S3Guard store becomes inconsistent after partial failure of rename

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803109#comment-16803109 ] 

Steve Loughran commented on HADOOP-15183:
-----------------------------------------

Back working on this; done in a new PR.

Revisiting the code, one thing I see we need to look at is the problem of a partial delete where a number of children have all been deleted. Are we confident that there are no longer any parent entries which are in that list of deleted files? That is, it is never the case that the list of files to rm is

/dir1/
/dir1/file1
/dir1/file2

And so in a partial delete failure (say of file2), is there a dir1 entry to purge? I'm not convinced this situation will arise: every path passed into the delete will be independent. 





> S3Guard store becomes inconsistent after partial failure of rename
> ------------------------------------------------------------------
>
>                 Key: HADOOP-15183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15183
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Blocker
>         Attachments: HADOOP-15183-001.patch, HADOOP-15183-002.patch, org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
>
>
> If an S3A rename() operation fails partway through, such as when the user doesn't have permissions to delete the source files after copying to the destination, then the s3guard view of the world ends up inconsistent. In particular the sequence
>  (assuming src/file* is a list of files file1...file10 and read only to caller)
>    
> # create file rename src/file1 dest/ ; expect AccessDeniedException in the delete, dest/file1 will exist
> # delete file dest/file1
> # rename src/file* dest/  ; expect failure
> # list dest; you will not see dest/file1
> You will not see file1 in the listing, presumably because it will have a tombstone marker and the update at the end of the rename() didn't take place: the old data is still there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org