You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2018/02/22 16:44:00 UTC
[jira] [Comment Edited] (HADOOP-15183) S3Guard store becomes
inconsistent after partial failure of rename
[ https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373049#comment-16373049 ]
Steve Loughran edited comment on HADOOP-15183 at 2/22/18 4:43 PM:
------------------------------------------------------------------
Patch 002.
Trying to make the s3guard inconsistency visible in a unit test. Failing. Will need to either go for 1K+ test files, or allow the page size for delete requests to be configured (best).
For setting the page size, two options
# Add a secret fs.s3a.x-test.delete.page.size. Simple, may get used, may need to be maintained.
# make it a config option of the inconsistent FS client, so iff you switch to the inconsistent client do you get to turn it on.
#2 is a bit more convoluted, but I like it. Initially though I'll do the secret config option
was (Author: stevel@apache.org):
Patch 003.
Trying to make the s3guard inconsistency visible in a unit test. Failing. Will need to either go for 1K+ test files, or allow the page size for delete requests to be configured (best).
For setting the page size, two options
# Add a secret fs.s3a.x-test.delete.page.size. Simple, may get used, may need to be maintained.
# make it a config option of the inconsistent FS client, so iff you switch to the inconsistent client do you get to turn it on.
#2 is a bit more convoluted, but I like it. Initially though I'll do the secret config option
> S3Guard store becomes inconsistent after partial failure of rename
> ------------------------------------------------------------------
>
> Key: HADOOP-15183
> URL: https://issues.apache.org/jira/browse/HADOOP-15183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
> Attachments: HADOOP-15183-001.patch, HADOOP-15183-002.patch, org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
>
>
> If an S3A rename() operation fails partway through, such as when the user doesn't have permissions to delete the source files after copying to the destination, then the s3guard view of the world ends up inconsistent. In particular the sequence
> (assuming src/file* is a list of files file1...file10 and read only to caller)
>
> # create file rename src/file1 dest/ ; expect AccessDeniedException in the delete, dest/file1 will exist
> # delete file dest/file1
> # rename src/file* dest/ ; expect failure
> # list dest; you will not see dest/file1
> You will not see file1 in the listing, presumably because it will have a tombstone marker and the update at the end of the rename() didn't take place: the old data is still there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org