[jira] [Commented] (HADOOP-13968) S3a FS to support ".temp_pending_put" path for pending put operations

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815840#comment-15815840 ] 

Steve Loughran commented on HADOOP-13968:
-----------------------------------------

+when creating a path {{.temp_pending_put/dest.txt}}, there must be no existing entry there. we will need to rely on a consistent FS for this, and may also want to do some PUT operations which bypass the s3a FS APIs, but still work with s3guard or an existing consistent object store. (e.g an atomic PUT/mkdir. 

> S3a FS to support ".temp_pending_put" path for pending put operations
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-13968
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13968
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Steve Loughran
>
> S3AFileSystem to add support for a special path, such as  {{.temp_pending_put/}} or similar, which, when used as the base of a path, indicates that the file is actually to be saved to the parent dir, but only via a delayed put commit operation.
> At the same time, we may need blocks on some normal fileIO ops under these dirs, especially rename and delete, as this would cause serious problems including data loss and large bills for pending data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org