[jira] [Commented] (HADOOP-12667) s3a: Support createNonRecursive API

["Chris Nauroth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068449#comment-15068449 ] 

Chris Nauroth commented on HADOOP-12667:
----------------------------------------

If the goal is to support HBase's WAL management, then another aspect is that HBase expects atomicity from this call.  Atomicity would guarantee that for 2 concurrent threads/processes calling {{createNonRecursive}}, one of the callers succeeds and the other fails.  HBASE-11045 has some great discussion of the expected semantics, especially this comment:

https://issues.apache.org/jira/browse/HBASE-11045?focusedCommentId=13977198&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13977198

This means that an implementation that just checks for existence before allowing the create to proceed is insufficient.  There would be a race condition allowing 2 concurrent callers to think the path exists and proceed with their creates.

In hadoop-azure, this problem is solved by making concurrent callers acquire a mutually exclusive lease on the blob before doing the create.  Blob leases are a feature of Azure Storage.  I'm not aware of any lease functionality like this available in S3.

> s3a: Support createNonRecursive API
> -----------------------------------
>
>                 Key: HADOOP-12667
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12667
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>            Reporter: Sean Mackrory
>            Assignee: Sean Mackrory
>         Attachments: HADOOP-12667.001.patch
>
>
> HBase and other clients rely on the createNonRecursive API, which was recently un-deprecated. S3A currently does not support it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)