[jira] [Commented] (HADOOP-15086) NativeAzureFileSystem file rename is not atomic

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299897#comment-16299897 ] 

Steve Loughran commented on HADOOP-15086:
-----------------------------------------


Production code LGTM. Tests need a bit of tuning to make sure that if a test
fails, we get as much information preserved and propagated into the HTML test
reports as we can.


ITestNativeAzureFileSystemLive
L82. You can use {{ContractTextUtils.fail(text, throwable)}} to raise an AssertionError with the text and nested exception


L108. throw the unexpectedError if it is there
{code}
if (unexpectedError.get() != null) {
 throw unexpectedError.get();
}
{code}

This makes me think we really need more per-store tests of the commit protocols; something like {{TestFileOutputCommitter}} but for testing the how well that committer's operations work with a store. {{AbstractITCommitProtocol}}  is the reference example, but that's trying the new committers. Really we just want to make sure that all the various rename ops in the classic committer work with the store


> NativeAzureFileSystem file rename is not atomic
> -----------------------------------------------
>
>                 Key: HADOOP-15086
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15086
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 2.7.3
>            Reporter: Shixiong Zhu
>            Assignee: Thomas Marquardt
>         Attachments: HADOOP-15086-001.patch, RenameReproducer.java
>
>
> When multiple threads rename files to the same target path, more than 1 threads can succeed. It's because check and copy file in `rename` is not atomic.
> I would expect it's atomic just like HDFS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org