[jira] [Commented] (HDFS-4849) Enable retries for create and append operations.

["Colin Patrick McCabe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706397#comment-13706397 ] 

Colin Patrick McCabe commented on HDFS-4849:
--------------------------------------------

Hi Konstantin,

I understand your frustration with the slow pace of progress on this JIRA.  Believe me.  But this is different than some of the other JIRAs we've done.  Usually, there is a configuration option that can turn off the new thing if there is a problem (as with HDFS-347) or people can continue using the old system (as with QuorumJournalManager).  People can't exactly turn off file creation-- it's a core part of the filesystem.  So we want to make sure that we get everything right the first time, even if that means some pain up front.

I read through the comments here, right from the start, as well as the patch.  It seems like the main issues raised were:
1. semantics-- some people are uncomfortable using the annotation "Idempotent"
2. problems with multithreaded behavior

Point #1 seems like something we can resolve without too much difficulty.  We can change the '@Idempotent' annotation to '@SkipRetryCache' or something like that.  I see that you've already renamed the JIRA to "something like that" :)

Point #2 was originally brought up by [~tlipcon].  He wrote:
bq. The idempotent create is also potentially problematic with multi-threaded clients. A given client may have multiple threads race to create the same file, and those threads would share the same client name (and hence lease).

If I understand correctly, Konstantin's response was that users could use {{FileSystem#newInstance}} to give each thread a separate DFSClient (and hence their own client names).  However, that doesn't help the people who are relying on the current behavior.

In HDFS-4942, Suresh proposes "adding a UUID as the request ID in the RPC client."  It seems like that might be helpful in resolving #2.  Does it make sense to collaborate on this?
                
> Enable retries for create and append operations.
> ------------------------------------------------
>
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>         Attachments: idempotentCreate-branch2.patch, idempotentCreate.patch, idempotentCreate.patch, idempotentCreate.patch, idempotentCreate.patch
>
>
> create, append and delete operations can be made retriable. This will reduce chances for a job or other app failures when NN fails over.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira