[jira] [Commented] (HADOOP-11525) FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.

["Chris Nauroth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301613#comment-14301613 ] 

Chris Nauroth commented on HADOOP-11525:
----------------------------------------

Thanks, Steve.  I had a hunch there was an existing jira.

I'm going to back off on my suggestion of a new {{FileSystem}} method for "transactional put".  Even if we did that, we'd still need the characteristics/semantics querying approach for other reasons, and I don't see a benefit to expanding the public API footprint in multiple different ways.

Eddy, I see that the HADOOP-9565 patch does include the same basic {{FsShell}} change that you want to achieve.  Are you comfortable resolving this as duplicate and participating over there?

> FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-11525
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11525
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: tools
>    Affects Versions: 2.6.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>         Attachments: HADOOP-11525.000.patch
>
>
> When running {{hadoop fs -put}},  {{FsShell}} creates a {{._COPYING_.}} file on the target directory, and then renames it to target file when the write is done. However, for some targeted systems, such as S3, Azure and Swift, a partial failure write request (i.e., {{PUT}}) has not side effect, while the {{rename}} operation is expensive. 
> {{FileSystem}} should expose some characteristics so that the operation such as {{CommandWithDestination#copyStreamToTarget()}} can detect and choose the right way to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)