[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

["Aaron Tokhy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198324#comment-15198324 ] 

Aaron Tokhy commented on HADOOP-9565:
-------------------------------------

FileOutputCommitter recently introduced a notion of 'algorithm version' (through mapreduce.fileoutputcommitter.algorithm.version).  Perhaps the change to FileOutputCommitter would involve adding another algorithm version (3) for performing a direct commit, separate from this change?  FileOutputCommitter could ideally be split into 3 subclasses today, 1 per algorithm version.

Another boolean configuration could be added to mapred-default.xml, called mapreduce.fileoutputcommitter.infer.algorithm.version, which would select the appropriate FileOutputCommitter algorithm based on the FileSystem for the Path in mapreduce.output.fileoutputformat.outputdir.  MRAppMaster could then select the appropriate algorithm depending on the semantics available in the target filesystem.

It might also make some sense to break up FileOutputCommitter algorithms into separate classes to accommodate for these changes, since the algorithm used may not work well on some FileSystem implementations (especially due to the attempts of performing atomic renames, on FileSystem implementations where directory renames are not atomic).

> Add a Blobstore interface to add to blobstore FileSystems
> ---------------------------------------------------------
>
>                 Key: HADOOP-9565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9565
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3, fs/swift
>    Affects Versions: 2.6.0
>            Reporter: Steve Loughran
>            Assignee: Thomas Demoor
>              Labels: BB2015-05-TBR
>         Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, HADOOP-9565-003.patch, HADOOP-9565-004.patch, HADOOP-9565-005.patch
>
>
> We can make the fact that some {{FileSystem}} implementations are really blobstores, with different atomicity and consistency guarantees, by adding a {{Blobstore}} interface to add to them. 
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that all blobstores implement at server-side copy operation as a substitute for rename.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)