[jira] [Created] (MAPREDUCE-6823) FileOutputFormat to support configurable FileOutputCommitter factory

["Steve Loughran (JIRA)" <ji...@apache.org>]

Steve Loughran created MAPREDUCE-6823:
-----------------------------------------

             Summary: FileOutputFormat to support configurable FileOutputCommitter factory
                 Key: MAPREDUCE-6823
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6823
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv2
    Affects Versions: 3.0.0-alpha2
         Environment: Targeting S3 as the output of work
            Reporter: Steve Loughran
            Assignee: Steve Loughran


In HADOOP-13786 I'm adding a custom subclass for FileOutputFormat, one which can talk direct to the S3A Filesystem for more efficient operations, better failure modes, and, most critically, as part of HADOOP-13345, atomic commit of output. The normal committer relies on directory rename() being atomic for this; for S3 we don't have that luxury.

To support a custom committer, we need to be able to tell FileOutputFormat (and implicitly, all subclasses which don't have their own custom committer), to use our new {{S3AOutputCommitter}}.

I propose: 

# {{FileOutputFormat}} takes a factory to create committers.
# The factory to take a URI and {{TaskAttemptContext}} and return a committer
# the default implementation always returns a {{FileOutputCommitter}}
# A configuration option allows a new factory to be named
# An {{S3AOutputCommitterFactory}} to return a  {{FileOutputCommitter}} or new {{S3AOutputCommitter}} depending upon the URI of the destination.

Note that MRv1 already supports configurable committers; this is only the V2 API



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org