[jira] [Created] (MAPREDUCE-7300) PathOutputCommitter to add method failedTaskAttemptCommitRecoverable()

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created MAPREDUCE-7300:
-----------------------------------------

             Summary: PathOutputCommitter to add method failedTaskAttemptCommitRecoverable()
                 Key: MAPREDUCE-7300
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7300
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: mrv2
    Affects Versions: 3.3.0
            Reporter: Steve Loughran
            Assignee: Steve Loughran


Related to HADOOP-7282, a variant solution

# we add a new method for committers to declare whether they can recover from a task attempt commit failure
# default = true; v2 and (external) EMR spark committers can return false

execution engine - MR, Spark, can look at this after a task attempt fails to commit and decide what to do

recoverable: execute/commit another task attempt

non-recoverable, one of (Configured)
* warn and continue
* abort the job

with the job abort  option, users would be confident that if a failure happened during the commit phase, they'd know about it and choose how to recover.

I'd use a long/unusual name, so that in, say, Spark, reflection to could be used to find and call the method & so compile against older releases



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org