You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2020/10/07 18:34:00 UTC
[jira] [Updated] (MAPREDUCE-7300) PathOutputCommitter to add method
failedTaskAttemptCommitRecoverable()
[ https://issues.apache.org/jira/browse/MAPREDUCE-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated MAPREDUCE-7300:
--------------------------------------
Description:
Related to MAPREDUCE-7282 a variant solution
# we add a new method for committers to declare whether they can recover from a task attempt commit failure
# default = true; v2 and (external) EMR spark committers can return false
execution engine - MR, Spark, can look at this after a task attempt fails to commit and decide what to do
recoverable: execute/commit another task attempt
non-recoverable, one of (Configured)
* warn and continue
* abort the job
with the job abort option, users would be confident that if a failure happened during the commit phase, they'd know about it and choose how to recover.
I'd use a long/unusual name, so that in, say, Spark, reflection to could be used to find and call the method & so compile against older releases
was:
Related to HADOOP-7282, a variant solution
# we add a new method for committers to declare whether they can recover from a task attempt commit failure
# default = true; v2 and (external) EMR spark committers can return false
execution engine - MR, Spark, can look at this after a task attempt fails to commit and decide what to do
recoverable: execute/commit another task attempt
non-recoverable, one of (Configured)
* warn and continue
* abort the job
with the job abort option, users would be confident that if a failure happened during the commit phase, they'd know about it and choose how to recover.
I'd use a long/unusual name, so that in, say, Spark, reflection to could be used to find and call the method & so compile against older releases
> PathOutputCommitter to add method failedTaskAttemptCommitRecoverable()
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-7300
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7300
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> Related to MAPREDUCE-7282 a variant solution
> # we add a new method for committers to declare whether they can recover from a task attempt commit failure
> # default = true; v2 and (external) EMR spark committers can return false
> execution engine - MR, Spark, can look at this after a task attempt fails to commit and decide what to do
> recoverable: execute/commit another task attempt
> non-recoverable, one of (Configured)
> * warn and continue
> * abort the job
> with the job abort option, users would be confident that if a failure happened during the commit phase, they'd know about it and choose how to recover.
> I'd use a long/unusual name, so that in, say, Spark, reflection to could be used to find and call the method & so compile against older releases
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org