[jira] [Commented] (MAPREDUCE-7282) MR v2 commit algorithm should be deprecated and not the default

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208145#comment-17208145 ] 

Steve Loughran commented on MAPREDUCE-7282:
-------------------------------------------

bq.  Tasks request permission from the AM to commit.

yes, and then we assume that they continue to completion, rather than pausing for an extended period of time, so by the time the AM/spark driver gets a timeout, it can be assumed to be one of a network failure or the worker has failed/VM/k8s container terminated. The "suspended for a long time and then continues" risk does exist,  and is unlikely on a physical cluster, but in a world of VMs, not entirely inconceivable. 

I note the MR AM does track its time from last heartbeat to the YARN RM to detect partitions, workers don't. 

> MR v2 commit algorithm should be deprecated and not the default
> ---------------------------------------------------------------
>
>                 Key: MAPREDUCE-7282
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7282
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 3.3.0, 3.2.1, 3.1.3, 3.3.1
>            Reporter: Steve Loughran
>            Priority: Major
>
> The v2 MR commit algorithm moves files from the task attempt dir into the dest dir on task commit -one by one
> It is therefore not atomic
> # if a task commit fails partway through and another task attempt commits -unless exactly the same filenames are used, output of the first attempt may be included in the final result
> # if a worker partitions partway through task commit, and then continues after another attempt has committed, it may partially overwrite the output -even when the filenames are the same
> Both MR and spark assume that task commits are atomic. Either they need to consider that this is not the case, we add a way to probe for a committer supporting atomic task commit, and the engines both add handling for task commit failures (probably fail job)
> Better: we remove this as the default, maybe also warn when it is being used



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org