[jira] [Updated] (MAPREDUCE-6957) shuffle hangs after a node manager connection timeout

["Jooseong Kim (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jooseong Kim updated MAPREDUCE-6957:
------------------------------------
    Attachment: MAPREDUCE-6957.001.patch

The patch removes the call to putBackKnownMapOutput from openShuffleUrl and leaves only one call site in copyFromHost.

> shuffle hangs after a node manager connection timeout
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-6957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6957
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Jooseong Kim
>         Attachments: MAPREDUCE-6957.001.patch
>
>
> After a connection failure from the reducer to the node manager, shuffles started to hang with the following message:
> {code}
> org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 - MergeManager returned status WAIT ...
> {code}
> There are two problems that leads to the hang.
> Problem 1.
> When a reducer has an issue connecting to the node manager, copyFromHost may call putBackKnownMapOutput on the same task attempt multiple times.
> There are two call sites of putBackKnownMapOutput in copyFromHost since MAPREDUCE-6303:
> 1. In the finally block of copyFromHost
> 2. In the catch block of openShuffleUrl.
> When openShuffleUrl fails to connect from the catch block in copyFromHost, it returns null.
> By the time openShuffleUrl returns null, putBackKnownMapOutput would have been called already for all remaining map outputs.
> However, the finally block calls putBackKnownMapOutput one more time on the map outputs.
> Problem 2. Problem 1 causes a leak in MergeManager.
> The problem occurs when multiple fetchers get the same set of map attempt outputs to fetch.
> Different fetchers reserves memory from MergeManager in Fetcher.copyMapOutput for the same map outputs.
> When the fetch succeeds, only the first map output gets committed through ShuffleSchedulerImpl.copySucceeded -> InMemoryMapOutput.commit, because commit() is gated by !finishedMaps[mapIndex].
> This may lead to a condition where usedMemory > memoryLimit, while commitMemory < mergeThreshold.
> This gets the MergeManager into a deadlock where a merge is never triggered while MergeManager cannot reserve additional space for map outputs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org