[jira] [Commented] (MAPREDUCE-6957) shuffle hangs after a node manager connection timeout

["daemeon reiydelle (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161742#comment-16161742 ] 

daemeon reiydelle commented on MAPREDUCE-6957:
----------------------------------------------

I always wondered what you were doing buried in the Java code every time I
walked up. Thank you for your hard work! It has made supporting Hadoop at
scale so much easier.




*Daemeon C.M. ReiydelleSan Francisco 1.415.501.0198London 44 020 8144 9872*


On Mon, Sep 11, 2017 at 11:26 AM, Jooseong Kim (JIRA) <ji...@apache.org>



> shuffle hangs after a node manager connection timeout
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-6957
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6957
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Jooseong Kim
>         Attachments: MAPREDUCE-6957.001.patch
>
>
> After a connection failure from the reducer to the node manager, shuffles started to hang with the following message:
> {code}
> org.apache.hadoop.mapreduce.task.reduce.Fetcher: fetcher#1 - MergeManager returned status WAIT ...
> {code}
> There are two problems that leads to the hang.
> Problem 1.
> When a reducer has an issue connecting to the node manager, copyFromHost may call putBackKnownMapOutput on the same task attempt multiple times.
> There are two call sites of putBackKnownMapOutput in copyFromHost since MAPREDUCE-6303:
> 1. In the finally block of copyFromHost
> 2. In the catch block of openShuffleUrl.
> When openShuffleUrl fails to connect from the catch block in copyFromHost, it returns null.
> By the time openShuffleUrl returns null, putBackKnownMapOutput would have been called already for all remaining map outputs.
> However, the finally block calls putBackKnownMapOutput one more time on the map outputs.
> Problem 2. Problem 1 causes a leak in MergeManager.
> The problem occurs when multiple fetchers get the same set of map attempt outputs to fetch.
> Different fetchers reserves memory from MergeManager in Fetcher.copyMapOutput for the same map outputs.
> When the fetch succeeds, only the first map output gets committed through ShuffleSchedulerImpl.copySucceeded -> InMemoryMapOutput.commit, because commit() is gated by !finishedMaps[mapIndex].
> This may lead to a condition where usedMemory > memoryLimit, while commitMemory < mergeThreshold.
> This gets the MergeManager into a deadlock where a merge is never triggered while MergeManager cannot reserve additional space for map outputs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org