[jira] [Updated] (MAPREDUCE-5584) ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors

["Jason Lowe (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Lowe updated MAPREDUCE-5584:
----------------------------------

    Priority: Blocker  (was: Major)

The reducers were timing out attempting to contact certain nodes for their map inputs.  Simple GET probes to the shuffle port on these nodes showed that they were indeed totally unresponsive.  Examination of the nodes showed that they had leaked a significant number of file descriptors with sockets in the CLOSE_WAIT state.

The jstacks of the NodeManager processes on these nodes also showed that all of the Netty handlers were stuck somewhere in LocalDirAllocator.getLocalPathToRead.  They were either stuck on the synchronized lock or waiting for the results of fs.exists() to return which now forks and execs {{stat}} since HADOOP-9652.

> ShuffleHandler becomes unresponsive during gridmix runs and can leak file descriptors
> -------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5584
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5584
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.3.0
>            Reporter: Jason Lowe
>            Priority: Blocker
>
> While running gridmix on 2.3 we noticed that jobs are running much slower than normal.  We tracked this down to reducers having difficulties shuffling data from maps.  Details to follow.



--
This message was sent by Atlassian JIRA
(v6.1#6144)