[jira] [Updated] (YARN-2410) Nodemanager ShuffleHandler can possible exhaust file descriptors

["Kuhu Shukla (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kuhu Shukla updated YARN-2410:
------------------------------
    Attachment: YARN-2410-v6.patch

Thank you so much [~jlowe] for the detailed feedback. I have made all but 2 changes and request your further comments on that.

{quote}
Actually I'm not really sure why SendMapOutputParams exists separate from ReduceContext. There should be a one-to-one relationship there. 
{quote}

I totally agree. The only reason was findbugs which does not allow more than 7 parameters in a function call( or the constructor that would populate these values). If this is not an issue, I can move them into a single class. For now I have made SendMapOutputParams an inner class to ReduceContext.

{quote}
Why was reduceContext added as a TestShuffleHandler instance variable? It's specific to the new test.
{quote}

The reduceContext is a variable holds the value set by the setAttachment() method and is used by the getAttachment() answer. If I declare it in the test method, it needs be final which cannot be done due to it being used by the setter. I am looking for another way. Let me know what you think.

All other items have been done. 

> Nodemanager ShuffleHandler can possible exhaust file descriptors
> ----------------------------------------------------------------
>
>                 Key: YARN-2410
>                 URL: https://issues.apache.org/jira/browse/YARN-2410
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.5.0
>            Reporter: Nathan Roberts
>            Assignee: Kuhu Shukla
>         Attachments: YARN-2410-v1.patch, YARN-2410-v2.patch, YARN-2410-v3.patch, YARN-2410-v4.patch, YARN-2410-v5.patch, YARN-2410-v6.patch
>
>
> The async nature of the shufflehandler can cause it to open a huge number of
> file descriptors, when it runs out it crashes.
> Scenario:
> Job with 6K reduces, slow start set to 0.95, about 40 map outputs per node.
> Let's say all 6K reduces hit a node at about same time asking for their
> outputs. Each reducer will ask for all 40 map outputs over a single socket in a
> single request (not necessarily all 40 at once, but with coalescing it is
> likely to be a large number).
> sendMapOutput() will open the file for random reading and then perform an async transfer of the particular portion of this file(). This will theoretically
> happen 6000*40=240000 times which will run the NM out of file descriptors and cause it to crash.
> The algorithm should be refactored a little to not open the fds until they're
> actually needed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)