You are viewing a plain text version of this content. The canonical link for it is here.

Posted to yarn-dev@hadoop.apache.org by "BELUGA BEHR (JIRA)" <ji...@apache.org> on 2018/09/18 16:04:00 UTC

[jira] [Created] (YARN-8789) Add BoundedQueue to AsyncDispatcher

BELUGA BEHR created YARN-8789:
---------------------------------

             Summary: Add BoundedQueue to AsyncDispatcher
                 Key: YARN-8789
                 URL: https://issues.apache.org/jira/browse/YARN-8789
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: applications
    Affects Versions: 3.2.0
            Reporter: BELUGA BEHR


I recently came across a scenario where an MR ApplicationMaster was failing with an OOM exception.  It had many thousands of Mappers and thousands of Reducers.  It was noted that in the logging that the event-queue of {{AsyncDispatcher}} had a very large number of item in it and was seemingly never decreasing.

I started looking at the code and thought it could use some clean up, simplification, and the ability to specify a bounded queue so that any incoming events are throttled until they can be processed.  This will protect the ApplicationMaster from a flood of events.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org