[jira] [Created] (MAPREDUCE-6694) Make AM more resilient to potential lost of any completed container notification

["Ming Ma (JIRA)" <ji...@apache.org>]

Ming Ma created MAPREDUCE-6694:
----------------------------------

             Summary: Make AM more resilient to potential lost of any completed container notification
                 Key: MAPREDUCE-6694
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6694
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
            Reporter: Ming Ma


YARN tries to guarantee any completed container notification is delivered to AM under any circumstance, YARN-1372 is an example to make sure for the case of RM restart. However, under some corner cases, it is still possible a completed container notifications is lost or significantly delayed. For example, if NM host becomes dead when RM fails over.

AM won't preempt reducers if it thought there is at least one mapper running.
{noformat}
  void preemptReducesIfNeeded() {
...
    if (assignedRequests.maps.size() > 0) {
      // there are assigned mappers
      return;
    }
...
{noformat}

Instead of completely depending on notification from RM, it can use TaskUmbilicalProtocol to help to decide if there is any mapper running. That will make AM more resilient to any bugs in YARN.
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org