[jira] [Created] (TEZ-3935) DAG aware scheduler should release unassigned new containers rather than hold them

["Jason Lowe (JIRA)" <ji...@apache.org>]

Jason Lowe created TEZ-3935:
-------------------------------

             Summary: DAG aware scheduler should release unassigned new containers rather than hold them
                 Key: TEZ-3935
                 URL: https://issues.apache.org/jira/browse/TEZ-3935
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Jason Lowe
            Assignee: Jason Lowe


I saw a case for a very large job with many containers where the DAG aware scheduler was getting behind on assigning containers.  Newly assigned containers were not finding any matching request, so they were queued for reuse processing.  However it took so long to get through all of the task and container events that the container allocations expired before the container was finally assigned and attempted to be launched.

Newly assigned containers are assigned to their matching requests, even if that violates the DAG priorities, so it should be safe to simply release these if no tasks could be found to use them.  The matching request has either been removed or already satisified with a reused container.  Besides, if we can't find any tasks to take the newly assigned container then it is very likely we have plenty of reusable containers already, and keeping more containers just makes the job a resource hog on the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)