[jira] [Commented] (TEZ-3535) YarnTaskScheduler can hold onto low priority containers until they expire

["Siddharth Seth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678285#comment-15678285 ] 

Siddharth Seth commented on TEZ-3535:
-------------------------------------

The change makes sense to me - to assign these containers, instead of just holding on to them. This will test out the preemption handling in the scheduler. Think the way the patch is setup - we'll look at the new container in the same order that we used to, so nothing much changes there. One thing I'm not to sure about is whether this will affect runtime adversely. These containers could spend the entire job not getting assigned. Is it possible that when the tasks that match the priority of these containers are ready to run - other containers are available and will be preferred for re-use, over the new containers. Don't think there's any code in place to do this though, so we could very well end up with a container launch.

> YarnTaskScheduler can hold onto low priority containers until they expire
> -------------------------------------------------------------------------
>
>                 Key: TEZ-3535
>                 URL: https://issues.apache.org/jira/browse/TEZ-3535
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.7.1, 0.8.4
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: TEZ-3535.001.patch
>
>
> With container reuse enabled, YarnTaskScheduler will retain but not schedule any container allocations that are lower priority than the highest priority task requests.  This can lead to poor performance as these lower priority containers clog up resources needed for high priority allocations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)