You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Gopal V (JIRA)" <ji...@apache.org> on 2017/12/03 08:30:01 UTC
[jira] [Created] (TEZ-3872) OneToOne Edge: Scheduling misses due to
released containers
Gopal V created TEZ-3872:
----------------------------
Summary: OneToOne Edge: Scheduling misses due to released containers
Key: TEZ-3872
URL: https://issues.apache.org/jira/browse/TEZ-3872
Project: Apache Tez
Issue Type: Bug
Reporter: Gopal V
https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/TaskSchedulerManager.java#L477
That's where it decides between using container or node/racks - it does not record the hosts/racks for the container, the container affinity ignores node affinity fall backs.
https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java#L986
Inside the YARN task scheduling impl, this only picks up the host if the container is being held at the moment, not if it has been released - this also has no checks for in use containers.
TaskSchedulerManager can grab ta.containerNodeId, directly off the attempt information to get the host info as well container info.
This needs a new allocateTask API which has container, host, rack in the order of scheduling preference.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)