[jira] [Created] (TEZ-3476) Need a way to account for container localization.

["Eric Payne (JIRA)" <ji...@apache.org>]

Eric Payne created TEZ-3476:
-------------------------------

             Summary: Need a way to account for container localization.
                 Key: TEZ-3476
                 URL: https://issues.apache.org/jira/browse/TEZ-3476
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Eric Payne


Tez task attempt start times don't reflect time spent in localization.

In the MapReduce framework, the time spent in localization was included in the total runtime of each task attempt. But since Tez reuses containers, the time spent localizing for a container is not captured. The start time of the first attempt in that container will only be set after the localization has completed.

The result is that attempts can appear as if they are not being run even though there are resources available in the queue. An attempt can be assigned to a container, but if the container is on a slow node and it takes a long time to localize, the attempt state will remain pending until localization completes.

The impact risk is that tasks will not speculate during localization since they haven't started



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)