[jira] [Created] (SPARK-16106) TaskSchedulerImpl does not correctly handle new executors on existing hosts

["Imran Rashid (JIRA)" <ji...@apache.org>]

Imran Rashid created SPARK-16106:
------------------------------------

             Summary: TaskSchedulerImpl does not correctly handle new executors on existing hosts
                 Key: SPARK-16106
                 URL: https://issues.apache.org/jira/browse/SPARK-16106
             Project: Spark
          Issue Type: Bug
          Components: Scheduler
    Affects Versions: 2.0.0
            Reporter: Imran Rashid


The TaskSchedulerImpl updates the set of executors and hosts in each call to {{resourceOffers}}.  During this call, it also tracks whether there are any new executors observed in {{newExecAvail}}:

{code}
      executorIdToHost(o.executorId) = o.host
      executorIdToTaskCount.getOrElseUpdate(o.executorId, 0)
      if (!executorsByHost.contains(o.host)) {
        executorsByHost(o.host) = new HashSet[String]()
        executorAdded(o.executorId, o.host)
        newExecAvail = true
      }
{code}

However, this only detects when a new *host* is added, not when an additional executor is added to an existing host (a relatively common event in dynamic allocation).

The end result is that task locality and {{failedEpochs}} is not updated correctly for new executors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org