[jira] [Commented] (SPARK-1524) TaskSetManager'd better not schedule tasks which has no preferred executorId using PROCESS_LOCAL in the first search process

["Mridul Muralidharan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972864#comment-13972864 ] 

Mridul Muralidharan commented on SPARK-1524:
--------------------------------------------

The expectation is to fallback to a previous schedule type in case the higher level is not valid : though this is tricky in general case.
Will need to take a look at it - though given that I am tied up with other things, if someone else wants to take a crack, please feel free to do so !

Btw, use of IP's and multiple hostnames for a host is not supported in spark - so that is something that will need to be resolved at the deployment end.

> TaskSetManager'd better not schedule tasks which has no preferred executorId using PROCESS_LOCAL in the first search process
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-1524
>                 URL: https://issues.apache.org/jira/browse/SPARK-1524
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: YanTang Zhai
>            Priority: Minor
>
> ShuffleMapTask is constructed with TaskLocation which has only host not (host, executorID) pair in DAGScheduler.
> When TaskSetManager schedules ShuffleMapTask which has no preferred executorId using specific execId host and PROCESS_LOCAL locality level, no tasks match the given locality constraint in the first search process.
> We also find that the host used by Scheduler is hostname while the host used by TaskLocation is IP in our cluster. The tow hosts do not match, that makes pendingTasksForHost HashMap empty and the finding task process against our expectation.



--
This message was sent by Atlassian JIRA
(v6.2#6252)