You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2015/07/28 15:24:05 UTC

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

    [ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644364#comment-14644364 ] 

Thomas Graves commented on SPARK-4352:
--------------------------------------

Sorry I haven't followed this. To be clear this doesn't allow a user to specify or figure out best node locality based on input, it just passes the existing locality information to allow to be used for dynamic allocation?  SPARK-2089 is to do locality based on user input or on hdfs?

> Incorporate locality preferences in dynamic allocation requests
> ---------------------------------------------------------------
>
>                 Key: SPARK-4352
>                 URL: https://issues.apache.org/jira/browse/SPARK-4352
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, YARN
>    Affects Versions: 1.2.0
>            Reporter: Sandy Ryza
>            Assignee: Saisai Shao
>            Priority: Critical
>             Fix For: 1.5.0
>
>         Attachments: Supportpreferrednodelocationindynamicallocation.pdf
>
>
> Currently, achieving data locality in Spark is difficult unless an application takes resources on every node in the cluster.  preferredNodeLocalityData provides a sort of hacky workaround that has been broken since 1.0.
> With dynamic executor allocation, Spark requests executors in response to demand from the application.  When this occurs, it would be useful to look at the pending tasks and communicate their location preferences to the cluster resource manager. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org