[jira] [Commented] (SPARK-4940) Support more evenly distributing cores for Mesos mode

["Iulian Dragos (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726784#comment-14726784 ] 

Iulian Dragos commented on SPARK-4940:
--------------------------------------

Would it make sense to allocate resources in a round-robin fashion? Supposing Spark gets several offers at the same time, it would have enough info to balance executors on the available resources (or optionally, define an interval during which it holds on to resources it receives to accumulate a larger set of slaves).

The algorithm may proceed by allocating a multiple `spark.task.cores` (below cap, see SPARK-9873 which might help on its own) on each slave in the set of resources, until it can't allocate anymore.

> Support more evenly distributing cores for Mesos mode
> -----------------------------------------------------
>
>                 Key: SPARK-4940
>                 URL: https://issues.apache.org/jira/browse/SPARK-4940
>             Project: Spark
>          Issue Type: Improvement
>          Components: Mesos
>            Reporter: Timothy Chen
>         Attachments: mesos-config-difference-3nodes-vs-2nodes.png
>
>
> Currently in Coarse grain mode the spark scheduler simply takes all the resources it can on each node, but can cause uneven distribution based on resources available on each slave.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org