[jira] [Resolved] (SPARK-2946) Allow specifying * for --num-executors in YARN

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-2946.
------------------------------
    Resolution: Duplicate

Looks like a subset of the larger request to be able to ask for all of everything in a YARN cluster.

> Allow specifying * for --num-executors in YARN
> ----------------------------------------------
>
>                 Key: SPARK-2946
>                 URL: https://issues.apache.org/jira/browse/SPARK-2946
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.0.0
>         Environment: Ubuntu precise, on YARN (CDH 5.1.0)
>            Reporter: Shay Rojansky
>            Priority: Minor
>
> It would be useful to allow specifying --num-executors * when submitting jobs to YARN, and to have Spark automatically determine how many total cores are available in the cluster by querying YARN.
> Our scenario is multiple users running research batch jobs. We never want to have a situation where cluster resources aren't being used, so ideally users would specify * and let YARN scheduling and preemption ensure fairness.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org