[jira] [Closed] (SPARK-1953) yarn client mode Application Master memory size is same as driver memory size

["Andrew Or (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Or closed SPARK-1953.
----------------------------
          Resolution: Fixed
       Fix Version/s: 1.3.0
            Assignee: WangTaoTheTonic
    Target Version/s: 1.3.0  (was: 1.1.0)

> yarn client mode Application Master memory size is same as driver memory size
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-1953
>                 URL: https://issues.apache.org/jira/browse/SPARK-1953
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.0.0
>            Reporter: Thomas Graves
>            Assignee: WangTaoTheTonic
>             Fix For: 1.3.0
>
>
> With Spark on yarn in client mode, the application master that gets created to allocated containers gets the same amount of memory as the driver running on the client. (--driver-memory option through spark-submit)  This could definitely be more then what is really needed, thus wasting resources.  The Application Master should be very small and require very little memory since all its doing is allocating and starting containers.  
> We should allow the memory for the application master to be configurable separate from the driver in client mode. 
>  We probably need to be careful about how we do this as to not cause confusion about what the options do in the various modes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org