You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ryan Williams (JIRA)" <ji...@apache.org> on 2014/12/01 06:46:13 UTC

[jira] [Created] (SPARK-4665) Config value for setting yarn container overhead to a fraction of executor memory

Ryan Williams created SPARK-4665:
------------------------------------

             Summary: Config value for setting yarn container overhead to a fraction of executor memory
                 Key: SPARK-4665
                 URL: https://issues.apache.org/jira/browse/SPARK-4665
             Project: Spark
          Issue Type: Improvement
          Components: YARN
            Reporter: Ryan Williams


Currently, the {{spark.yarn.executor.memoryOverhead}} config lets you specify a number of MB of "overhead" memory to allocate for each executor.

It turns out that it is more useful to set this to a specific fraction of the executor memory size; currently a lower-bound of 7% is the default.

This lower-bound percentage should be configurable, as well; I've heard rules of thumb that closer to 10% is desirable, and in general it makes more sense to be able to specify the fraction you're targeting than to specify an absolute number of MB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org