You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "liupengcheng (JIRA)" <ji...@apache.org> on 2019/01/28 10:12:00 UTC

[jira] [Created] (SPARK-26750) Estimate memory overhead with multi-cores

liupengcheng created SPARK-26750:
------------------------------------

             Summary: Estimate memory overhead with multi-cores
                 Key: SPARK-26750
                 URL: https://issues.apache.org/jira/browse/SPARK-26750
             Project: Spark
          Issue Type: Improvement
          Components: YARN
    Affects Versions: 2.4.0
            Reporter: liupengcheng


Currently, spark esitmate the memory overhead without taking multi-cores into account, sometimes, it might cause direct memory oom, or killed by yarn for exceeding requested physical memory. 

I think the memory overhead is related to the executor's core number(mainly the spark direct memory and some related jvm native memory, for instance, the thread stacks, GC data etc.). so maybe we can improve this estimation by taking the core number into account.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org