You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "angerszhu (Jira)" <ji...@apache.org> on 2021/08/11 07:32:00 UTC

[jira] [Updated] (SPARK-36473) Add non-unified memory metrics

     [ https://issues.apache.org/jira/browse/SPARK-36473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

angerszhu updated SPARK-36473:
------------------------------
    Description: 
Spark split memory to managed memory (storage + execution) and un-managed memory. And use `spark.memory.fraction` to adjust the fraction of this to memory.
But we have no metrics to know how much we used for unmanaged memory and we have no data evidence to optimize configuration of `spark.memory.fraction`. Maybe we should add it

> Add non-unified memory metrics
> ------------------------------
>
>                 Key: SPARK-36473
>                 URL: https://issues.apache.org/jira/browse/SPARK-36473
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.2.0
>            Reporter: angerszhu
>            Priority: Major
>
> Spark split memory to managed memory (storage + execution) and un-managed memory. And use `spark.memory.fraction` to adjust the fraction of this to memory.
> But we have no metrics to know how much we used for unmanaged memory and we have no data evidence to optimize configuration of `spark.memory.fraction`. Maybe we should add it



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org