[jira] [Commented] (SPARK-21157) Report Total Memory Used by Spark Executors

["assia ydroudj (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372789#comment-16372789 ] 

assia ydroudj commented on SPARK-21157:
---------------------------------------

this paragraph is well interesting for me: "Perhaps we could
show a trend line of memory usage allowing a user to see at which stage boundary memory usage
increased." Is it done please ???? 
how can we get memory reporting values gathered in a json file or other formats to further exploitation?
thanks 

> Report Total Memory Used by Spark Executors
> -------------------------------------------
>
>                 Key: SPARK-21157
>                 URL: https://issues.apache.org/jira/browse/SPARK-21157
>             Project: Spark
>          Issue Type: Improvement
>          Components: Input/Output
>    Affects Versions: 2.1.1
>            Reporter: Jose Soltren
>            Priority: Major
>         Attachments: TotalMemoryReportingDesignDoc.pdf
>
>
> Building on some of the core ideas of SPARK-9103, this JIRA proposes tracking total memory used by Spark executors, and a means of broadcasting, aggregating, and reporting memory usage data in the Spark UI.
> Here, "total memory used" refers to memory usage that is visible outside of Spark, to an external observer such as YARN, Mesos, or the operating system. The goal of this enhancement is to give Spark users more information about how Spark clusters are using memory. Total memory will include non-Spark JVM memory and all off-heap memory.
> Please consult the attached design document for further details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org