[jira] [Resolved] (SPARK-27071) Expose additional metrics in status.api.v1.StageData

["Herman van Hovell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-27071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Herman van Hovell resolved SPARK-27071.
---------------------------------------
       Resolution: Fixed
    Fix Version/s: 3.0.0

> Expose additional metrics in status.api.v1.StageData
> ----------------------------------------------------
>
>                 Key: SPARK-27071
>                 URL: https://issues.apache.org/jira/browse/SPARK-27071
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.4.0
>            Reporter: Tom van Bussel
>            Assignee: Tom van Bussel
>            Priority: Major
>             Fix For: 3.0.0
>
>
> Currently StageData exposes the following metrics:
>  * executorRunTime
>  * executorCpuTime
>  * inputBytes
>  * inputRecords
>  * outputBytes
>  * outputRecords
>  * shuffleReadBytes
>  * shuffleReadRecords
>  * shuffleWriteBytes
>  * shuffleWriteRecords
>  * memoryBytesSpilled
>  * diskBytesSpilled
> These metrics are computed by aggregating the metrics of the tasks in the stage. For the task metrics however we keep track of a lot more metrics. Currently these metrics are also computed for stages (such shuffle read fetch wait time), but these are not exposed through the api. It would be very useful if these were also exposed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org