[jira] [Assigned] (SPARK-27071) Expose additional metrics in status.api.v1.StageData

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-27071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-27071:
------------------------------------

    Assignee:     (was: Apache Spark)

> Expose additional metrics in status.api.v1.StageData
> ----------------------------------------------------
>
>                 Key: SPARK-27071
>                 URL: https://issues.apache.org/jira/browse/SPARK-27071
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.4.0
>            Reporter: Tom van Bussel
>            Priority: Major
>
> Currently StageData exposes the following metrics:
>  * executorRunTime
>  * executorCpuTime
>  * inputBytes
>  * inputRecords
>  * outputBytes
>  * outputRecords
>  * shuffleReadBytes
>  * shuffleReadRecords
>  * shuffleWriteBytes
>  * shuffleWriteRecords
>  * memoryBytesSpilled
>  * diskBytesSpilled
> These metrics are computed by aggregating the metrics of the tasks in the stage. For the task metrics however we keep track of a lot more metrics. Currently these metrics are also computed for stages (such shuffle read fetch wait time), but these are not exposed through the api. It would be very useful if these were also exposed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org