[jira] [Created] (SPARK-33266) Add total duration, read duration, and write duration as task level metrics

["Noritaka Sekiyama (Jira)" <ji...@apache.org>]

Noritaka Sekiyama created SPARK-33266:
-----------------------------------------

             Summary: Add total duration, read duration, and write duration as task level metrics
                 Key: SPARK-33266
                 URL: https://issues.apache.org/jira/browse/SPARK-33266
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.0.1
            Reporter: Noritaka Sekiyama


Sometimes we need to identify performance bottlenecks, for example, how long it took to read from data store, how long it took to write into another data store.

It would be great if we can have total duration, read duration, and write duration as task level metrics.

Currently it seems that both `InputMetrics` and `OutputMetrics` do not have duration related metrics.

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/InputMetrics.scala#L42-L58]

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/OutputMetrics.scala#L41-L56
]

On the other hand, other metrics such as `ShuffleWriteMetrics` has write time. We might need similar metrics for input/output.

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/ShuffleWriteMetrics.scala]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org