[jira] [Comment Edited] (SPARK-48397) Add data write time metric to FileFormatDataWriter/BasicWriteJobStatsTracker

["Eric Yang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-48397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848824#comment-17848824 ] 

Eric Yang edited comment on SPARK-48397 at 5/23/24 6:38 AM:
------------------------------------------------------------

The PR: https://github.com/apache/spark/pull/46714


was (Author: JIRAUSER304132):
I'm working on a PR for it.

> Add data write time metric to FileFormatDataWriter/BasicWriteJobStatsTracker
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-48397
>                 URL: https://issues.apache.org/jira/browse/SPARK-48397
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: Eric Yang
>            Priority: Major
>              Labels: pull-request-available
>
> For FileFormatDataWriter we currently record metrics of "task commit time" and "job commit time" in `org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker#metrics`. We may also record the time spent on "data write" (together with the time spent on producing records from the iterator), which is usually one of the major parts of the total duration of a writing operation. It helps us identify the bottleneck and time skew, and also the generic performance tuning.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org