You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Kent Yao (Jira)" <ji...@apache.org> on 2021/07/09 08:31:00 UTC

[jira] [Created] (SPARK-36070) Add time cost info for writing rows out and committing the task.

Kent Yao created SPARK-36070:
--------------------------------

             Summary: Add time cost info for writing rows out and committing the task.
                 Key: SPARK-36070
                 URL: https://issues.apache.org/jira/browse/SPARK-36070
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.2.0
            Reporter: Kent Yao


We have a job that has a stage that contains about 8k tasks.  Most tasks take about 1~10min to finish but 3 of them tasks run extremely slow. The root cause is most likely the delay of the storage system. On the spark side, we can record the time cost in logs for better bug hunting or performance tuning.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org