You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kent Yao (Jira)" <ji...@apache.org> on 2021/07/09 16:56:00 UTC
[jira] [Resolved] (SPARK-36070) Add time cost info for writing rows
out and committing the task.
[ https://issues.apache.org/jira/browse/SPARK-36070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kent Yao resolved SPARK-36070.
------------------------------
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 33279
[https://github.com/apache/spark/pull/33279]
> Add time cost info for writing rows out and committing the task.
> ----------------------------------------------------------------
>
> Key: SPARK-36070
> URL: https://issues.apache.org/jira/browse/SPARK-36070
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Kent Yao
> Assignee: Kent Yao
> Priority: Minor
> Fix For: 3.3.0
>
>
> We have a job that has a stage that contains about 8k tasks. Most tasks take about 1~10min to finish but 3 of them tasks run extremely slow. They take about 1 hour each to finish and also do their speculations. The root cause is most likely the delay of the storage system. On the spark side, we can record the time cost in logs for better bug hunting or performance tuning.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org