You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:38:01 UTC

[jira] [Resolved] (SPARK-7413) Time to write shuffle spill files is not captured in ShuffleWriteMetrics

     [ https://issues.apache.org/jira/browse/SPARK-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-7413.
---------------------------------
    Resolution: Incomplete

> Time to write shuffle spill files is not captured in ShuffleWriteMetrics
> ------------------------------------------------------------------------
>
>                 Key: SPARK-7413
>                 URL: https://issues.apache.org/jira/browse/SPARK-7413
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>            Reporter: Josh Rosen
>            Priority: Major
>              Labels: bulk-closed
>
> In ExternalSorter's {{spillToMergeableFile()}} method, we pass ShuffleWriteMetrics instances to the disk writers, but discard the {{shuffleWriteTime}} metrics captured here.  I think that we should account for this IO time, possibly by introducing new metrics to distinguish time spent writing spills vs. writing final shuffle output and extending the UI to break down the overall IO write time in terms of these two components.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org