[jira] [Commented] (SPARK-3577) Add task metric to report spill time

["Tzach Zohar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417916#comment-15417916 ] 

Tzach Zohar commented on SPARK-3577:
------------------------------------

Does this mean that currently, spill time will be displayed as part of the *Scheduler Delay*? 
Scheduler Delay is calculated pretty much as "everything that isn't specifically measured" (see [StagePage.getSchedulerDelay|https://github.com/apache/spark/blob/v2.0.0/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala#L770]), so I'm wondering if indeed it might include  spill time if it's not included anywhere else. 

If so - this might explain long Scheduler Delay values which would be hard to make sense of otherwise (which I think is what I'm seeing...).

Thanks

> Add task metric to report spill time
> ------------------------------------
>
>                 Key: SPARK-3577
>                 URL: https://issues.apache.org/jira/browse/SPARK-3577
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle, Spark Core
>    Affects Versions: 1.1.0
>            Reporter: Kay Ousterhout
>            Priority: Minor
>
> The {{ExternalSorter}} passes its own {{ShuffleWriteMetrics}} into {{ExternalSorter}}.  The write time recorded in those metrics is never used.  We should probably add task metrics to report this spill time, since for shuffles, this would have previously been reported as part of shuffle write time (with the original hash-based sorter).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org