[jira] [Comment Edited] (SPARK-3577) Add task metric to report spill time

["Ming Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578240#comment-14578240 ] 

Ming Chen edited comment on SPARK-3577 at 6/9/15 5:26 AM:
----------------------------------------------------------

Why have not the metric been added? I think this is rather important, for that it may affect the results of the research work on this :  https://kayousterhout.github.io/trace-analysis/


was (Author: mammothcm):
Why have not the metric been added? I think this is rather important, it may affect the results of the research work on this :  https://kayousterhout.github.io/trace-analysis/

> Add task metric to report spill time
> ------------------------------------
>
>                 Key: SPARK-3577
>                 URL: https://issues.apache.org/jira/browse/SPARK-3577
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle, Spark Core
>    Affects Versions: 1.1.0
>            Reporter: Kay Ousterhout
>            Assignee: Sandy Ryza
>            Priority: Minor
>
> The {{ExternalSorter}} passes its own {{ShuffleWriteMetrics}} into {{ExternalSorter}}.  The write time recorded in those metrics is never used.  We should probably add task metrics to report this spill time, since for shuffles, this would have previously been reported as part of shuffle write time (with the original hash-based sorter).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org