You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Kay Ousterhout (JIRA)" <ji...@apache.org> on 2014/09/17 22:20:34 UTC

[jira] [Created] (SPARK-3574) Shuffle finish time always reported as -1

Kay Ousterhout created SPARK-3574:
-------------------------------------

             Summary: Shuffle finish time always reported as -1
                 Key: SPARK-3574
                 URL: https://issues.apache.org/jira/browse/SPARK-3574
             Project: Spark
          Issue Type: Bug
    Affects Versions: 1.1.0
            Reporter: Kay Ousterhout


shuffleFinishTime is always reported as -1.  I think the way we fix this should be to set the shuffleFinishTime in each ShuffleWriteMetrics as the shuffles finish, but when aggregating the metrics, only report shuffleFinishTime as something other than -1 when *all* of the shuffles have completed.

[~sandyr], it looks like this was introduced in your recent patch to incrementally report metrics.  Any chance you can fix this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org