You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gaoxiang Liu (JIRA)" <ji...@apache.org> on 2016/10/10 18:18:20 UTC
[jira] [Comment Edited] (SPARK-3577) Add task metric to report
spill time
[ https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563038#comment-15563038 ]
Gaoxiang Liu edited comment on SPARK-3577 at 10/10/16 6:17 PM:
---------------------------------------------------------------
I find that the spill size metrics is already added in https://github.com/apache/spark/commit/bb8098f203e61111faddf2e1a04b03d62037e6c7#diff-1bd3dc38f6306e0a822f93d62c32b1d0, and I have confirm in the UI. (please refer to the attachment of this JIRA )
Also, we notices that it's wield that the spill size is somehow not reported in the reducer , but reported in the mapper.
Back to the previous question, for the spill time, if it's still relevant to add, then I plan to work on it if there is no objections.
was (Author: dreamworks007):
I find that the spill size metrics is already added in https://github.com/apache/spark/commit/bb8098f203e61111faddf2e1a04b03d62037e6c7#diff-1bd3dc38f6306e0a822f93d62c32b1d0, and I have confirm in the UI.
Also, we notices that it's wield that the spill size is somehow not reported in the reducer , but reported in the mapper.
Back to the previous question, for the spill time, if it's still relevant to add, then I plan to work on it if there is no objections.
> Add task metric to report spill time
> ------------------------------------
>
> Key: SPARK-3577
> URL: https://issues.apache.org/jira/browse/SPARK-3577
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, Spark Core
> Affects Versions: 1.1.0
> Reporter: Kay Ousterhout
> Priority: Minor
> Attachments: spill_size.jpg
>
>
> The {{ExternalSorter}} passes its own {{ShuffleWriteMetrics}} into {{ExternalSorter}}. The write time recorded in those metrics is never used. We should probably add task metrics to report this spill time, since for shuffles, this would have previously been reported as part of shuffle write time (with the original hash-based sorter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org