[jira] [Comment Edited] (SPARK-1972) Add support for setting and visualizing custom task-related metrics

["Kalpit Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14013446#comment-14013446 ] 

Kalpit Shah edited comment on SPARK-1972 at 5/30/14 9:12 AM:
-------------------------------------------------------------

I have implemented this in my fork and have submitted a PR. Here's a link to it : https://github.com/apache/spark/pull/918


was (Author: kalpit):
I have implemented this improvement in my fork and have submitted a PR. Here's a link to it : https://github.com/apache/spark/pull/918

> Add support for setting and visualizing custom task-related metrics
> -------------------------------------------------------------------
>
>                 Key: SPARK-1972
>                 URL: https://issues.apache.org/jira/browse/SPARK-1972
>             Project: Spark
>          Issue Type: Improvement
>    Affects Versions: 0.9.1
>            Reporter: Kalpit Shah
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Various RDDs may want to set/track custom metrics for improved monitoring and performance tuning. For e.g.:
> 1. A Task involving a JdbcRDD may want to track some metric related to JDBC execution.
> 2. A Task involving a user-defined RDD may want to track some metric specific to user's application.
> We currently use TaskMetrics for tracking task-related metrics, which provides no way of tracking custom task-related metrics. It is not good to introduce a new field in TaskMetric everytime we want to track a custom metric. That approach would be cumbersome and ugly. Besides, some of these custom metrics may only make sense for a specific RDD-subclass. Therefore, we need TaskMetrics to provide a generic way to allow RDD-subclasses to track custom metrics when computing partitions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)