[jira] [Created] (SPARK-1972) Add support for tracking custom metrics in TaskMetrics

["Kalpit Shah (JIRA)" <ji...@apache.org>]

Kalpit Shah created SPARK-1972:
----------------------------------

             Summary: Add support for tracking custom metrics in TaskMetrics
                 Key: SPARK-1972
                 URL: https://issues.apache.org/jira/browse/SPARK-1972
             Project: Spark
          Issue Type: Improvement
    Affects Versions: 0.9.1
            Reporter: Kalpit Shah


Various RDDs may want to track custom metrics for improved monitoring and performance tuning. For e.g.:
1. A Task involving a JdbcRDD may want to track some metric related to JDBC execution.
2. A Task involving a user-defined RDD may want to track some metric specific to user's application.

We currently use TaskMetrics for tracking task-related metrics, which provides no way of tracking custom task-related metrics. It is not good to introduce a new field in TaskMetric everytime we want to track a custom metric. That approach would be cumbersome and ugly. Besides, some of these custom metrics may only make sense for a specific RDD-subclass. Therefore, we need TaskMetrics to provide a generic way to allow RDD-subclasses to track custom metrics when computing partitions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)