[jira] [Created] (SPARK-25285) Add executor task metrics to track the number of tasks started and of tasks successfully completed

["Luca Canali (JIRA)" <ji...@apache.org>]

Luca Canali created SPARK-25285:
-----------------------------------

             Summary: Add executor task metrics to track the number of tasks started and of tasks successfully completed
                 Key: SPARK-25285
                 URL: https://issues.apache.org/jira/browse/SPARK-25285
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.4.0
            Reporter: Luca Canali


The motivation for these additional metrics is to help in troubleshooting situations when tasks fail, are killed and/or restarted. Currently available metrics include executor threadpool metrics for task completed and for active tasks. The addition of threadpool tasStarted metric will allow for example to collect info on the (approximate) number of failed tasks by computing the difference thread started – (active threads + completed tasks and/or successfully completed tasks).

The proposed metric successfulTasks is also intended for this type of troubleshooting. The difference between  successfulTasks and threadpool.completeTasks, is that the latter is a (dropwizard library) gauge taken from the threadpool, while the former is a (dropwizard) counter computed in the [[Executor]] class, when a task successfully completes, together with several other task metrics counters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org