[jira] [Resolved] (SPARK-25285) Add executor task metrics to track the number of tasks started and of tasks successfully completed

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-25285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marcelo Vanzin resolved SPARK-25285.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 3.0.0

Issue resolved by pull request 22290
[https://github.com/apache/spark/pull/22290]

> Add executor task metrics to track the number of tasks started and of tasks successfully completed
> --------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-25285
>                 URL: https://issues.apache.org/jira/browse/SPARK-25285
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: Luca Canali
>            Assignee: Luca Canali
>            Priority: Minor
>             Fix For: 3.0.0
>
>
> The motivation for these additional metrics is to help in troubleshooting and monitoring task execution on a cluster. Currently available metrics include executor threadpool metrics for task completed and for active tasks. The addition of threadpool tasStarted metric will allow for example to collect info on the (approximate) number of failed tasks by computing the difference thread started – (active threads + completed tasks and/or successfully finished tasks).
>  The proposed metric finishedTasks is also intended for this type of troubleshooting. The difference between finshedTasks and threadpool.completeTasks, is that the latter is a (dropwizard library) gauge taken from the threadpool, while the former is a (dropwizard) counter computed in the [[Executor]] class, when a task successfully finishes, together with several other task metrics counters.
>  Note, there are similarities with some of the metrics introduced in SPARK-24398, however there are key differences, coming from the fact that this PR concerns the executor source, therefore providing metric values per executor + metric values do not require to pass through the listerner bus in this case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org