You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by LucaCanali <gi...@git.apache.org> on 2018/08/30 19:47:01 UTC
[GitHub] spark pull request #22290: [SPARK-25285][CORE] Add executor task metrics, su...
GitHub user LucaCanali opened a pull request:
https://github.com/apache/spark/pull/22290
[SPARK-25285][CORE] Add executor task metrics, successfulTasks and threadpool.startedTasks
## What changes were proposed in this pull request?
The motivation for these additional metrics is to help in troubleshooting situations when tasks fail, are killed and/or restarted. Currently available metrics include executor threadpool metrics for task completed and for active tasks. The addition of threadpool tasStarted metric will allow for example to collect info on the (approximate) number of failed tasks by computing the difference thread started – (active threads + completed tasks and/or successfully completed tasks).
The proposed metric successfulTasks is also intended for this type of troubleshooting. The difference between successfulTasks and threadpool.completeTasks, is that the latter is a (dropwizard library) gauge taken from the threadpool, while the former is a (dropwizard) counter computed in the [[Executor]] class, when a task successfully completes, together with several other task metrics counters.
## How was this patch tested?
Manually tested on a YARN cluster
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/LucaCanali/spark AddMetricExecutorStartedTasks
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22290.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22290
----
commit 72bacd30eea3abf49c00c88ffcbfdefea7e758ff
Author: LucaCanali <lu...@...>
Date: 2018-08-30T15:16:24Z
Add executor metrics, successfulTasks and threadpool.startedTasks
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22290: [SPARK-25285][CORE] Add executor task metrics, successfu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22290
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22290: [SPARK-25285][CORE] Add executor task metrics, successfu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22290
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22290: [SPARK-25285][CORE] Add executor task metrics, successfu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22290
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22290: [SPARK-25285][CORE] Add executor task metrics, successfu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22290
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org