[jira] [Updated] (SPARK-22547) Don't include executor ID in metrics name

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-22547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon updated SPARK-22547:
---------------------------------
    Labels: bulk-closed  (was: )

> Don't include executor ID in metrics name
> -----------------------------------------
>
>                 Key: SPARK-22547
>                 URL: https://issues.apache.org/jira/browse/SPARK-22547
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: Li Haoyi
>            Priority: Major
>              Labels: bulk-closed
>
> Spark's metrics system prefixes all metrics collected from executors with the executor ID. 
> * https://github.com/apache/spark/blob/fccb337f9d1e44a83cfcc00ce33eae1fad367695/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L136
> This behavior causes two problems: 
> * it's not possible to aggregate over executors (since the metric name is different for each host) 
> * upstream metrics systems like Ganglia or Prometheus are put under high load because of the number of time series to store.
> By removing the `executorId` from the name of the metric we register, that solves both the above problems



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org