[jira] [Updated] (SPARK-7171) Allow for more flexible use of metric sources

["Andrew Or (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Or updated SPARK-7171:
-----------------------------
    Assignee: Jacek Lewandowski

> Allow for more flexible use of metric sources
> ---------------------------------------------
>
>                 Key: SPARK-7171
>                 URL: https://issues.apache.org/jira/browse/SPARK-7171
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.3.1
>            Reporter: Jacek Lewandowski
>            Assignee: Jacek Lewandowski
>            Priority: Minor
>             Fix For: 1.5.0
>
>
> With the current API, the user is allowed to add a custom metric source by providing its class in metrics configuration. Metrics themselves are provided by Codahale and therefore they allow to register multiple metrics in a single source. Basically we can break the available types of metrics into two types: "push" and "pull" - by push metrics I mean that some execution code updates the metric by itself either periodically or every n events. On the other hand, the pull metrics include some function which pulls the data from the execution environment, when triggered. 
> h5.Problem
> The metric source is instantiated and registered during initialisation. Then, the user has no way to access the instantiated object. It is also almost impossible to access the execution environment of the current task. Therefore, the user who wanted to provide his own {{RDD}} implementation along with a dedicated metrics source, would find it very difficult to do this in a safe, concise and elegant way.
> h5.Proposed solution
> At least, for the "push" metrics, it would be nice to be able to retrieve the metrics source of particular type or with particular id from {{TaskContext}}. It would allow custom tasks to update various metrics and would greatly improve the usability of metrics.
> This could be achieved quite easily since {{TaskContext}} is created by {{Executor}}, which has access to the metrics system, it would inject some method to retrieve the particular metrics source. 
> This solution wouldn't change the current API, but just introduce one more method in {{TaskContext}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org