[jira] [Comment Edited] (AMBARI-4881) Clean up JMXPropertyProvider hacks for STORM metrics

["Sumit Mohanty (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMBARI-4881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917219#comment-13917219 ] 

Sumit Mohanty edited comment on AMBARI-4881 at 3/1/14 11:25 PM:
----------------------------------------------------------------

Unrelated - if we have a choice can we rename STORM_REST_API to something else such as STORM_REST_WEB_SERVICE.

{noformat}
"properties": {
     "default_port": "8745" # Storm port is always hardcoded, so we will just use the default value
     "relevant_config": "storm-site" # Just example, not needed for Storm
     "port_property": "storm.port" # Just example, not needed for Storm
  }
{noformat}

Can the above be consumed by AbstractProviderModule (modify JMXHostProvider into Generic HostProvider that provided host, port, protocol) and we can slowly move port related details for the jmx providers to the metrics.json file? APM can then make it available to JMXPropertyProvider as well as RESTPropertyProvider.

Instead of "relevant_config" and "port_property" I would suggest "port_config_type" and "port_property_name". I am thinking the eventually we can have "some_other_default", "some_other_config_type" and "some_other_property_name". 

Little confused by 
{noformat}
"metrics": {
    "metrics/api/cluster/summary/nimbus.uptime": {
      "metric": "nimbus.uptime",
      "pointInTime": true,
      "temporal": true
    }
{noformat}

Should we try to maintain similarity with the jmx/ganglia metrics?
{noformat}
"metrics": {
    "metrics/nimbus/nimbus.uptime": {    <== URL part from Ambari Resource
      "metric": "api/cluster/summary/nimbus.uptime",   <== selector/filter for the JSON received from Storm
      "pointInTime": true,
      "temporal": true
    }
{noformat}






was (Author: sumitmohanty):
Unrelated - if we have a choice can we rename STORM_REST_API to something else such as STORM_REST_WEB_SERVICE.

{quote}
"properties": {
     "default_port": "8745" # Storm port is always hardcoded, so we will just use the default value
     "relevant_config": "storm-site" # Just example, not needed for Storm
     "port_property": "storm.port" # Just example, not needed for Storm
  }
{quote}

Can the above be consumed by AbstractProviderModule (modify JMXHostProvider into Generic HostProvider that provided host, port, protocol) and we can slowly move port related details for the jmx providers to the metrics.json file? APM can then make it available to JMXPropertyProvider as well as RESTPropertyProvider.

Instead of "relevant_config" and "port_property" I would suggest "port_config_type" and "port_property_name". I am thinking the eventually we can have "some_other_default", "some_other_config_type" and "some_other_property_name". 

Little confused by 
{noformat}
"metrics": {
    "metrics/api/cluster/summary/nimbus.uptime": {
      "metric": "nimbus.uptime",
      "pointInTime": true,
      "temporal": true
    }
{noformat}

Should we try to maintain similarity with the jmx/ganglia metrics?
{noformat}
"metrics": {
    "metrics/nimbus/nimbus.uptime": {    <== URL part from Ambari Resource
      "metric": "api/cluster/summary/nimbus.uptime",   <== selector/filter for the JSON received from Storm
      "pointInTime": true,
      "temporal": true
    }
{noformat}





> Clean up JMXPropertyProvider hacks for STORM metrics
> ----------------------------------------------------
>
>                 Key: AMBARI-4881
>                 URL: https://issues.apache.org/jira/browse/AMBARI-4881
>             Project: Ambari
>          Issue Type: Task
>          Components: controller, test
>    Affects Versions: 1.5.0
>            Reporter: Dmitry Lysnichenko
>            Assignee: Dmitry Lysnichenko
>             Fix For: 1.5.0
>
>
> h2. The task
> Clean up a JMXPropertyProvider hacks for STORM metrics
> Also,
> - probably, create a separate provider for rest api 
> - current implementation is only addressing /api/cluster/summary from Storm.  Allow accessing other urls like  /api/supervisors/summary etc
> - When we refactor (assuming using the HttpProxyPropertyProvider), make that be able to take any type of reader.  Also, we want to move to Gson, not Jackson.
> - Is it possible to set the port of the STORM_REST_API component using the UI?  If so, then this will result in a failed query.
> h3. Create RestPropertyProvider
> RestPropertyProvider implementation will be registered at StackDefinedPropertyProvider like it is done for JMXPropertyProvider and GangliaPropertyProvider. Also I'm going to reuse JMXHostProvider for resolving component hostsnames. 
> h3. Add new metrics type "rest" 
> {code}
> {
>   "type": "rest",
>   "properties": {
>      "default_port": "8745" # Storm port is always hardcoded, so we will just use the default value
>      "relevant_config": "storm-site" # Just example, not needed for Storm
>      "port_property": "storm.port" # Just example, not needed for Storm
>   }
>   "metrics": {
>     "metrics/api/cluster/summary/nimbus.uptime": {
>       "metric": "nimbus.uptime",
>       "pointInTime": true,
>       "temporal": true
>     },
>    ...
> {code}
> Metric group properties define port-related settings. If "port_property" and "relevant_config" keys are defined, provider will try to extract port setting from the service config. Otherwise, provider will use the default port. RestPropertyProvider maintains a cache of per-component port numbers just like we do at org.apache.ambari.server.controller.internal.AbstractProviderModule#getPort
> Metric id "metrics/api/cluster/summary/nimbus.uptime" contains both url and property name. We can not use sole url as metric id because defining few metrics with the same id is not possible.
> h3. Adding to RestPropertyProvider support of nested json properties.
> Given a json file like:
> {code}
> {
>   "a" : "b",
>   "c" : {
>     "d" : "e",
>   }
> }
> {code}
> We may get "e" value by defining metric in form metrics/api/cluster/summary/c#d where "api/cluster/summary" is a json url, and "c#d" points to property "d" nested under property "c". Any nesting depth is supported. Accessing properties inside json value lists is not supported (I don't see any use case for that).
> Going this way, we will be able to access metrics defined at (almost) arbitrary json files at arbitrary ports/urls. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)