[GitHub] [hadoop-ozone] elek opened a new pull request #127: HDDS-2300. Publish normalized Ratis metrics via the prometheus endpoint

[GitBox <gi...@apache.org>]

elek opened a new pull request #127: HDDS-2300. Publish normalized Ratis metrics via the prometheus endpoint 
URL: https://github.com/apache/hadoop-ozone/pull/127
 
 
   ## What changes were proposed in this pull request?
   
   Latest Ratis contains very good metrics about the status of the ratis ring.
   
   After RATIS-702 it will be possible to adjust the reporter of the Dropwizard based Ratis metrics and export them directly to the /prom HTTP endpoint (used by ozone insight and ratis).
   
   Unfortunately Dropwizard is very simple, there is no tag support. All of the instance specific strings are part of the metric name. For example:
   
   ```
   "ratis_grpc.log_appender.72caaf3a-fb1c-4da4-9cc0-a2ce21bb8e67@group"
    + "-72caaf3a-fb1c-4da4-9cc0-a2ce21bb8e67"
    + ".grpc_log_appender_follower_75fa730a-59f0-4547"
    + "-bd68-216162c263eb_latency", 
   ```
   
   In this patch I will use a simple method: during the export of the dropwizard metrics based on the well-known format of the ratis metrics, they are converted to proper prometheus metrics where the instance information is included as tags:
   
   ```
   ratis_grpc.log_appender.grpc_log_appender_follower_latency{instance="72caaf3a-fb1c-4da4-9cc0-a2ce21bb8e67"}
    ```
   
   With this approach we can:
   
     1. monitor easily all the Ratis pipelines with one simple query
     2. Use the metrics for ozone insight which will show health state of the Ratis pipeline
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-2300
   
   ## How was this patch tested?
   
    1. Start an ozoneperf docker-compose cluster.
    2. scale up to 3 datanodes
    3. do something (eg. freon or key upload)
    4. Check http://localhost:9090 prometheus endpoint
    5. Check the metrics with 'ratis_' prefix. There shouldn't be any unique id in the name of the metrics.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org