[jira] [Closed] (FLINK-7200) Make metrics more Datadog friendly

["Robert Batts (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Batts closed FLINK-7200.
-------------------------------
    Resolution: Information Provided

> Make metrics more Datadog friendly
> ----------------------------------
>
>                 Key: FLINK-7200
>                 URL: https://issues.apache.org/jira/browse/FLINK-7200
>             Project: Flink
>          Issue Type: Improvement
>          Components: Metrics
>    Affects Versions: 1.3.1
>            Reporter: Robert Batts
>            Priority: Minor
>
> The current output of the Datadog Reporter is a little unfriendly to the platform they are going to from a metrics name perspective. Take for example the metric used reporting with the Datadog Kafka integration.
> kafka.consumer_lag=0000 [topic:xxxx, consumer_group: yyyy, partition: 0000]
> Through the use of tags (in this case topic, consumer_group, and partition) you can create graphs in Datadog filtered to a specific topic and consumer_group and then averaged on each partition. This allows you to visualize something like a heatmap for lag on each partition for a consumer.
> So what am I suggesting for Flink? Currently, I think the tags for Datadog are in a great place. Tags like job_id and subtask_id would be great for filtering and grouping. But, the metric name is currently too specific to a taskmanager and subtask. Currently, the metrics look something like this:
> flink_w04.taskmanager.4f378aff5730.TwitterExample.ExtractHashtags.7.numRecordsOut
> {host}.taskmanager.{tm_id}.{job_name}.{operator_name}.{subtask_index}.{metric_name}
> What I am suggesting is something more like this:
> taskmanager.TwitterExample.ExtractHashtags.numRecordsOut
> taskmanager.{job_name}.{operator_name}.{metric_name}
> (or even taskmanager.{metric_name}, but that would be a lot of tags on a single metric)
> By doing this someone could create a graph on the numRecordsOut for an entire task's metric with a single metric in Datadog rather than combining the metric for every subtask_index using the tm_id metric (that could change if a tm_id dropped out of the cluster.) Additionally, given the current set of tags being output to Datadog there is a ton of grouping and filtering that will be available if everything was on a simplified metric.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)