[jira] [Assigned] (IMPALA-8946) Prometheus histograms do not follow conventions

["Tim Armstrong (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong reassigned IMPALA-8946:
-------------------------------------

    Assignee: Guillem

> Prometheus histograms do not follow conventions
> -----------------------------------------------
>
>                 Key: IMPALA-8946
>                 URL: https://issues.apache.org/jira/browse/IMPALA-8946
>             Project: IMPALA
>          Issue Type: Bug
>            Reporter: Guillem
>            Assignee: Guillem
>            Priority: Minor
>
> We've been using Prometheus metrics and we've found that some standard Prometheus parser can not properly interpret histograms from Impala.
> For example, Python official client ([https://github.com/prometheus/client_python)] can not properly read them. I've been digging a little bit why it can't read them and I've found that Impala does not adhere to textual histogram conventions.
> The following link describes the conventions for rendering histograms on Prometheus textual format: [https://prometheus.io/docs/instrumenting/exposition_formats/#histograms-and-summaries]
> This is an example of a rendered histogram on Impala 3.3 on Prometheus endpoint:
> {code:java}
> # HELP impala_thrift_server_backend_svc_thread_wait_time Amount of time clients of Impala Backend Server spent waiting for service threads
> # TYPE impala_thrift_server_backend_svc_thread_wait_time histogram
> impala_thrift_server_backend_svc_thread_wait_time{le="0.2"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.5"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.7"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.9"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.95"} 0
> impala_thrift_server_backend_svc_thread_wait_time{le="0.999"} 0
> impala_thrift_server_backend_svc_thread_wait_time_max 0
> impala_thrift_server_backend_svc_thread_wait_time_min 0
> impala_thrift_server_backend_svc_thread_wait_time_count 49
> {code}
> The linked histogram conventions say that
> {quote}Each bucket count of a histogram named x is given as a separate sample line with the name x_bucket and a label \{le="y"} (where y is the upper bound of the bucket).
> {quote}
> And also
> {quote}A histogram must have a bucket with \{le="+Inf"}. Its value must be identical to the value of x_count.
> {quote}
> The previous example should be formatted as:
> {code:java}
> # HELP impala_thrift_server_backend_svc_thread_wait_time Amount of time clients of Impala Backend Server spent waiting for service threads
> # TYPE impala_thrift_server_backend_svc_thread_wait_time histogram
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.2"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.5"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.7"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.9"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.95"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="0.999"} 0
> impala_thrift_server_backend_svc_thread_wait_time_bucket{le="+Inf"} 49
> impala_thrift_server_backend_svc_thread_wait_time_count 49
> {code}
> I've found that with this format, the official python client is able to properly read the histograms.
> Note also that metrics suffixed with `_min` and `_max` are also out of the convention and they also break histogram parsing and maybe they need to be reported as separated metrics (maybe as gauges?)
> If you are fine with doing this changes, I already have a patch to improve the histogram formatting and I can submit it to review.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org