You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by ZhangJian He <sh...@gmail.com> on 2022/02/08 07:21:16 UTC
[discuss] prometheus metrics doesn't satisfy with OpenMetrics format
Before, I am working on bumping Prometheus client to 0.12.0, but they
introduce a breaking change,
https://github.com/prometheus/client_java/pull/615, adopt the `OpenMetrics
format`, which acquired all counters have `_total` suffix,
but our metrics now have these metrics, there are not satisfied with the
OpenMetrics format, for example:
- pulsar_connection_closed_total_count
- pulsar_connection_created_total_count
- pulsar_source_received_total_1min
- system_exceptions_total_1min
I want to discuss, Should we adapt the `OpenMetrics format`?
If we want to be compatible with Open Metrics, I suggest adding metrics
named `_total` in a release version like 2.10.0, and removing the origin
metric in the next release like 2.11.0.
Re: [discuss] prometheus metrics doesn't satisfy with OpenMetrics format
Posted by Michael Marshall <mm...@apache.org>.
> I am working on bumping Prometheus client to 0.12.0
What is the motivation for the update? If it is security related, that
might help us make a decision here.
> If we want to be compatible with Open Metrics, I suggest adding metrics
> named `_total` in a release version like 2.10.0, and removing the origin
> metric in the next release like 2.11.0.
My main concern with this solution is that it would increase the size
of the metrics payload by double (for the counters). Pulsar already
produces a lot of metrics. (Tangentially, I think we should make the
number of buckets configurable to cut down on the metrics payload
size, as proposed here [0].)
I think this is the kind of change that we can make during a minor
version bump as long as we give users proper notice. The main drawback
is that users will lose historical context for prometheus data from
one version of Pulsar to the next, but since Prometheus is pushing us
to make this change, it's already a change we'll have to make.
Thanks,
Michael
[0] https://github.com/apache/pulsar/issues/12069
On Wed, Feb 23, 2022 at 9:41 AM ZhangJian He <sh...@gmail.com> wrote:
>
> ping @enrico @matteo
> Please take a look when you have time.
>
> Thanks
> ZhangJian He
>
> ZhangJian He <sh...@gmail.com> 于2022年2月13日周日 09:47写道：
>
> > ping @enrico @matteo
> > Please take a look when you have time.
> >
> > Thanks
> > ZhangJian He
> >
> > ZhangJian He <sh...@gmail.com> 于2022年2月11日周五 14:09写道：
> >
> >> ping @enrico @matteo
> >>
> >> ZhangJian He <sh...@gmail.com> 于2022年2月8日周二 16:07写道：
> >>
> >>> Sorry for missing the information.
> >>> Before I upgrade the prom client, pulsar metrics is
> >>> ```
> >>>
> >>> - pulsar_connection_closed_total_count
> >>>
> >>> - pulsar_connection_created_total_count
> >>>
> >>> - pulsar_source_received_total_1min
> >>>
> >>> - system_exceptions_total_1min
> >>>
> >>> ```
> >>>
> >>> After
> >>>
> >>> ```
> >>>
> >>> - pulsar_connection_closed_total_count_total
> >>>
> >>> - pulsar_connection_created_total_count_total
> >>>
> >>> - pulsar_source_received_total_1min_total
> >>>
> >>> - system_exceptions_total_1min_total
> >>>
> >>> ```
> >>>
> >>> Prometheus client adds a `_total` suffix in pulsar metrics, because they
> >>> require all counters to have `_total` suffix, if your metric name is
> >>> not ended with `_total`, they will add it.
> >>>
> >>> I believe that the right name which satisfies `OpenMetrics` should be
> >>> ```
> >>>
> >>> - pulsar_connection_closed_total
> >>>
> >>> - pulsar_connection_created_total
> >>>
> >>> - pulsar_source_received_1min_total
> >>>
> >>> - system_exceptions_1min_total
> >>>
> >>> ```
> >>>
> >>> Summary, upgrade prometheus client introduces breaking change for these
> >>> metrics names which did not end with `_total`.
> >>>
> >>>
> >>> PS: If you let the prometheus client add `_total` in the previous
> >>> version, these metrics are not impacted.
> >>>
> >>> Enrico Olivelli <eo...@gmail.com> 于2022年2月8日周二 15:54写道：
> >>>
> >>>> What happens when you upgrade the Prometheus client ?
> >>>>
> >>>> Can you share some examples of "before" and "after" ?
> >>>> My understanding is that you posted how it looks like "after" the
> >>>> upgrade
> >>>>
> >>>> Thanks for working on this
> >>>>
> >>>> Enrico
> >>>>
> >>>> Il giorno mar 8 feb 2022 alle ore 08:21 ZhangJian He
> >>>> <sh...@gmail.com> ha scritto:
> >>>> >
> >>>> > Before, I am working on bumping Prometheus client to 0.12.0, but they
> >>>> > introduce a breaking change,
> >>>> > https://github.com/prometheus/client_java/pull/615, adopt the
> >>>> `OpenMetrics
> >>>> > format`, which acquired all counters have `_total` suffix,
> >>>> >
> >>>> > but our metrics now have these metrics, there are not satisfied with
> >>>> the
> >>>> > OpenMetrics format, for example:
> >>>> >
> >>>> > - pulsar_connection_closed_total_count
> >>>> >
> >>>> > - pulsar_connection_created_total_count
> >>>> >
> >>>> > - pulsar_source_received_total_1min
> >>>> >
> >>>> > - system_exceptions_total_1min
> >>>> >
> >>>> >
> >>>> > I want to discuss, Should we adapt the `OpenMetrics format`?
> >>>> >
> >>>> > If we want to be compatible with Open Metrics, I suggest adding
> >>>> metrics
> >>>> > named `_total` in a release version like 2.10.0, and removing the
> >>>> origin
> >>>> > metric in the next release like 2.11.0.
> >>>>
> >>>
Re: [discuss] prometheus metrics doesn't satisfy with OpenMetrics format
Posted by ZhangJian He <sh...@gmail.com>.
ping @enrico @matteo
Please take a look when you have time.
Thanks
ZhangJian He
ZhangJian He <sh...@gmail.com> 于2022年2月13日周日 09:47写道：
> ping @enrico @matteo
> Please take a look when you have time.
>
> Thanks
> ZhangJian He
>
> ZhangJian He <sh...@gmail.com> 于2022年2月11日周五 14:09写道：
>
>> ping @enrico @matteo
>>
>> ZhangJian He <sh...@gmail.com> 于2022年2月8日周二 16:07写道：
>>
>>> Sorry for missing the information.
>>> Before I upgrade the prom client, pulsar metrics is
>>> ```
>>>
>>> - pulsar_connection_closed_total_count
>>>
>>> - pulsar_connection_created_total_count
>>>
>>> - pulsar_source_received_total_1min
>>>
>>> - system_exceptions_total_1min
>>>
>>> ```
>>>
>>> After
>>>
>>> ```
>>>
>>> - pulsar_connection_closed_total_count_total
>>>
>>> - pulsar_connection_created_total_count_total
>>>
>>> - pulsar_source_received_total_1min_total
>>>
>>> - system_exceptions_total_1min_total
>>>
>>> ```
>>>
>>> Prometheus client adds a `_total` suffix in pulsar metrics, because they
>>> require all counters to have `_total` suffix, if your metric name is
>>> not ended with `_total`, they will add it.
>>>
>>> I believe that the right name which satisfies `OpenMetrics` should be
>>> ```
>>>
>>> - pulsar_connection_closed_total
>>>
>>> - pulsar_connection_created_total
>>>
>>> - pulsar_source_received_1min_total
>>>
>>> - system_exceptions_1min_total
>>>
>>> ```
>>>
>>> Summary, upgrade prometheus client introduces breaking change for these
>>> metrics names which did not end with `_total`.
>>>
>>>
>>> PS: If you let the prometheus client add `_total` in the previous
>>> version, these metrics are not impacted.
>>>
>>> Enrico Olivelli <eo...@gmail.com> 于2022年2月8日周二 15:54写道：
>>>
>>>> What happens when you upgrade the Prometheus client ?
>>>>
>>>> Can you share some examples of "before" and "after" ?
>>>> My understanding is that you posted how it looks like "after" the
>>>> upgrade
>>>>
>>>> Thanks for working on this
>>>>
>>>> Enrico
>>>>
>>>> Il giorno mar 8 feb 2022 alle ore 08:21 ZhangJian He
>>>> <sh...@gmail.com> ha scritto:
>>>> >
>>>> > Before, I am working on bumping Prometheus client to 0.12.0, but they
>>>> > introduce a breaking change,
>>>> > https://github.com/prometheus/client_java/pull/615, adopt the
>>>> `OpenMetrics
>>>> > format`, which acquired all counters have `_total` suffix,
>>>> >
>>>> > but our metrics now have these metrics, there are not satisfied with
>>>> the
>>>> > OpenMetrics format, for example:
>>>> >
>>>> > - pulsar_connection_closed_total_count
>>>> >
>>>> > - pulsar_connection_created_total_count
>>>> >
>>>> > - pulsar_source_received_total_1min
>>>> >
>>>> > - system_exceptions_total_1min
>>>> >
>>>> >
>>>> > I want to discuss, Should we adapt the `OpenMetrics format`?
>>>> >
>>>> > If we want to be compatible with Open Metrics, I suggest adding
>>>> metrics
>>>> > named `_total` in a release version like 2.10.0, and removing the
>>>> origin
>>>> > metric in the next release like 2.11.0.
>>>>
>>>
Re: [discuss] prometheus metrics doesn't satisfy with OpenMetrics format
Posted by ZhangJian He <sh...@gmail.com>.
ping @enrico @matteo
Please take a look when you have time.
Thanks
ZhangJian He
ZhangJian He <sh...@gmail.com> 于2022年2月11日周五 14:09写道：
> ping @enrico @matteo
>
> ZhangJian He <sh...@gmail.com> 于2022年2月8日周二 16:07写道：
>
>> Sorry for missing the information.
>> Before I upgrade the prom client, pulsar metrics is
>> ```
>>
>> - pulsar_connection_closed_total_count
>>
>> - pulsar_connection_created_total_count
>>
>> - pulsar_source_received_total_1min
>>
>> - system_exceptions_total_1min
>>
>> ```
>>
>> After
>>
>> ```
>>
>> - pulsar_connection_closed_total_count_total
>>
>> - pulsar_connection_created_total_count_total
>>
>> - pulsar_source_received_total_1min_total
>>
>> - system_exceptions_total_1min_total
>>
>> ```
>>
>> Prometheus client adds a `_total` suffix in pulsar metrics, because they
>> require all counters to have `_total` suffix, if your metric name is not
>> ended with `_total`, they will add it.
>>
>> I believe that the right name which satisfies `OpenMetrics` should be
>> ```
>>
>> - pulsar_connection_closed_total
>>
>> - pulsar_connection_created_total
>>
>> - pulsar_source_received_1min_total
>>
>> - system_exceptions_1min_total
>>
>> ```
>>
>> Summary, upgrade prometheus client introduces breaking change for these
>> metrics names which did not end with `_total`.
>>
>>
>> PS: If you let the prometheus client add `_total` in the previous
>> version, these metrics are not impacted.
>>
>> Enrico Olivelli <eo...@gmail.com> 于2022年2月8日周二 15:54写道：
>>
>>> What happens when you upgrade the Prometheus client ?
>>>
>>> Can you share some examples of "before" and "after" ?
>>> My understanding is that you posted how it looks like "after" the upgrade
>>>
>>> Thanks for working on this
>>>
>>> Enrico
>>>
>>> Il giorno mar 8 feb 2022 alle ore 08:21 ZhangJian He
>>> <sh...@gmail.com> ha scritto:
>>> >
>>> > Before, I am working on bumping Prometheus client to 0.12.0, but they
>>> > introduce a breaking change,
>>> > https://github.com/prometheus/client_java/pull/615, adopt the
>>> `OpenMetrics
>>> > format`, which acquired all counters have `_total` suffix,
>>> >
>>> > but our metrics now have these metrics, there are not satisfied with
>>> the
>>> > OpenMetrics format, for example:
>>> >
>>> > - pulsar_connection_closed_total_count
>>> >
>>> > - pulsar_connection_created_total_count
>>> >
>>> > - pulsar_source_received_total_1min
>>> >
>>> > - system_exceptions_total_1min
>>> >
>>> >
>>> > I want to discuss, Should we adapt the `OpenMetrics format`?
>>> >
>>> > If we want to be compatible with Open Metrics, I suggest adding metrics
>>> > named `_total` in a release version like 2.10.0, and removing the
>>> origin
>>> > metric in the next release like 2.11.0.
>>>
>>
Re: [discuss] prometheus metrics doesn't satisfy with OpenMetrics format
Posted by ZhangJian He <sh...@gmail.com>.
ping @enrico @matteo
ZhangJian He <sh...@gmail.com> 于2022年2月8日周二 16:07写道：
> Sorry for missing the information.
> Before I upgrade the prom client, pulsar metrics is
> ```
>
> - pulsar_connection_closed_total_count
>
> - pulsar_connection_created_total_count
>
> - pulsar_source_received_total_1min
>
> - system_exceptions_total_1min
>
> ```
>
> After
>
> ```
>
> - pulsar_connection_closed_total_count_total
>
> - pulsar_connection_created_total_count_total
>
> - pulsar_source_received_total_1min_total
>
> - system_exceptions_total_1min_total
>
> ```
>
> Prometheus client adds a `_total` suffix in pulsar metrics, because they
> require all counters to have `_total` suffix, if your metric name is not
> ended with `_total`, they will add it.
>
> I believe that the right name which satisfies `OpenMetrics` should be
> ```
>
> - pulsar_connection_closed_total
>
> - pulsar_connection_created_total
>
> - pulsar_source_received_1min_total
>
> - system_exceptions_1min_total
>
> ```
>
> Summary, upgrade prometheus client introduces breaking change for these
> metrics names which did not end with `_total`.
>
>
> PS: If you let the prometheus client add `_total` in the previous version,
> these metrics are not impacted.
>
> Enrico Olivelli <eo...@gmail.com> 于2022年2月8日周二 15:54写道：
>
>> What happens when you upgrade the Prometheus client ?
>>
>> Can you share some examples of "before" and "after" ?
>> My understanding is that you posted how it looks like "after" the upgrade
>>
>> Thanks for working on this
>>
>> Enrico
>>
>> Il giorno mar 8 feb 2022 alle ore 08:21 ZhangJian He
>> <sh...@gmail.com> ha scritto:
>> >
>> > Before, I am working on bumping Prometheus client to 0.12.0, but they
>> > introduce a breaking change,
>> > https://github.com/prometheus/client_java/pull/615, adopt the
>> `OpenMetrics
>> > format`, which acquired all counters have `_total` suffix,
>> >
>> > but our metrics now have these metrics, there are not satisfied with the
>> > OpenMetrics format, for example:
>> >
>> > - pulsar_connection_closed_total_count
>> >
>> > - pulsar_connection_created_total_count
>> >
>> > - pulsar_source_received_total_1min
>> >
>> > - system_exceptions_total_1min
>> >
>> >
>> > I want to discuss, Should we adapt the `OpenMetrics format`?
>> >
>> > If we want to be compatible with Open Metrics, I suggest adding metrics
>> > named `_total` in a release version like 2.10.0, and removing the origin
>> > metric in the next release like 2.11.0.
>>
>
Re: [discuss] prometheus metrics doesn't satisfy with OpenMetrics format
Posted by ZhangJian He <sh...@gmail.com>.
Sorry for missing the information.
Before I upgrade the prom client, pulsar metrics is
```
- pulsar_connection_closed_total_count
- pulsar_connection_created_total_count
- pulsar_source_received_total_1min
- system_exceptions_total_1min
```
After
```
- pulsar_connection_closed_total_count_total
- pulsar_connection_created_total_count_total
- pulsar_source_received_total_1min_total
- system_exceptions_total_1min_total
```
Prometheus client adds a `_total` suffix in pulsar metrics, because they
require all counters to have `_total` suffix, if your metric name is not
ended with `_total`, they will add it.
I believe that the right name which satisfies `OpenMetrics` should be
```
- pulsar_connection_closed_total
- pulsar_connection_created_total
- pulsar_source_received_1min_total
- system_exceptions_1min_total
```
Summary, upgrade prometheus client introduces breaking change for these
metrics names which did not end with `_total`.
PS: If you let the prometheus client add `_total` in the previous version,
these metrics are not impacted.
Enrico Olivelli <eo...@gmail.com> 于2022年2月8日周二 15:54写道：
> What happens when you upgrade the Prometheus client ?
>
> Can you share some examples of "before" and "after" ?
> My understanding is that you posted how it looks like "after" the upgrade
>
> Thanks for working on this
>
> Enrico
>
> Il giorno mar 8 feb 2022 alle ore 08:21 ZhangJian He
> <sh...@gmail.com> ha scritto:
> >
> > Before, I am working on bumping Prometheus client to 0.12.0, but they
> > introduce a breaking change,
> > https://github.com/prometheus/client_java/pull/615, adopt the
> `OpenMetrics
> > format`, which acquired all counters have `_total` suffix,
> >
> > but our metrics now have these metrics, there are not satisfied with the
> > OpenMetrics format, for example:
> >
> > - pulsar_connection_closed_total_count
> >
> > - pulsar_connection_created_total_count
> >
> > - pulsar_source_received_total_1min
> >
> > - system_exceptions_total_1min
> >
> >
> > I want to discuss, Should we adapt the `OpenMetrics format`?
> >
> > If we want to be compatible with Open Metrics, I suggest adding metrics
> > named `_total` in a release version like 2.10.0, and removing the origin
> > metric in the next release like 2.11.0.
>
Re: [discuss] prometheus metrics doesn't satisfy with OpenMetrics format
Posted by Enrico Olivelli <eo...@gmail.com>.
What happens when you upgrade the Prometheus client ?
Can you share some examples of "before" and "after" ?
My understanding is that you posted how it looks like "after" the upgrade
Thanks for working on this
Enrico
Il giorno mar 8 feb 2022 alle ore 08:21 ZhangJian He
<sh...@gmail.com> ha scritto:
>
> Before, I am working on bumping Prometheus client to 0.12.0, but they
> introduce a breaking change,
> https://github.com/prometheus/client_java/pull/615, adopt the `OpenMetrics
> format`, which acquired all counters have `_total` suffix,
>
> but our metrics now have these metrics, there are not satisfied with the
> OpenMetrics format, for example:
>
> - pulsar_connection_closed_total_count
>
> - pulsar_connection_created_total_count
>
> - pulsar_source_received_total_1min
>
> - system_exceptions_total_1min
>
>
> I want to discuss, Should we adapt the `OpenMetrics format`?
>
> If we want to be compatible with Open Metrics, I suggest adding metrics
> named `_total` in a release version like 2.10.0, and removing the origin
> metric in the next release like 2.11.0.