Separate topic and partition label for Prometheus metrics in 2.9.0

[PengHui Li <pe...@apache.org>]

Hi all,

Hope you are well, currently, we are exposing the topic metrics with an
open one label topic which
is the complete topic name such as
`persistent://public/default/xxx-partition-0`, but as
https://github.com/apache/pulsar/issues/11432 mentioned, we are not able to
have an aggregated metrics for the partitioned topic, so PR
https://github.com/apache/pulsar/pull/12225
will improve this case which can allow users to expose 2 labels, topic, and
partition.

But in order not to break compatibility, I have added an option `
splitTopicAndPartitionLabelInPrometheus` to enable the separate partition
label and by default, it should be false. So it's a 2 step upgrade story,
upgrade the broker, and enable the option.

I want to start a discussion for can we remove the option in 2.9? we have
many options and for the new major version, we do not want users to have an
option for the single label for 2 labels. Since we should ship the
enhancement in branch-2.8, so I have merged the PR which I can cherry-pick
to branch-2.8.

If we agree to remove the option in 2.9.0, I can push a PR to remove the
option in the master branch.

Thanks,
Penghui

Re: Separate topic and partition label for Prometheus metrics in 2.9.0

[Michael Marshall <mi...@gmail.com>]

> I believe that a middle ground option is to set it to 'true' in 2.9.
> It is a burden to maintain so many options for us, so I agree that we can
> drop it for 2.10.

+1, I think Enrico's idea sounds like a good way to ensure a gradual change.

By keeping the config in 2.9, we can add a note in the `broker.conf`,
the `standalone.conf`, and the `ServiceConfiguration.java` files to
let users know know the config is deprecated and will be removed in
2.10. We can also explain the upgrade path in the release notes to
help users update their monitoring stack (like grafana graphs)
accordingly.

Thanks,
Michael


On Wed, Sep 29, 2021 at 11:46 PM Enrico Olivelli <eo...@gmail.com> wrote:
>
> Penghui,
>
> Il Gio 30 Set 2021, 06:10 PengHui Li <pe...@apache.org> ha scritto:
>
> > Hi all,
> >
> > Hope you are well, currently, we are exposing the topic metrics with an
> > open one label topic which
> > is the complete topic name such as
> > `persistent://public/default/xxx-partition-0`, but as
> > https://github.com/apache/pulsar/issues/11432 mentioned, we are not able
> > to
> > have an aggregated metrics for the partitioned topic, so PR
> > https://github.com/apache/pulsar/pull/12225
> > will improve this case which can allow users to expose 2 labels, topic, and
> > partition.
> >
> > But in order not to break compatibility, I have added an option `
> > splitTopicAndPartitionLabelInPrometheus` to enable the separate partition
> > label and by default, it should be false. So it's a 2 step upgrade story,
> > upgrade the broker, and enable the option.
> >
> > I want to start a discussion for can we remove the option in 2.9? we have
> > many options and for the new major version, we do not want users to have an
> > option for the single label for 2 labels. Since we should ship the
> > enhancement in branch-2.8, so I have merged the PR which I can cherry-pick
> > to branch-2.8.
> >
> > If we agree to remove the option in 2.9.0, I can push a PR to remove the
> > option in the master branch.
> >
>
> I believe that a middle ground option is to set it to 'true' in 2.9.
>
> There may be users that need to adapt their monitoring systems, so without
> a configuration option they will have to both upgrade Pulsar and their
> systems in one shot.
>
> It is a burden to maintain so many options for us, so I agree that we can
> drop it for 2.10.
>
> So my proposal is to not drop it but to set it to 'true' for 2.9.
>
> Thanks
> Enrico
>
> >
> > Thanks,
> > Penghui
> >

Re: Separate topic and partition label for Prometheus metrics in 2.9.0

[Enrico Olivelli <eo...@gmail.com>]

Penghui,

Il Gio 30 Set 2021, 06:10 PengHui Li <pe...@apache.org> ha scritto:

> Hi all,
>
> Hope you are well, currently, we are exposing the topic metrics with an
> open one label topic which
> is the complete topic name such as
> `persistent://public/default/xxx-partition-0`, but as
> https://github.com/apache/pulsar/issues/11432 mentioned, we are not able
> to
> have an aggregated metrics for the partitioned topic, so PR
> https://github.com/apache/pulsar/pull/12225
> will improve this case which can allow users to expose 2 labels, topic, and
> partition.
>
> But in order not to break compatibility, I have added an option `
> splitTopicAndPartitionLabelInPrometheus` to enable the separate partition
> label and by default, it should be false. So it's a 2 step upgrade story,
> upgrade the broker, and enable the option.
>
> I want to start a discussion for can we remove the option in 2.9? we have
> many options and for the new major version, we do not want users to have an
> option for the single label for 2 labels. Since we should ship the
> enhancement in branch-2.8, so I have merged the PR which I can cherry-pick
> to branch-2.8.
>
> If we agree to remove the option in 2.9.0, I can push a PR to remove the
> option in the master branch.
>

I believe that a middle ground option is to set it to 'true' in 2.9.

There may be users that need to adapt their monitoring systems, so without
a configuration option they will have to both upgrade Pulsar and their
systems in one shot.

It is a burden to maintain so many options for us, so I agree that we can
drop it for 2.10.

So my proposal is to not drop it but to set it to 'true' for 2.9.

Thanks
Enrico

>
> Thanks,
> Penghui
>