You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Michael Marshall <mm...@apache.org> on 2022/09/02 04:05:04 UTC
[DISCUSS] Remove timestamp from Prometheus metrics
Hi Pulsar Community,
Recently, we noticed in certain Grafana metrics from the broker that
it appeared a topic had metrics reported by two different brokers at
the same time.
It turns out that the root of the problem is a concept called
"staleness" in prometheus and it is directly related to the fact that
we export timestamps with our metrics.
As such, I wrote a PR to remove these timestamps [0]. In it, I propose
that we remove the timestamps and cherry pick this fix to all active
branches of Pulsar. The PR has more detail, so please see it if you're
interested.
If removing these timestamps will break your use case, please let me
know. By my reading, we do not qualify as an application that needs to
report timestamps. Additionally, I tried to make it configurable, but
many of these classes are static, so it would be non-trivial to make
the behavior configurable.
Thanks,
Michael
[0] https://github.com/apache/pulsar/pull/17419
Re: [DISCUSS] Remove timestamp from Prometheus metrics
Posted by Asaf Mesika <as...@gmail.com>.
Great addition!
It also reduces the Prometheus response size.
On Wed, Sep 7, 2022 at 7:47 AM Michael Marshall <mm...@apache.org>
wrote:
> Merged. Depending on whether [0] will be cherry picked to release
> branches, I will cherry pick [1] or [2] to all active release
> branches.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/pull/15558
> [1] https://github.com/apache/pulsar/pull/17419
> [2]
> https://github.com/apache/pulsar/commit/b5cb02deb06760a2b6fe7b6c221e08acfabdf830
>
> On Thu, Sep 1, 2022 at 11:05 PM Michael Marshall <mm...@apache.org>
> wrote:
> >
> > Hi Pulsar Community,
> >
> > Recently, we noticed in certain Grafana metrics from the broker that
> > it appeared a topic had metrics reported by two different brokers at
> > the same time.
> >
> > It turns out that the root of the problem is a concept called
> > "staleness" in prometheus and it is directly related to the fact that
> > we export timestamps with our metrics.
> >
> > As such, I wrote a PR to remove these timestamps [0]. In it, I propose
> > that we remove the timestamps and cherry pick this fix to all active
> > branches of Pulsar. The PR has more detail, so please see it if you're
> > interested.
> >
> > If removing these timestamps will break your use case, please let me
> > know. By my reading, we do not qualify as an application that needs to
> > report timestamps. Additionally, I tried to make it configurable, but
> > many of these classes are static, so it would be non-trivial to make
> > the behavior configurable.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/pull/17419
>
Re: [DISCUSS] Remove timestamp from Prometheus metrics
Posted by Michael Marshall <mm...@apache.org>.
Merged. Depending on whether [0] will be cherry picked to release
branches, I will cherry pick [1] or [2] to all active release
branches.
Thanks,
Michael
[0] https://github.com/apache/pulsar/pull/15558
[1] https://github.com/apache/pulsar/pull/17419
[2] https://github.com/apache/pulsar/commit/b5cb02deb06760a2b6fe7b6c221e08acfabdf830
On Thu, Sep 1, 2022 at 11:05 PM Michael Marshall <mm...@apache.org> wrote:
>
> Hi Pulsar Community,
>
> Recently, we noticed in certain Grafana metrics from the broker that
> it appeared a topic had metrics reported by two different brokers at
> the same time.
>
> It turns out that the root of the problem is a concept called
> "staleness" in prometheus and it is directly related to the fact that
> we export timestamps with our metrics.
>
> As such, I wrote a PR to remove these timestamps [0]. In it, I propose
> that we remove the timestamps and cherry pick this fix to all active
> branches of Pulsar. The PR has more detail, so please see it if you're
> interested.
>
> If removing these timestamps will break your use case, please let me
> know. By my reading, we do not qualify as an application that needs to
> report timestamps. Additionally, I tried to make it configurable, but
> many of these classes are static, so it would be non-trivial to make
> the behavior configurable.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/pull/17419