You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Slotterback, Chris" <Ch...@comcast.com> on 2020/06/22 17:00:47 UTC
Re: [EXTERNAL] Re: Renaming the metrics
Hi Ori,
Another more temporary brute-force option, while not officially flink, could be building a modified version of the metrics plugin into flink where you manually manipulate the prefixes yourself. It’s actually pretty easy to build the jar, and to test it you drop the jar into the plugin path. I’ve done something similar where I actually filter our a lot of the prefixes that I don’t want, because too many metric points were being generated from some custom metrics. The config for the filter is loaded from the flink conf, you could possibly implement something similar where you pass the job name in each clusters config:
https://github.com/cslotterback/flink/commit/fd8e1f77a83a3ae1253da53596d22471bb6fe902
and
https://github.com/cslotterback/flink/commit/ce3797ea46f3321885c4352ecc36b9385b7ca0ce
This isn’t what I’d call ideal, but it gets the job done. I would love a generic flink-approved method of configuring Prometheus metrics.
Chris
From: Ori Popowski <or...@gmail.com>
Date: Monday, June 22, 2020 at 12:22 PM
Cc: user <us...@flink.apache.org>
Subject: [EXTERNAL] Re: Renaming the metrics
Thanks for answering.
Unrelated to Flink, but if anyone knows a way to rename the metrics inside Prometheus I'd appreciate if you can share.
About the push gateway - I think I'll stick with the pull options because it looks like a better fit to the use case
On Mon, Jun 22, 2020 at 4:47 PM Chesnay Schepler <ch...@apache.org>> wrote:
There's currently no way to change this.
A related enhancement was proposed on FLINK-17495 that would at least allow you to attach a custom label, but the initial implementation wasn't general enough.
On 22/06/2020 15:08, Arvid Heise wrote:
Hi Ori,
I see that the PrometheusPushGatewayReporter [1] has an option for a job name, maybe you can use that.
I'm also including Chesnay who probably has more ideas.
[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html*prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter__;Iw!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNL143GF4$>
On Mon, Jun 22, 2020 at 9:01 AM Ori Popowski <or...@gmail.com>> wrote:
I have two Flink clusters sending metrics via Prometheus and they share all the metric names (i.e. flink_taskmanager_job_task_operator_currentOutputWatermark).
I want to change the flink_ prefix to something else to distinguish between the clusters (maybe the job-name).
How can I do it?
Thanks.
--
Arvid Heise | Senior Java Developer
[https://lh5.googleusercontent.com/ODbO0aq1IqKMfuoy_pw2YH8r6dqDRTq37rg3ytg11FCGJx12jJ1ff_SANPBxTHzSJTUQY9JLuoXq4NB7Om7j6Vq1lg6jIOKz8S5g2VKDGwicbj5fbY09PVb6mD5TdRuWEUvEMZTG]<https://urldefense.com/v3/__https:/www.ververica.com/__;!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNOG4n0F8$>
Follow us @VervericaData
--
Join Flink Forward<https://urldefense.com/v3/__https:/flink-forward.org/__;!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNm4cAkLw$> - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
Re: [EXTERNAL] Re: Renaming the metrics
Posted by Ori Popowski <or...@gmail.com>.
Thanks for the suggestion.
After digging a bit, we've found it most convenient to just add labels to
all our Prometheus queries, like this:
flink_taskmanager_job_task_operator_currentOutputWatermark{job_name="<job-name>"}
The job_name label will be exposed if you run your job with a job name like
this:
senv.execute("<job-name>")
On Mon, Jun 22, 2020 at 8:01 PM Slotterback, Chris <
Chris_Slotterback@comcast.com> wrote:
> Hi Ori,
>
>
>
> Another more temporary brute-force option, while not officially flink,
> could be building a modified version of the metrics plugin into flink where
> you manually manipulate the prefixes yourself. It’s actually pretty easy to
> build the jar, and to test it you drop the jar into the plugin path. I’ve
> done something similar where I actually filter our a lot of the prefixes
> that I don’t want, because too many metric points were being generated from
> some custom metrics. The config for the filter is loaded from the flink
> conf, you could possibly implement something similar where you pass the job
> name in each clusters config:
>
>
>
>
> https://github.com/cslotterback/flink/commit/fd8e1f77a83a3ae1253da53596d22471bb6fe902
>
> and
>
>
> https://github.com/cslotterback/flink/commit/ce3797ea46f3321885c4352ecc36b9385b7ca0ce
>
>
>
> This isn’t what I’d call ideal, but it gets the job done. I would love a
> generic flink-approved method of configuring Prometheus metrics.
>
>
>
> Chris
>
>
>
>
>
> *From: *Ori Popowski <or...@gmail.com>
> *Date: *Monday, June 22, 2020 at 12:22 PM
> *Cc: *user <us...@flink.apache.org>
> *Subject: *[EXTERNAL] Re: Renaming the metrics
>
>
>
> Thanks for answering.
>
>
>
> Unrelated to Flink, but if anyone knows a way to rename the metrics inside
> Prometheus I'd appreciate if you can share.
>
>
>
> About the push gateway - I think I'll stick with the pull options because
> it looks like a better fit to the use case
>
>
>
> On Mon, Jun 22, 2020 at 4:47 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
> There's currently no way to change this.
>
>
>
> A related enhancement was proposed on FLINK-17495 that would at least
> allow you to attach a custom label, but the initial implementation wasn't
> general enough.
>
>
>
> On 22/06/2020 15:08, Arvid Heise wrote:
>
> Hi Ori,
>
>
>
> I see that the PrometheusPushGatewayReporter [1] has an option for a job
> name, maybe you can use that.
>
>
>
> I'm also including Chesnay who probably has more ideas.
>
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter
> <https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html*prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter__;Iw!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNL143GF4$>
>
>
>
> On Mon, Jun 22, 2020 at 9:01 AM Ori Popowski <or...@gmail.com> wrote:
>
> I have two Flink clusters sending metrics via Prometheus and they share
> all the metric names (i.e.
> flink_taskmanager_job_task_operator_currentOutputWatermark).
>
>
>
> I want to change the flink_ prefix to something else to distinguish
> between the clusters (maybe the job-name).
>
>
>
> How can I do it?
>
>
>
> Thanks.
>
>
>
> --
>
> *Arvid Heise *| Senior Java Developer
>
>
> <https://urldefense.com/v3/__https:/www.ververica.com/__;!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNOG4n0F8$>
>
>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward
> <https://urldefense.com/v3/__https:/flink-forward.org/__;!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNm4cAkLw$>
> - The Apache Flink Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
>
> Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>
>
>
>