You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Theo Diefenthal <th...@scoop-software.de> on 2020/02/28 16:33:39 UTC
Best way to initialize a custom metric task for all task managers
and job manager
Hi,
From my backend service, I appreciate to collect metrics about the log messages, i.e. how many error and warn messages were printed over time, see e.g. for Micrometer: [ https://github.com/micrometer-metrics/micrometer/blob/master/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/logging/LogbackMetrics.java | https://github.com/micrometer-metrics/micrometer/blob/master/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/logging/LogbackMetrics.java ]
I'd like to collect those metrics for Flink as well, for all task managers and the job manager(s).
For the Task Manager, I probably could just define a static variable which initialisizes the logging metrics for logback/log4j, but how to integrate best with flink and register the counters into flink metric groups? And how to do that for the job manager as well?
What do you think is the best way to integrate the logging metrics to flink?
Best regards
Theo
--
SCOOP Software GmbH - Gut Maarhausen - Eiler Straße 3 P - D-51107 Köln
Theo Diefenthal
T +49 221 801916-196 - F +49 221 801916-17 - M +49 160 90506575
theo.diefenthal@scoop-software.de - www.scoop-software.de
Sitz der Gesellschaft: Köln, Handelsregister: Köln,
Handelsregisternummer: HRB 36625
Geschäftsführung: Dr. Oleg Balovnev, Frank Heinen,
Martin Müller-Rohde, Dr. Wolfgang Reddig, Roland Scheel
Re: Best way to initialize a custom metric task for all task managers
and job manager
Posted by Chesnay Schepler <ch...@apache.org>.
A proper solution will required a custom Flink build, were you want to
modify
org.apache.flink.runtime.metrics.util.MetricUtils#instantiateProcessMetricGroup
and
org.apache.flink.runtime.metrics.util.MetricUtils#instantiateTaskManagerMetricGroup
to add your custom metrics.
This is where the jm/tm really start initializing metrics, and as such
is your best bet.
In there you will want to define some (thread-safe!) counters that you
expose, which are modified by a custom appender that you define
programmatically and initialize at runtime.
On 28/02/2020 17:33, Theo Diefenthal wrote:
> Hi,
>
> From my backend service, I appreciate to collect metrics about the log
> messages, i.e. how many error and warn messages were printed over
> time, see e.g. for Micrometer:
> https://github.com/micrometer-metrics/micrometer/blob/master/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/logging/LogbackMetrics.java
>
> I'd like to collect those metrics for Flink as well, for all task
> managers and the job manager(s).
> For the Task Manager, I probably could just define a static variable
> which initialisizes the logging metrics for logback/log4j, but how to
> integrate best with flink and register the counters into flink metric
> groups? And how to do that for the job manager as well?
>
> What do you think is the best way to integrate the logging metrics to
> flink?
>
> Best regards
> Theo
>
> --
> SCOOP Software GmbH - Gut Maarhausen - Eiler Straße 3 P - D-51107 Köln
> Theo Diefenthal
>
> T +49 221 801916-196 - F +49 221 801916-17 - M +49 160 90506575
> theo.diefenthal@scoop-software.de - www.scoop-software.de
> Sitz der Gesellschaft: Köln, Handelsregister: Köln,
> Handelsregisternummer: HRB 36625
> Geschäftsführung: Dr. Oleg Balovnev, Frank Heinen,
> Martin Müller-Rohde, Dr. Wolfgang Reddig, Roland Scheel