You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Nikola Hrusov <n....@gmail.com> on 2020/08/11 14:57:31 UTC
Missing metrics when using metric reporter on high parallelism
Hello,
I am doing some tests with flink 1.11.1 and I have noticed something
strange/wrong going on with the exported metrics.
I have a configuration like such:
*metrics.reporter.graphite.class:
org.apache.flink.metrics.graphite.GraphiteReporterFactorymetrics.reporter.graphite.host:
graphitemetrics.reporter.graphite.port:
8080metrics.reporter.graphite.protocol:
tcpmetrics.reporter.graphite.interval: 10 SECONDS*
which should produce metrics to graphite every 10 seconds.
And that works with low parallelism (e.g. <= 20). Then we get all metrics,
all the time, every 10th second.
However, when I scale my job to 200 parallelism or more, the metrics are
not sent every 10 seconds. Sometimes they are missing for up to 3 reporting
cycles.
I have had a brief look in the code here:
https://github.com/apache/flink/blob/release-1.11.1/flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistryImpl.java#L107-L144
and
it looks like there is a separate thread. That was my first guess, if it is
doing too much work on the same thread.
I have tried lowering the reporting interval from 10 SECONDS to 6-7
SECONDS, but even in that case there will be missing metrics. Even for
simpler jobs such as "source -> map -> sink" with higher parallelism that
would happen.
What can I do to further debug/make this work? Has anyone come across this
before?
Regards
,
Nikola Hrusov
Re: Missing metrics when using metric reporter on high parallelism
Posted by Chesnay Schepler <ch...@apache.org>.
IIRC this can be caused by the Carbon MAX_CREATES_PER_MINUTE setting.
I would deem it unlikely that the reporter thread is busy for 30 seconds.
On 11/08/2020 16:57, Nikola Hrusov wrote:
> Hello,
>
> I am doing some tests with flink 1.11.1 and I have noticed something
> strange/wrong going on with the exported metrics.
>
> I have a configuration like such:
> /
> metrics.reporter.graphite.class:
> org.apache.flink.metrics.graphite.GraphiteReporterFactory
> metrics.reporter.graphite.host: graphite
> metrics.reporter.graphite.port: 8080
> metrics.reporter.graphite.protocol: tcp
> metrics.reporter.graphite.interval: 10 SECONDS/
>
> which should produce metrics to graphite every 10 seconds.
>
> And that works with low parallelism (e.g. <= 20). Then we get all
> metrics, all the time, every 10th second.
> However, when I scale my job to 200 parallelism or more, the metrics
> are not sent every 10 seconds. Sometimes they are missing for up to 3
> reporting cycles.
> I have had a brief look in the code here:
> https://github.com/apache/flink/blob/release-1.11.1/flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistryImpl.java#L107-L144 and
> it looks like there is a separate thread. That was my first guess, if
> it is doing too much work on the same thread.
>
> I have tried lowering the reporting interval from 10 SECONDS to 6-7
> SECONDS, but even in that case there will be missing metrics. Even for
> simpler jobs such as "source -> map -> sink" with higher parallelism
> that would happen.
>
> What can I do to further debug/make this work? Has anyone come across
> this before?
>
> Regards
> ,
> Nikola Hrusov
>