Missing metrics in Flink v 1.15.0 rc-0

[Peter Schrott <pe...@bluerootlabs.io>]

Hi there,

I just successfully upgraded our Flink cluster to 1.15.0 rc0 - also the
corresponding job is running on this version. Looks great so far!

In the Web UI I noticed some metrics are missing, especially "Records
Received", "Bytes Sent" and "Records Sent". Those were shown in v 1.14.4.
See attached screenshot.

Other than that I noticed, when using
org.apache.flink.metrics.prometheus.PrometheusReporter , the taskmanager on
which the job is running does not report the metrics on the configured
port. Rather it returns:

➜  ~ curl http://flink-taskmanager-xxxxxx:xxxx/
curl: (52) Empty reply from server

The other taskmanager reports metrics.

The exporter is configured as followed:

# Prometheus metrics
metrics.reporters: prom
metrics.reporter.prom.class:
org.apache.flink.metrics.prometheus.PrometheusReporter
metrics.reporter.prom.port: xxxxxx

Is this a known issue with flink 1.15 rc0?

Best, Peter

[image: missingmetricsinui.png]

Re: Missing metrics in Flink v 1.15.0 rc-0

[Jing Ge <ji...@ververica.com>]

Hi,

Flink 1.15 has developed a new feature to support different sink pre- and
post-topologies[1].  New metrics e.g. NumRecordsSend has been developed to
measure records sent to the external system.  Metrics like "Bytes Sent" and
"Records Sent" measure records sent to the next task. So, in your case, it
is expected to be 0.

There are some further improvements that need to be done on the WebUI.
Task[2] has been created.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
[2]https://issues.apache.org/jira/browse/FLINK-27112

Best regards
Jing

On Thu, Apr 7, 2022 at 4:03 AM Xintong Song <to...@gmail.com> wrote:

> Hi Peter,
>
> Have you compared the DAT topologies in 1.15 / 1.14?
>
> I think it's expected that "Records Received", "Bytes Sent" and "Records
> Sent" are 0. These metrics trace the internal data exchanges between Flink
> tasks. External data changes, i.e., source reading / sink writing data from
> / to external systems, are not counted. In your case, there's only 1
> vertex in the DAG, thus no internal data exchanges.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Wed, Apr 6, 2022 at 11:21 PM Peter Schrott <pe...@bluerootlabs.io>
> wrote:
>
>> Hi there,
>>
>> I just successfully upgraded our Flink cluster to 1.15.0 rc0 - also the
>> corresponding job is running on this version. Looks great so far!
>>
>> In the Web UI I noticed some metrics are missing, especially "Records
>> Received", "Bytes Sent" and "Records Sent". Those were shown in v 1.14.4.
>> See attached screenshot.
>>
>> Other than that I noticed, when using
>> org.apache.flink.metrics.prometheus.PrometheusReporter , the taskmanager
>> on which the job is running does not report the metrics on the configured
>> port. Rather it returns:
>>
>> ➜  ~ curl http://flink-taskmanager-xxxxxx:xxxx/
>> curl: (52) Empty reply from server
>>
>> The other taskmanager reports metrics.
>>
>> The exporter is configured as followed:
>>
>> # Prometheus metrics
>> metrics.reporters: prom
>> metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
>> metrics.reporter.prom.port: xxxxxx
>>
>> Is this a known issue with flink 1.15 rc0?
>>
>> Best, Peter
>>
>> [image: missingmetricsinui.png]
>>
>

Re: Missing metrics in Flink v 1.15.0 rc-0

[Xintong Song <to...@gmail.com>]

Hi Peter,

Have you compared the DAT topologies in 1.15 / 1.14?

I think it's expected that "Records Received", "Bytes Sent" and "Records
Sent" are 0. These metrics trace the internal data exchanges between Flink
tasks. External data changes, i.e., source reading / sink writing data from
/ to external systems, are not counted. In your case, there's only 1
vertex in the DAG, thus no internal data exchanges.

Thank you~

Xintong Song



On Wed, Apr 6, 2022 at 11:21 PM Peter Schrott <pe...@bluerootlabs.io> wrote:

> Hi there,
>
> I just successfully upgraded our Flink cluster to 1.15.0 rc0 - also the
> corresponding job is running on this version. Looks great so far!
>
> In the Web UI I noticed some metrics are missing, especially "Records
> Received", "Bytes Sent" and "Records Sent". Those were shown in v 1.14.4.
> See attached screenshot.
>
> Other than that I noticed, when using
> org.apache.flink.metrics.prometheus.PrometheusReporter , the taskmanager
> on which the job is running does not report the metrics on the configured
> port. Rather it returns:
>
> ➜  ~ curl http://flink-taskmanager-xxxxxx:xxxx/
> curl: (52) Empty reply from server
>
> The other taskmanager reports metrics.
>
> The exporter is configured as followed:
>
> # Prometheus metrics
> metrics.reporters: prom
> metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
> metrics.reporter.prom.port: xxxxxx
>
> Is this a known issue with flink 1.15 rc0?
>
> Best, Peter
>
> [image: missingmetricsinui.png]
>