You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Geldenhuys, Morgan Karl" <mo...@tu-berlin.de> on 2021/12/10 17:31:51 UTC
Latency monitoring in Flink 1.14.0
Greetings all,
I am attempting to setup latency monitoring for a flink 1.14.0 job. According to the documentation<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/>, I have done the following:
In my kubernetes setup I have added the following to the kubernetes-session.sh command:
-Dmetrics.latency.granularity=\"operator\" \
-Dmetrics.latency.interval=1000 \
However, when looking at Prometheus, I do not see histograms related to latency (prometheus is configured correctly).
I have added the following to my job pom and am using the new KafkaSource/KafkaSink classes:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>1.14.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.11</artifactId>
<version>1.14.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.11</artifactId>
<version>1.14.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb_2.11</artifactId>
<version>1.14.0</version>
</dependency>
Would really appreciate some help here. Thanks in advance!
Regards,
Morgan.
Re: Latency monitoring in Flink 1.14.0
Posted by Timo Walther <tw...@apache.org>.
It turned out this was a bug and will be fixed in the next (non-log4j)
patch version:
https://issues.apache.org/jira/browse/FLINK-23704
Regards,
Timo
On 13.12.21 14:11, Timo Walther wrote:
> Hi Morgan,
>
> I was assuming that it is caused by some invalid metrics configuration.
> But I wasn't aware that this worked before and didn't read that you
> switched to the new Kafka connector. Indeed, this might be the reason. I
> will loop-in experts on this topic.
>
> Regards,
> Timo
>
>
> On 13.12.21 13:38, Geldenhuys, Morgan Karl wrote:
>> Hi Timo,
>>
>>
>> Thank you for the reply. Not really sure how that link helps besides
>> explaining what a histogram is or accessing the metrics through the UI
>> which is not what im interested in. With flink 1.12 and 1.13 the
>> latency metric was working great, however, with 1.14 and the new
>> KafkaSource/KafkaSink these metrics are just not being generated. Can
>> we confirm that it has been implemented?
>>
>>
>> Regards,
>>
>> Morgan.
>>
>> ------------------------------------------------------------------------
>> *From:* Timo Walther <tw...@apache.org>
>> *Sent:* 13 December 2021 09:53:08
>> *To:* user@flink.apache.org
>> *Subject:* Re: Latency monitoring in Flink 1.14.0
>> Hi Morgan,
>>
>> did you see this:
>>
>> https://stackguides.com/questions/68917956/read-flink-latency-tracking-metric-in-datadog
>> <https://stackguides.com/questions/68917956/read-flink-latency-tracking-metric-in-datadog>
>>
>>
>> Also `metrics.latency.granularity` must be set in the Flink
>> configuration. Not sure if `-D` forwards this properly.
>>
>> Timo
>>
>>
>>
>>
>>
>> On 10.12.21 18:31, Geldenhuys, Morgan Karl wrote:
>>> Greetings all,
>>>
>>>
>>> I am attempting to setup latency monitoring for a flink 1.14.0 job.
>>> According to the documentation
>>> <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/
>>
>> <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/>>,
>>
>>> I have done the following:
>>>
>>>
>>> In my kubernetes setup I have added the following to the
>>> kubernetes-session.sh command:
>>>
>>> -Dmetrics.latency.granularity=\"operator\" \
>>> -Dmetrics.latency.interval=1000 \
>>>
>>> However, when looking at Prometheus, I do not see histograms related
>>> to latency (prometheus is configured correctly).
>>>
>>>
>>> I have added the following to my job pom and am using the new
>>> KafkaSource/KafkaSink classes:
>>>
>>>
>>> <dependency>
>>> <groupId>org.apache.flink</groupId>
>>> <artifactId>flink-streaming-java_2.11</artifactId>
>>> <version>1.14.0</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.flink</groupId>
>>> <artifactId>flink-clients_2.11</artifactId>
>>> <version>1.14.0</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.flink</groupId>
>>> <artifactId>flink-connector-kafka_2.11</artifactId>
>>> <version>1.14.0</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.flink</groupId>
>>> <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
>>> <version>1.14.0</version>
>>> </dependency>
>>>
>>>
>>> Would really appreciate some help here. Thanks in advance!
>>>
>>>
>>> Regards,
>>>
>>> Morgan.
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Latency monitoring in Flink 1.14.0
Posted by Timo Walther <tw...@apache.org>.
Hi Morgan,
I was assuming that it is caused by some invalid metrics configuration.
But I wasn't aware that this worked before and didn't read that you
switched to the new Kafka connector. Indeed, this might be the reason. I
will loop-in experts on this topic.
Regards,
Timo
On 13.12.21 13:38, Geldenhuys, Morgan Karl wrote:
> Hi Timo,
>
>
> Thank you for the reply. Not really sure how that link helps besides
> explaining what a histogram is or accessing the metrics through the UI
> which is not what im interested in. With flink 1.12 and 1.13 the latency
> metric was working great, however, with 1.14 and the new
> KafkaSource/KafkaSink these metrics are just not being generated. Can we
> confirm that it has been implemented?
>
>
> Regards,
>
> Morgan.
>
> ------------------------------------------------------------------------
> *From:* Timo Walther <tw...@apache.org>
> *Sent:* 13 December 2021 09:53:08
> *To:* user@flink.apache.org
> *Subject:* Re: Latency monitoring in Flink 1.14.0
> Hi Morgan,
>
> did you see this:
>
> https://stackguides.com/questions/68917956/read-flink-latency-tracking-metric-in-datadog
> <https://stackguides.com/questions/68917956/read-flink-latency-tracking-metric-in-datadog>
>
> Also `metrics.latency.granularity` must be set in the Flink
> configuration. Not sure if `-D` forwards this properly.
>
> Timo
>
>
>
>
>
> On 10.12.21 18:31, Geldenhuys, Morgan Karl wrote:
>> Greetings all,
>>
>>
>> I am attempting to setup latency monitoring for a flink 1.14.0 job.
>> According to the documentation
>> <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/
> <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/>>,
>
>> I have done the following:
>>
>>
>> In my kubernetes setup I have added the following to the
>> kubernetes-session.sh command:
>>
>> -Dmetrics.latency.granularity=\"operator\" \
>> -Dmetrics.latency.interval=1000 \
>>
>> However, when looking at Prometheus, I do not see histograms related to
>> latency (prometheus is configured correctly).
>>
>>
>> I have added the following to my job pom and am using the new
>> KafkaSource/KafkaSink classes:
>>
>>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-streaming-java_2.11</artifactId>
>> <version>1.14.0</version>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-clients_2.11</artifactId>
>> <version>1.14.0</version>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-connector-kafka_2.11</artifactId>
>> <version>1.14.0</version>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
>> <version>1.14.0</version>
>> </dependency>
>>
>>
>> Would really appreciate some help here. Thanks in advance!
>>
>>
>> Regards,
>>
>> Morgan.
>>
>>
>>
>>
>>
>
Re: Latency monitoring in Flink 1.14.0
Posted by "Geldenhuys, Morgan Karl" <mo...@tu-berlin.de>.
Hi Timo,
Thank you for the reply. Not really sure how that link helps besides explaining what a histogram is or accessing the metrics through the UI which is not what im interested in. With flink 1.12 and 1.13 the latency metric was working great, however, with 1.14 and the new KafkaSource/KafkaSink these metrics are just not being generated. Can we confirm that it has been implemented?
Regards,
Morgan.
________________________________
From: Timo Walther <tw...@apache.org>
Sent: 13 December 2021 09:53:08
To: user@flink.apache.org
Subject: Re: Latency monitoring in Flink 1.14.0
Hi Morgan,
did you see this:
https://stackguides.com/questions/68917956/read-flink-latency-tracking-metric-in-datadog
Also `metrics.latency.granularity` must be set in the Flink
configuration. Not sure if `-D` forwards this properly.
Timo
On 10.12.21 18:31, Geldenhuys, Morgan Karl wrote:
> Greetings all,
>
>
> I am attempting to setup latency monitoring for a flink 1.14.0 job.
> According to the documentation
> <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/>,
> I have done the following:
>
>
> In my kubernetes setup I have added the following to the
> kubernetes-session.sh command:
>
> -Dmetrics.latency.granularity=\"operator\" \
> -Dmetrics.latency.interval=1000 \
>
> However, when looking at Prometheus, I do not see histograms related to
> latency (prometheus is configured correctly).
>
>
> I have added the following to my job pom and am using the new
> KafkaSource/KafkaSink classes:
>
>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-streaming-java_2.11</artifactId>
> <version>1.14.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-clients_2.11</artifactId>
> <version>1.14.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-connector-kafka_2.11</artifactId>
> <version>1.14.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
> <version>1.14.0</version>
> </dependency>
>
>
> Would really appreciate some help here. Thanks in advance!
>
>
> Regards,
>
> Morgan.
>
>
>
>
>
Re: Latency monitoring in Flink 1.14.0
Posted by Timo Walther <tw...@apache.org>.
Hi Morgan,
did you see this:
https://stackguides.com/questions/68917956/read-flink-latency-tracking-metric-in-datadog
Also `metrics.latency.granularity` must be set in the Flink
configuration. Not sure if `-D` forwards this properly.
Timo
On 10.12.21 18:31, Geldenhuys, Morgan Karl wrote:
> Greetings all,
>
>
> I am attempting to setup latency monitoring for a flink 1.14.0 job.
> According to the documentation
> <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/metrics/>,
> I have done the following:
>
>
> In my kubernetes setup I have added the following to the
> kubernetes-session.sh command:
>
> -Dmetrics.latency.granularity=\"operator\" \
> -Dmetrics.latency.interval=1000 \
>
> However, when looking at Prometheus, I do not see histograms related to
> latency (prometheus is configured correctly).
>
>
> I have added the following to my job pom and am using the new
> KafkaSource/KafkaSink classes:
>
>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-streaming-java_2.11</artifactId>
> <version>1.14.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-clients_2.11</artifactId>
> <version>1.14.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-connector-kafka_2.11</artifactId>
> <version>1.14.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
> <version>1.14.0</version>
> </dependency>
>
>
> Would really appreciate some help here. Thanks in advance!
>
>
> Regards,
>
> Morgan.
>
>
>
>
>