You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Padarn Wilson <pa...@gmail.com> on 2019/03/23 12:03:15 UTC
Flink metrics missing from UI 1.7.2
Hi User,
I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job
manager separate.
I'm having trouble seeing the metrics from my Flink job in the UI
dashboard. Actually I'm using the Datadog reporter to expose most of my
metrics, but latency tracking does not seem to be exported.
Is there anything extra that needs to be enabled to make sure metrics are
exported and viewable to the Flink UI?
Thanks
Re: Flink metrics missing from UI 1.7.2
Posted by Padarn Wilson <pa...@gmail.com>.
Thanks David. I cannot see the metrics there, so let me play around a bit
more and make sure they are enabled correctly.
On Sat, Mar 23, 2019 at 9:19 PM David Anderson <da...@ververica.com> wrote:
> > I have done this (actually I do it in my flink-conf.yaml), but I am not
> seeing any metrics at all in the Flink UI,
> > let alone the latency tracking. The latency tracking itself does not
> seem to be exported to datadog (should it be?)
>
> The latency metrics are job metrics, and are not shown in the Flink UI.
> They are available via the REST API, and I believe they should also be
> exported to datadog. You will find them at
>
> http://localhost:8081/jobs/<job id>/metrics
>
> with IDs like
>
>
> latency.source_id.bc764cd8ddf7a0cff126f51c16239658.operator_id.ea632d67b7d595e5b851708ae9ad79d6.operator_subtask_index.0.latency_p90
>
> On Sat, Mar 23, 2019 at 1:53 PM Padarn Wilson <pa...@gmail.com> wrote:
>
>> Thanks David.
>>
>> I have done this (actually I do it in my flink-conf.yaml), but I am not
>> seeing any metrics at all in the Flink UI, let alone the latency tracking.
>> The latency tracking itself does not seem to be exported to datadog (should
>> it be?)
>>
>>
>>
>> On Sat, Mar 23, 2019 at 8:43 PM David Anderson <da...@ververica.com>
>> wrote:
>>
>>> Because latency tracking is expensive, it is turned off by default. You
>>> turn it on by setting the interval; that looks something like this:
>>>
>>> env.getConfig().setLatencyTrackingInterval(1000);
>>>
>>> The full set of configuration options is described in the docs:
>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#metrics
>>>
>>> David Anderson | Training Coordinator
>>> Follow us @VervericaData
>>>
>>> --
>>> Join Flink Forward - The Apache Flink Conference
>>> Stream Processing | Event Driven | Real Time
>>>
>>>
>>> On Sat, Mar 23, 2019 at 1:03 PM Padarn Wilson <pa...@gmail.com> wrote:
>>> >
>>> > Hi User,
>>> >
>>> > I am running Fink 1.7.2 on Kubernetes in a setup with task manager and
>>> job manager separate.
>>> >
>>> > I'm having trouble seeing the metrics from my Flink job in the UI
>>> dashboard. Actually I'm using the Datadog reporter to expose most of my
>>> metrics, but latency tracking does not seem to be exported.
>>> >
>>> > Is there anything extra that needs to be enabled to make sure metrics
>>> are exported and viewable to the Flink UI?
>>> >
>>> > Thanks
>>>
>>>>
Re: Flink metrics missing from UI 1.7.2
Posted by David Anderson <da...@ververica.com>.
> I have done this (actually I do it in my flink-conf.yaml), but I am not
seeing any metrics at all in the Flink UI,
> let alone the latency tracking. The latency tracking itself does not seem
to be exported to datadog (should it be?)
The latency metrics are job metrics, and are not shown in the Flink UI.
They are available via the REST API, and I believe they should also be
exported to datadog. You will find them at
http://localhost:8081/jobs/<job id>/metrics
with IDs like
latency.source_id.bc764cd8ddf7a0cff126f51c16239658.operator_id.ea632d67b7d595e5b851708ae9ad79d6.operator_subtask_index.0.latency_p90
On Sat, Mar 23, 2019 at 1:53 PM Padarn Wilson <pa...@gmail.com> wrote:
> Thanks David.
>
> I have done this (actually I do it in my flink-conf.yaml), but I am not
> seeing any metrics at all in the Flink UI, let alone the latency tracking.
> The latency tracking itself does not seem to be exported to datadog (should
> it be?)
>
>
>
> On Sat, Mar 23, 2019 at 8:43 PM David Anderson <da...@ververica.com>
> wrote:
>
>> Because latency tracking is expensive, it is turned off by default. You
>> turn it on by setting the interval; that looks something like this:
>>
>> env.getConfig().setLatencyTrackingInterval(1000);
>>
>> The full set of configuration options is described in the docs:
>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#metrics
>>
>> David Anderson | Training Coordinator
>> Follow us @VervericaData
>>
>> --
>> Join Flink Forward - The Apache Flink Conference
>> Stream Processing | Event Driven | Real Time
>>
>>
>> On Sat, Mar 23, 2019 at 1:03 PM Padarn Wilson <pa...@gmail.com> wrote:
>> >
>> > Hi User,
>> >
>> > I am running Fink 1.7.2 on Kubernetes in a setup with task manager and
>> job manager separate.
>> >
>> > I'm having trouble seeing the metrics from my Flink job in the UI
>> dashboard. Actually I'm using the Datadog reporter to expose most of my
>> metrics, but latency tracking does not seem to be exported.
>> >
>> > Is there anything extra that needs to be enabled to make sure metrics
>> are exported and viewable to the Flink UI?
>> >
>> > Thanks
>>
>>>
Re: Flink metrics missing from UI 1.7.2
Posted by David Anderson <da...@ververica.com>.
Because latency tracking is expensive, it is turned off by default. You
turn it on by setting the interval; that looks something like this:
env.getConfig().setLatencyTrackingInterval(1000);
The full set of configuration options is described in the docs:
https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#metrics
David Anderson | Training Coordinator
Follow us @VervericaData
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
On Sat, Mar 23, 2019 at 1:03 PM Padarn Wilson <pa...@gmail.com> wrote:
>
> Hi User,
>
> I am running Fink 1.7.2 on Kubernetes in a setup with task manager and
job manager separate.
>
> I'm having trouble seeing the metrics from my Flink job in the UI
dashboard. Actually I'm using the Datadog reporter to expose most of my
metrics, but latency tracking does not seem to be exported.
>
> Is there anything extra that needs to be enabled to make sure metrics are
exported and viewable to the Flink UI?
>
> Thanks
>
Re: Flink metrics missing from UI 1.7.2
Posted by Padarn Wilson <pa...@gmail.com>.
Aha! This is almost certainly it. I remembered thinking something like this
might be a problem. I'll need to change the deployment a bit to add this
(not straightforward to edit the YAML in my case, but thanks!
On Sun, Mar 24, 2019 at 10:01 AM dawid <
apache-flink-user-mailing-list-archive@davidhaglund.se> wrote:
> Padarn Wilson-2 wrote
> > I am running Fink 1.7.2 on Kubernetes in a setup with task manager and
> job
> > manager separate.
> >
> > I'm having trouble seeing the metrics from my Flink job in the UI
> > dashboard. Actually I'm using the Datadog reporter to expose most of my
> > metrics, but latency tracking does not seem to be exported.
> >
> > Is there anything extra that needs to be enabled to make sure metrics are
> > exported and viewable to the Flink UI?
>
> With Flink 1.7 on Kubernetes you need to make sure the task managers are
> registering to the job manager with their IP addresses and not the
> hostnames, see the taskmanager-deployment.yaml manifest in [1], with the
> K8S_POD_IP environment variable and setting
> -Dtaskmanager.host=$(K8S_POD_IP).
>
> [1]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/kubernetes.html#appendix
>
> /David
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Flink metrics missing from UI 1.7.2
Posted by dawid <ap...@davidhaglund.se>.
Padarn Wilson-2 wrote
> I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job
> manager separate.
>
> I'm having trouble seeing the metrics from my Flink job in the UI
> dashboard. Actually I'm using the Datadog reporter to expose most of my
> metrics, but latency tracking does not seem to be exported.
>
> Is there anything extra that needs to be enabled to make sure metrics are
> exported and viewable to the Flink UI?
With Flink 1.7 on Kubernetes you need to make sure the task managers are
registering to the job manager with their IP addresses and not the
hostnames, see the taskmanager-deployment.yaml manifest in [1], with the
K8S_POD_IP environment variable and setting
-Dtaskmanager.host=$(K8S_POD_IP).
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/kubernetes.html#appendix
/David
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/