Beam Metrics using FlinkRunner

[Phil Franklin <ph...@gmail.com>]

I’m having difficulty accessing Beam metrics when using FlinkRunner in streaming mode. I don’t get any metrics from MetricsPusher, though the same setup delivered metrics from SparkRunner.  Probably for the same reason that MetricsPusher doesn’t work, I also don’t get any output when I call an instance of MetricsHttpSink directly.  The problem seems to be that Flink never returns from pipeline.run(), an issue that others have referred to as FlinkRunner hanging.  

Is there a solution for getting metrics in this case that I’m missing?

Thanks!
-Phil

Re: Beam Metrics using FlinkRunner

["Kaymak, Tobias" <to...@ricardo.ch>]

I am using the Flink Prometheus connector and I get metrics in Prometheus
for my running pipelines. - I am not looking at metrics in the Flink
dashboard directly.
Have you tried that? (Talk with background and example:
https://github.com/mbode/flink-prometheus-example)

On Tue, Dec 4, 2018 at 7:49 PM Phil Franklin <ph...@gmail.com>
wrote:

> I’m having difficulty accessing Beam metrics when using FlinkRunner in
> streaming mode. I don’t get any metrics from MetricsPusher, though the same
> setup delivered metrics from SparkRunner.  Probably for the same reason
> that MetricsPusher doesn’t work, I also don’t get any output when I call an
> instance of MetricsHttpSink directly.  The problem seems to be that Flink
> never returns from pipeline.run(), an issue that others have referred to as
> FlinkRunner hanging.
>
> Is there a solution for getting metrics in this case that I’m missing?
>
> Thanks!
> -Phil



-- 
Tobias Kaymak
Data Engineer

tobias.kaymak@ricardo.ch
www.ricardo.ch

Re: Beam Metrics using FlinkRunner

[Etienne Chauchot <ec...@apache.org>]

Hi Phil,
MetricsPusher is tested on all the runners in both batch and streaming mode. I just ran this test in Flink in streaming
mode and it works.
What is the command line you are using and which version of Beam?

Please also remember that, as discussed,  metrics (other flink features ) do not work if flink is used in detached mode.

Etienne


Le mardi 04 décembre 2018 à 12:49 -0600, Phil Franklin a écrit :
> I’m having difficulty accessing Beam metrics when using FlinkRunner in streaming mode. I don’t get any metrics from MetricsPusher, though the same setup delivered metrics from SparkRunner.  Probably for the same reason that MetricsPusher doesn’t work, I also don’t get any output when I call an instance of MetricsHttpSink directly.  The problem seems to be that Flink never returns from pipeline.run(), an issue that others have referred to as FlinkRunner hanging.  
> 
> Is there a solution for getting metrics in this case that I’m missing?
> 
> Thanks!
> -Phil