You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Jozef Vilcek <jo...@gmail.com> on 2018/08/01 14:45:01 UTC

Runner agnostic Metrics

Hello,

I would like to ask about Beam's runner agnostic metrics I found in 2.5.0
release notes.

I am considering to abandon Flink specific reporter in favour of this one,
but when looking at code, it seems feature is not fully integrated. Here:

https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java#L127

I noticed comment:
   // no metricsPusher because metrics are not supported in detached mode

Question. Is there a source where I can read and learn about current
limitations and plans to solve them. To require attached mode for metrics
to work seems to bit fragile to me.

Thanks,
Jozef

Re: Runner agnostic Metrics

Posted by Lukasz Cwik <lc...@google.com>.

The JIRA is https://issues.apache.org/jira/browse/BEAM-3310
Here is the link to the discussion thread on the dev ML:
https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
And the design doc:
https://s.apache.org/runner_independent_metrics_extraction

Feel free to reach back out with any questions based on what you read above.

On Wed, Aug 1, 2018 at 7:45 AM Jozef Vilcek <jo...@gmail.com> wrote:

> Hello,
>
> I would like to ask about Beam's runner agnostic metrics I found in 2.5.0
> release notes.
>
> I am considering to abandon Flink specific reporter in favour of this one,
> but when looking at code, it seems feature is not fully integrated. Here:
>
>
> https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java#L127
>
> I noticed comment:
>    // no metricsPusher because metrics are not supported in detached mode
>
> Question. Is there a source where I can read and learn about current
> limitations and plans to solve them. To require attached mode for metrics
> to work seems to bit fragile to me.
>
> Thanks,
> Jozef
>
>

Re: Runner agnostic Metrics

Posted by Etienne Chauchot <ec...@apache.org>.

Hi Jozef,There is a detailed discussion in the runner agnostic metrics PR about metrics in Flink in detached mode. Here
it is: https://github.com/apache/beam/pull/4548. See the discussion I had with Zorro786
BestEtienne
Le mercredi 01 août 2018 à 16:45 +0200, Jozef Vilcek a écrit :
> Hello,
> I would like to ask about Beam's runner agnostic metrics I found in 2.5.0 release notes. 
> 
> I am considering to abandon Flink specific reporter in favour of this one, but when looking at code, it seems feature
> is not fully integrated. Here:
> 
> https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/runners/flink/src/main/java/org/apache/be
> am/runners/flink/FlinkRunner.java#L127
> 
> I noticed comment:
>    // no metricsPusher because metrics are not supported in detached mode
> 
> 
> Question. Is there a source where I can read and learn about current limitations and plans to solve them. To require
> attached mode for metrics to work seems to bit fragile to me.
> 
> Thanks,
> Jozef
> 
>