You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Etienne Chauchot (Jira)" <ji...@apache.org> on 2020/05/15 08:48:00 UTC
[jira] [Comment Edited] (BEAM-3926) Support MetricsPusher in
Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108082#comment-17108082 ]
Etienne Chauchot edited comment on BEAM-3926 at 5/15/20, 8:47 AM:
------------------------------------------------------------------
[~foegler], [~pabloem], [~ajamato@google.com], I have a user who asks for this feature in Dataflow. Is there a willingness to implement it for the Dataflow runner?
If so, as the MetricsPusher needs to be instanciated at the engine side (cf arguments in the description of the ticket), I was wondering if the worker part of the dataflow runner could be the correct spot and as it was donated it would enable the community to implement the feature for Dataflow.
was (Author: echauchot):
[~foegler], [~pabloem], I have a user who asks for this feature in Dataflow. Is there a willingness to implement it for the Dataflow runner?
If so, as the MetricsPusher needs to be instanciated at the engine side (cf arguments in the description of the ticket), I was wondering if the worker part of the dataflow runner could be the correct spot and as it was donated it would enable the community to implement the feature for Dataflow.
> Support MetricsPusher in Dataflow Runner
> ----------------------------------------
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
> Reporter: Scott Wegner
> Assignee: Pablo Estrada
> Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> See [relevant email thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E]. From [~echauchot]:
>
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very different from the others. It just submits a job to the cloud hosted engine. So, no access to metrics container etc... from the runner. So I think that the MetricsPusher (component responsible for merging metrics and pushing them to a sink backend) must not be instanciated in DataflowRunner otherwise it would be more a client (driver) piece of code and we will lose all the interest of being close to the execution engine (among other things instrumentation of the execution of the pipelines). I think that the MetricsPusher needs to be instanciated in the actual Dataflow engine._
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)