You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/07/05 12:49:00 UTC
[jira] [Work logged] (BEAM-3310) Push metrics to a backend in an
runner agnostic way
[ https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=119349&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119349 ]
ASF GitHub Bot logged work on BEAM-3310:
----------------------------------------
Author: ASF GitHub Bot
Created on: 05/Jul/18 12:48
Start Date: 05/Jul/18 12:48
Worklog Time Spent: 10m
Work Description: echauchot commented on a change in pull request #4548: [BEAM-3310] Metrics pusher
URL: https://github.com/apache/beam/pull/4548#discussion_r200338718
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
##########
@@ -120,7 +121,9 @@ public PipelineResult run(Pipeline pipeline) {
if (result instanceof DetachedEnvironment.DetachedJobExecutionResult) {
LOG.info("Pipeline submitted in Detached mode");
- return new FlinkDetachedRunnerResult();
+ FlinkDetachedRunnerResult flinkDetachedRunnerResult = new FlinkDetachedRunnerResult();
+ // no metricsPusher because metrics are not supported in detached mode
Review comment:
@zorro786 I checked and I'm afraid but supporting metrics pusher in detached mode does not seem to be possible without a modification of the upstream flink. Indeed, in this mode even flink accumulators are not available so we will have no access to the metrics at all, it is not just a matter of terminating the pusher thread as I first thought. To help you understand the architecture, here is a diagram (that is part of my upcoming presentation at the apacheCon)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 119349)
Time Spent: 14h 20m (was: 14h 10m)
> Push metrics to a backend in an runner agnostic way
> ---------------------------------------------------
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
> Issue Type: New Feature
> Components: runner-extensions-metrics, sdk-java-core
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Major
> Time Spent: 14h 20m
> Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics (either at the end of the pipeline or while it runs) because they don't have all the same capabilities towards metrics (e.g. spark runner configures sinks like csv, graphite or in memory sinks using the spark engine conf). The target is to push the metrics in the common runner code so that no matter the chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)