You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/08/31 17:14:00 UTC
[jira] [Work logged] (BEAM-5246) Beam metrics exported as flink
metrics are not correct
[ https://issues.apache.org/jira/browse/BEAM-5246?focusedWorklogId=140174&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-140174 ]
ASF GitHub Bot logged work on BEAM-5246:
----------------------------------------
Author: ASF GitHub Bot
Created on: 31/Aug/18 17:13
Start Date: 31/Aug/18 17:13
Worklog Time Spent: 10m
Work Description: JozoVilcek opened a new pull request #6319: [BEAM-5246] Metric container should forward only metrics of the stepName
URL: https://github.com/apache/beam/pull/6319
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 140174)
Time Spent: 10m
Remaining Estimate: 0h
> Beam metrics exported as flink metrics are not correct
> ------------------------------------------------------
>
> Key: BEAM-5246
> URL: https://issues.apache.org/jira/browse/BEAM-5246
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Affects Versions: 2.6.0
> Reporter: Jozef Vilcek
> Assignee: Aljoscha Krettek
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In Flink UI and fink native MetricReported, I am seeing too many instances of my Beam metric counter. It looks like the counter is materialised for every operator running within the task, although is is emitter from only one beam step (which should map to one operator?). This produces double counting.
> A bit debugging I noticed this is happening for stream jobs. In batch I was not able to reproduce it. Problem might be in FlinkMetricContainer.
> [https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L86]
> The update seems to be called from operators after finishing the bundle. Data from accumulator are flushed to `runtimeContext.getMetricGroup()`. The scope of accumulator seems to be different than metricGroup as in there with different call the scope components change, especially for operatorID. It seems like during the run, `metricResult.getStep()` does not match operatorName of metricGroup where metric is being pushed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)