You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/03/01 05:50:00 UTC
[jira] [Work logged] (BEAM-6165) Send metrics to Flink in portable
Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6165?focusedWorklogId=206253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-206253 ]
ASF GitHub Bot logged work on BEAM-6165:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Mar/19 05:49
Start Date: 01/Mar/19 05:49
Worklog Time Spent: 10m
Work Description: ryan-williams commented on pull request #7971: [BEAM-6165] Flink portable metrics: get ptransform from MonitoringInfo, not stage name
URL: https://github.com/apache/beam/pull/7971
This fixes a problem with portable Flink metrics that dates back to #7183: the runner's "step name" is just the name of an executable stage, not the name of the metrics that have been received over the fn API.
Metrics weren't properly tagged with the ptransform they came from until #7624; it wasn't possible to do the right thing before that.
Versions of these Flink-specific changes exist in both #7915 and #7934, so I'm seeing about factoring them out here; if this is straightforward and can go in first, I can simplify those others by rebasing them on top of this.
R: @mxm, @ajamato
CC: @robertwb
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) <br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 206253)
Time Spent: 7h 20m (was: 7h 10m)
> Send metrics to Flink in portable Flink runner
> ----------------------------------------------
>
> Key: BEAM-6165
> URL: https://issues.apache.org/jira/browse/BEAM-6165
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Affects Versions: 2.8.0
> Reporter: Ryan Williams
> Assignee: Ryan Williams
> Priority: Major
> Labels: metrics, portability, portability-flink
> Fix For: 2.10.0
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> Metrics are sent from the fn harness to runner in the Python SDK (and likely Java soon), but the portable Flink runner doesn't pass them on to Flink, which it should, so that users can see them in e.g. the Flink UI or via any Flink metrics reporters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)