You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "talatuyarer (via GitHub)" <gi...@apache.org> on 2023/12/17 00:29:35 UTC
[PR] Backlog metrics do not showing up in FlinkRunner [beam]
talatuyarer opened a new pull request, #29793:
URL: https://github.com/apache/beam/pull/29793
We have a Flink Job which does not emit backlog metrics. Actually metrics are emitting in KafkaIO. However I could not see them on Flink Metric system. Looks like Beam -> Flink wiring is broken. I set metricContext in Checkpointing phase which is the place metrics emit on UnBoundedReader.
I see previous PR is closed. I recreated against latest master
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "oswidan97 (via GitHub)" <gi...@apache.org>.
oswidan97 commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2152795686
Re-adding my comment from the original [PR](https://github.com/apache/beam/pull/25554) here since this one is now open.
I am facing a similar issue while using JdbcIO, I can't see the metrics that are emitted in the Flink Metric system. Do the changes here fix the issue, or are there other changes that need to be made for sources/sinks that are not `UnboundedSources`?
I am using the apache beam version 2.56.0 and beam-runners-flink-1.17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "oswidan97 (via GitHub)" <gi...@apache.org>.
oswidan97 commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2152969621
> You don't see _any_ metrics, or metrics related to checkpoints?
I am referring specifically to these [2 metrics](https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L2535-L2538), the value is always `NaN` even though data is being ingested in the sink database that I am using
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2151506117
R: @je-ik
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-1962349818
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #29793: Backlog metrics do not showing up in FlinkRunner
URL: https://github.com/apache/beam/pull/29793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2152844378
You don't see _any_ metrics, or metrics related to checkpoints?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2151507405
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2151757415
There was a significant change in https://github.com/apache/beam/pull/25525 that affects this. The `ReaderInvocationUtil#invokeCheckpoint` must be used in `FlinkUnboundedSourceReader#getReaderCheckpoint` as well. For this functionality to be preserved during refactorings, it would be nice to add at least a basic test to verify it works (which would prove it will not - at least for classic runner - in this case, because of the missing use in `FlinkUnboundedSourceReader`. The `UnboudedSourceWrapper` should be removed as drafted in https://github.com/apache/beam/pull/31530.
In the original PR there was a suspicion that this causes [some issues][1], is this resolved?
[1]: https://github.com/apache/beam/pull/25554#issuecomment-1460892717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-1948317565
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2271200033
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2154216158
> @je-ik Because Metrics are creating when Flink's open function but because they are not getting update at advance time. Flink does not receive updated metrics. My code change make metrics update part of checkpointing time. We are using this fix with 30K+ Flink jobs and build a autoscaling for Beam. https://beam.apache.org/blog/apache-beam-flink-and-kubernetes-part3/
Yes, I understand that. I'm a little struggling why @oswidan97 does not see any metrics in his case. Might be because he is apparently using bounded reader. On the other hand, the patch in this PR affects only portable runner, so I suppose for your autoscaling case you use the portable one, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "oswidan97 (via GitHub)" <gi...@apache.org>.
oswidan97 commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2154666515
New issue created [here](https://github.com/apache/beam/issues/31544) for reference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2154259892
> > @je-ik Because Metrics are creating when Flink's open function but because they are not getting update at advance time. Flink does not receive updated metrics. My code change make metrics update part of checkpointing time. We are using this fix with 30K+ Flink jobs and build a autoscaling for Beam. https://beam.apache.org/blog/apache-beam-flink-and-kubernetes-part3/
>
> Yes, I understand that. I'm a little struggling why @oswidan97 does not see any metrics in his case. Might be because he is apparently using bounded reader. On the other hand, the patch in this PR affects only portable runner, so I suppose for your autoscaling case you use the portable one, right?
No, I see the confusion. @oswidan97 refers to _sink_ not _source_. Can you please create different issue for that? This seems unrelated to this issue, because this one is about source metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2286173294
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #29793: Backlog metrics do not showing up in FlinkRunner
URL: https://github.com/apache/beam/pull/29793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-2152996624
@je-ik Because Metrics are creating when Flink's open function but because they are not getting update at advance time. Flink does not receive updated metrics. My code change make metrics update part of checkpointing time. We are using this fix with 30K+ Flink jobs and build a autoscaling for Beam. https://beam.apache.org/blog/apache-beam-flink-and-kubernetes-part3/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Backlog metrics do not showing up in FlinkRunner [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29793:
URL: https://github.com/apache/beam/pull/29793#issuecomment-1859363411
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org