You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Yu Zhang <yz...@gmail.com> on 2021/05/05 22:49:05 UTC
Re: Exporting beam custom metrics to Prometheus
Hi Feba,
Yes, AFAIK, detach mode doesn’t support metrics currently [1], so you cannot query metrics result like [2]. If anyone knows how to achieve this with Flink runner, please reply here.
But you can still get some runtime metrics as Flink will push some run time metrics available to Prometheus at default 5s period.
[1] https://github.com/apache/beam/blob/2b476f55d2f67c974483426a5117a6e10fbbe4c1/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkDetachedRunnerResult.java#L40 <https://github.com/apache/beam/blob/2b476f55d2f67c974483426a5117a6e10fbbe4c1/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkDetachedRunnerResult.java#L40>
[2] https://beam.apache.org/documentation/programming-guide/#querying-metrics <https://beam.apache.org/documentation/programming-guide/#querying-metrics>
Thanks,
Yu
> On Apr 30, 2021, at 11:53, Feba Fathima <fe...@gmail.com> wrote:
>
> Hi Yu,
>
> Thanks for your suggestion. We are currently using a flink runner web submission for running the beam program. When we tried to execute the beam metrics like this, we faced an issue that custom metrics (querymetrics and metricspusher) are not supported by detached mode flink runner ( apparently all the flink web submissions are only in detached mode by default now) with the following error :
>
>
> Could not execute application. at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ... 7 more Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: The FlinkRunner does not currently support metrics. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78) ... 10 more Caused by: java.lang.UnsupportedOperationException: The FlinkRunner does not currently support metrics. at org.apache.beam.runners.flink.FlinkDetachedRunnerResult.metrics(FlinkDetachedRunnerResult.java:40)
>
> On further analysis, we also found certain discussions supporting the same. Below are the links for that.
>
> https://github.com/apache/beam/pull/4548 <https://github.com/apache/beam/pull/4548>
>
> https://ask.csdn.net/questions/7180575 <https://ask.csdn.net/questions/7180575>
>
> Is there any workaround for this so that we could successfully export the beam custom metrics to Prometheus through the metrics reporter which is running through a flink web submission (flink in detached mode)
>
> Thanks and regards
> Feba Fathima
>
>
> On Thu, 29 Apr, 2021, 10:59 pm Yu Zhang, <yzhan559233@gmail.com <ma...@gmail.com>> wrote:
> Hi Feba,
>
> I did a POC based on Beam’s doc 10.5 [1] and Flink metrics reporter [2]. You can add metrics reporter configuration in flink-conf.yaml so that metrics can be scraped from port you configured. Be sure to set disable_metrics [3] to false.
>
>
> [1] https://beam.apache.org/documentation/programming-guide/#metrics <https://beam.apache.org/documentation/programming-guide/#metrics>
> [2] https://ci.apache.org/projects/flink/flink-docs-stable/deployment/metric_reporters.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter <https://ci.apache.org/projects/flink/flink-docs-stable/deployment/metric_reporters.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter>
> [3] https://beam.apache.org/documentation/runners/flink/ <https://beam.apache.org/documentation/runners/flink/>
>
> Thanks,
> Yu
>
>> On Apr 28, 2021, at 22:58, Feba Fathima <febafathima2811@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi,
>>
>> Thanks Yu. Can you please let me know how you are displaying the counter metrics in Prometheus. Are you using any metrics sink to push the metrics from beam and is there any kind of mechanism to pull the metrics into Prometheus?
>>
>> Thanks in advance.
>>
>> Thanks & Regards,
>> Feba Fathima
>>
>> On Wed, 28 Apr, 2021, 11:28 pm Yu Zhang, <yzhan559233@gmail.com <ma...@gmail.com>> wrote:
>> Hi,
>>
>> Having similar issues. I tried with Beam 2.27.0, Flink 1.12.1 and Prometheus 2.25.0. The distribution and gauge metrics are always 0 in both Flink UI and Prometheus. Counter can be displayed correctly. This should be related with issue 10928.
>>
>> Thanks,
>> Yu
>>
>>
>> On 2021/04/28 17:39:29, Alexey Romanenko <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
>> > Hi,
>> >
>> > Could it be related [1][2]?
>> >
>> > [1] https://issues.apache.org/jira/browse/BEAM-7438 <https://issues.apache.org/jira/browse/BEAM-7438>
>> > [2] https://issues.apache.org/jira/browse/BEAM-10928 <https://issues.apache.org/jira/browse/BEAM-10928>
>> >
>> > —
>> > Alexey
>> >
>> > > On 28 Apr 2021, at 08:45, Feba Fathima <febafathima2811@gmail.com <ma...@gmail.com>> wrote:
>> > >
>> > > Hi,
>> > >
>> > > I have a java beam pipeline which is run using a flink runner. I tried adding custom metrics to it which I want to export to Prometheus. I would like to know how to export the beam custom metrics to Prometheus. Please help me with this.
>> > >
>> > > Thanks in advance
>> > > Feba Fathima
>> >
>> >
>