You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Atul Lal <at...@gmail.com> on 2023/03/09 09:48:47 UTC

Flink metric collection

Hi everyone,

I am trying to do some experiments with Flink. I am trying to modify the
source code of Flink for this experiment, and I am starting a process
thread from JobMaster.java constructor whenever a new job is started in
Flink. In this thread, I want to monitor a few metrics related to the job
and make some decisions based on it.

Is there any way to collect metrics related to a job without using REST
endpoints? Because I think using REST endpoints here is pointless as this
is internal code running on JobMaster. If there is no other way than using
REST endpoints, is there any easy way to serialize or parse JSON responses
from those endpoints as the response structures are already defined in the
classes.

I would really appreciate it if someone could help me with this.

Thank you,
Atul

Re: Flink metric collection

Posted by Atul Lal <at...@gmail.com>.

Hi everyone,

Thank you very much for the prompt responses and help. I was able to get
the MetricFetcher connected with my thread and am successfully able to
collect metrics needed for my research and experiments.

Thank you,
Atul

On Thu, Mar 9, 2023 at 7:53 PM Atul Lal <at...@gmail.com> wrote:

> Hi everyone,
>
> I also think the reporter won't work for my use case, it could work but
> then I am trying to modify the placement of operators on taskmanagers
> manually based on my some decisions made from metric collection, reporter
> makes it harder to do so. Right now, I have REST APIs, but I was thinking
> if in some way I could create a new MetricFetcher object and use that for
> collecting metrics instead of REST APIs.
>
> Is it possible to use MetricFetcher?
>
> Thanks a lot for the help,
> Atul
>
> On Thu, Mar 9, 2023 at 7:39 PM Mason Chen <ma...@gmail.com> wrote:
>
>> Hi all,
>>
>> Metric reporter may be useful if you only need per component level metrics
>> like jobmanager and each taskmanager since the metric reporter runs in
>> each
>> component. However, for an aggregated job level view of metrics, there is
>> no better out-of-the-box/user-facing way to get metrics than the REST API.
>> A good example in using the REST API is the k8s operator autoscaler which
>> needs to scrape metrics:
>>
>> https://github.com/apache/flink-kubernetes-operator/tree/main/flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler
>> .
>>
>> Best,
>> Mason
>>
>> On Thu, Mar 9, 2023 at 4:34 AM Hang Ruan <ru...@gmail.com> wrote:
>>
>> > Hi, Atul,
>> >
>> > I think the metric reporter[1] will be helpful for you.
>> >
>> > Best,
>> > Hang
>> >
>> > [1]
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/metric_reporters/
>> >
>> > Atul Lal <at...@gmail.com> 于2023年3月9日周四 17:49写道：
>> >
>> > > Hi everyone,
>> > >
>> > > I am trying to do some experiments with Flink. I am trying to modify
>> the
>> > > source code of Flink for this experiment, and I am starting a process
>> > > thread from JobMaster.java constructor whenever a new job is started
>> in
>> > > Flink. In this thread, I want to monitor a few metrics related to the
>> job
>> > > and make some decisions based on it.
>> > >
>> > > Is there any way to collect metrics related to a job without using
>> REST
>> > > endpoints? Because I think using REST endpoints here is pointless as
>> this
>> > > is internal code running on JobMaster. If there is no other way than
>> > using
>> > > REST endpoints, is there any easy way to serialize or parse JSON
>> > responses
>> > > from those endpoints as the response structures are already defined in
>> > the
>> > > classes.
>> > >
>> > > I would really appreciate it if someone could help me with this.
>> > >
>> > > Thank you,
>> > > Atul
>> > >
>> >
>>
>

Re: Flink metric collection

Posted by Atul Lal <at...@gmail.com>.

Hi everyone,

I also think the reporter won't work for my use case, it could work but
then I am trying to modify the placement of operators on taskmanagers
manually based on my some decisions made from metric collection, reporter
makes it harder to do so. Right now, I have REST APIs, but I was thinking
if in some way I could create a new MetricFetcher object and use that for
collecting metrics instead of REST APIs.

Is it possible to use MetricFetcher?

Thanks a lot for the help,
Atul

On Thu, Mar 9, 2023 at 7:39 PM Mason Chen <ma...@gmail.com> wrote:

> Hi all,
>
> Metric reporter may be useful if you only need per component level metrics
> like jobmanager and each taskmanager since the metric reporter runs in each
> component. However, for an aggregated job level view of metrics, there is
> no better out-of-the-box/user-facing way to get metrics than the REST API.
> A good example in using the REST API is the k8s operator autoscaler which
> needs to scrape metrics:
>
> https://github.com/apache/flink-kubernetes-operator/tree/main/flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler
> .
>
> Best,
> Mason
>
> On Thu, Mar 9, 2023 at 4:34 AM Hang Ruan <ru...@gmail.com> wrote:
>
> > Hi, Atul,
> >
> > I think the metric reporter[1] will be helpful for you.
> >
> > Best,
> > Hang
> >
> > [1]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/metric_reporters/
> >
> > Atul Lal <at...@gmail.com> 于2023年3月9日周四 17:49写道：
> >
> > > Hi everyone,
> > >
> > > I am trying to do some experiments with Flink. I am trying to modify
> the
> > > source code of Flink for this experiment, and I am starting a process
> > > thread from JobMaster.java constructor whenever a new job is started in
> > > Flink. In this thread, I want to monitor a few metrics related to the
> job
> > > and make some decisions based on it.
> > >
> > > Is there any way to collect metrics related to a job without using REST
> > > endpoints? Because I think using REST endpoints here is pointless as
> this
> > > is internal code running on JobMaster. If there is no other way than
> > using
> > > REST endpoints, is there any easy way to serialize or parse JSON
> > responses
> > > from those endpoints as the response structures are already defined in
> > the
> > > classes.
> > >
> > > I would really appreciate it if someone could help me with this.
> > >
> > > Thank you,
> > > Atul
> > >
> >
>

Re: Flink metric collection

Posted by Mason Chen <ma...@gmail.com>.

Hi all,

Metric reporter may be useful if you only need per component level metrics
like jobmanager and each taskmanager since the metric reporter runs in each
component. However, for an aggregated job level view of metrics, there is
no better out-of-the-box/user-facing way to get metrics than the REST API.
A good example in using the REST API is the k8s operator autoscaler which
needs to scrape metrics:
https://github.com/apache/flink-kubernetes-operator/tree/main/flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler
.

Best,
Mason

On Thu, Mar 9, 2023 at 4:34 AM Hang Ruan <ru...@gmail.com> wrote:

> Hi, Atul,
>
> I think the metric reporter[1] will be helpful for you.
>
> Best,
> Hang
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/metric_reporters/
>
> Atul Lal <at...@gmail.com> 于2023年3月9日周四 17:49写道：
>
> > Hi everyone,
> >
> > I am trying to do some experiments with Flink. I am trying to modify the
> > source code of Flink for this experiment, and I am starting a process
> > thread from JobMaster.java constructor whenever a new job is started in
> > Flink. In this thread, I want to monitor a few metrics related to the job
> > and make some decisions based on it.
> >
> > Is there any way to collect metrics related to a job without using REST
> > endpoints? Because I think using REST endpoints here is pointless as this
> > is internal code running on JobMaster. If there is no other way than
> using
> > REST endpoints, is there any easy way to serialize or parse JSON
> responses
> > from those endpoints as the response structures are already defined in
> the
> > classes.
> >
> > I would really appreciate it if someone could help me with this.
> >
> > Thank you,
> > Atul
> >
>

Re: Flink metric collection

Posted by Hang Ruan <ru...@gmail.com>.

Hi, Atul,

I think the metric reporter[1] will be helpful for you.

Best,
Hang

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/metric_reporters/

Atul Lal <at...@gmail.com> 于2023年3月9日周四 17:49写道：

> Hi everyone,
>
> I am trying to do some experiments with Flink. I am trying to modify the
> source code of Flink for this experiment, and I am starting a process
> thread from JobMaster.java constructor whenever a new job is started in
> Flink. In this thread, I want to monitor a few metrics related to the job
> and make some decisions based on it.
>
> Is there any way to collect metrics related to a job without using REST
> endpoints? Because I think using REST endpoints here is pointless as this
> is internal code running on JobMaster. If there is no other way than using
> REST endpoints, is there any easy way to serialize or parse JSON responses
> from those endpoints as the response structures are already defined in the
> classes.
>
> I would really appreciate it if someone could help me with this.
>
> Thank you,
> Atul
>