You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Lei Nie <ly...@gmail.com> on 2019/11/08 00:38:36 UTC
Retrieving Flink job ID/YARN Id programmatically
Hello,
I am currently executing streaming jobs via StreamExecutionEnvironment. Is
it possible to retrieve the Flink job ID/YARN ID within the context of a
job? I'd like to be able to automatically register the job such that
monitoring jobs can run (REST api requires for example job id).
Thanks
Re: Retrieving Flink job ID/YARN Id programmatically
Posted by Arvid Heise <ar...@ververica.com>.
Hi Lei,
if you use
public JobExecutionResult StreamExecutionEnvironment#execute()
You can retrieve the job id through the result.
result.getJobID()
Best,
Arvid
On Mon, Nov 25, 2019 at 3:50 AM Ana <ot...@gmail.com> wrote:
> Hi Lei,
>
> To add, you may use Hadoop Resource Manager REST APIs
> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html.
> I'm also running Flink application on YARN and use this API for that
> purpose. If you find other way or a much better solution, please let me
> know!
>
> Regards,
> Ana
>
> On Fri, Nov 22, 2019 at 10:58 AM vino yang <ya...@gmail.com> wrote:
>
>> Hi Lei,
>>
>> It would be better to use Flink's RESTful API to fetch the information of
>> the running jobs[1].
>>
>> [1]:
>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-1
>>
>> Best,
>> Vino
>>
>> Lei Nie <ly...@gmail.com> 于2019年11月22日周五 上午4:14写道：
>>
>>> I looked at the code, and
>>> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is
>>> generating a random ID unrelated to the actual ID used.
>>>
>>> Is there any way to fetch the real ID at runtime?
>>> Use case: fetch most recent checkpoint from stable storage for
>>> automated restarts. Most recent checkpoint has form
>>> ".../checkpoints/flink_app_id/chk-123"
>>>
>>> On Thu, Nov 21, 2019 at 11:28 AM Lei Nie <ly...@gmail.com> wrote:
>>> >
>>> > This does not get the correct id:
>>> > StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
>>> > eea5abc21dd8743a4090f4a3a660f9e8
>>> > Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a
>>> >
>>> >
>>> >
>>> > On Thu, Nov 7, 2019 at 6:56 PM vino yang <ya...@gmail.com>
>>> wrote:
>>> > >
>>> > > Hi Lei Nie,
>>> > >
>>> > > You can use
>>> `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the
>>> job id.
>>> > >
>>> > > Best,
>>> > > Vino
>>> > >
>>> > > Lei Nie <ly...@gmail.com> 于2019年11月8日周五 上午8:38写道：
>>> > >>
>>> > >> Hello,
>>> > >> I am currently executing streaming jobs via
>>> StreamExecutionEnvironment. Is it possible to retrieve the Flink job
>>> ID/YARN ID within the context of a job? I'd like to be able to
>>> automatically register the job such that monitoring jobs can run (REST api
>>> requires for example job id).
>>> > >>
>>> > >> Thanks
>>>
>>
Re: Retrieving Flink job ID/YARN Id programmatically
Posted by Ana <ot...@gmail.com>.
Hi Lei,
To add, you may use Hadoop Resource Manager REST APIs
https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html.
I'm also running Flink application on YARN and use this API for that
purpose. If you find other way or a much better solution, please let me
know!
Regards,
Ana
On Fri, Nov 22, 2019 at 10:58 AM vino yang <ya...@gmail.com> wrote:
> Hi Lei,
>
> It would be better to use Flink's RESTful API to fetch the information of
> the running jobs[1].
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-1
>
> Best,
> Vino
>
> Lei Nie <ly...@gmail.com> 于2019年11月22日周五 上午4:14写道：
>
>> I looked at the code, and
>> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is
>> generating a random ID unrelated to the actual ID used.
>>
>> Is there any way to fetch the real ID at runtime?
>> Use case: fetch most recent checkpoint from stable storage for
>> automated restarts. Most recent checkpoint has form
>> ".../checkpoints/flink_app_id/chk-123"
>>
>> On Thu, Nov 21, 2019 at 11:28 AM Lei Nie <ly...@gmail.com> wrote:
>> >
>> > This does not get the correct id:
>> > StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
>> > eea5abc21dd8743a4090f4a3a660f9e8
>> > Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a
>> >
>> >
>> >
>> > On Thu, Nov 7, 2019 at 6:56 PM vino yang <ya...@gmail.com> wrote:
>> > >
>> > > Hi Lei Nie,
>> > >
>> > > You can use
>> `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the
>> job id.
>> > >
>> > > Best,
>> > > Vino
>> > >
>> > > Lei Nie <ly...@gmail.com> 于2019年11月8日周五 上午8:38写道：
>> > >>
>> > >> Hello,
>> > >> I am currently executing streaming jobs via
>> StreamExecutionEnvironment. Is it possible to retrieve the Flink job
>> ID/YARN ID within the context of a job? I'd like to be able to
>> automatically register the job such that monitoring jobs can run (REST api
>> requires for example job id).
>> > >>
>> > >> Thanks
>>
>
Re: Retrieving Flink job ID/YARN Id programmatically
Posted by vino yang <ya...@gmail.com>.
Hi Lei,
It would be better to use Flink's RESTful API to fetch the information of
the running jobs[1].
[1]:
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-1
Best,
Vino
Lei Nie <ly...@gmail.com> 于2019年11月22日周五 上午4:14写道：
> I looked at the code, and
> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is
> generating a random ID unrelated to the actual ID used.
>
> Is there any way to fetch the real ID at runtime?
> Use case: fetch most recent checkpoint from stable storage for
> automated restarts. Most recent checkpoint has form
> ".../checkpoints/flink_app_id/chk-123"
>
> On Thu, Nov 21, 2019 at 11:28 AM Lei Nie <ly...@gmail.com> wrote:
> >
> > This does not get the correct id:
> > StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
> > eea5abc21dd8743a4090f4a3a660f9e8
> > Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a
> >
> >
> >
> > On Thu, Nov 7, 2019 at 6:56 PM vino yang <ya...@gmail.com> wrote:
> > >
> > > Hi Lei Nie,
> > >
> > > You can use
> `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the
> job id.
> > >
> > > Best,
> > > Vino
> > >
> > > Lei Nie <ly...@gmail.com> 于2019年11月8日周五 上午8:38写道：
> > >>
> > >> Hello,
> > >> I am currently executing streaming jobs via
> StreamExecutionEnvironment. Is it possible to retrieve the Flink job
> ID/YARN ID within the context of a job? I'd like to be able to
> automatically register the job such that monitoring jobs can run (REST api
> requires for example job id).
> > >>
> > >> Thanks
>
Re: Retrieving Flink job ID/YARN Id programmatically
Posted by Lei Nie <ly...@gmail.com>.
I looked at the code, and
StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is
generating a random ID unrelated to the actual ID used.
Is there any way to fetch the real ID at runtime?
Use case: fetch most recent checkpoint from stable storage for
automated restarts. Most recent checkpoint has form
".../checkpoints/flink_app_id/chk-123"
On Thu, Nov 21, 2019 at 11:28 AM Lei Nie <ly...@gmail.com> wrote:
>
> This does not get the correct id:
> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
> eea5abc21dd8743a4090f4a3a660f9e8
> Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a
>
>
>
> On Thu, Nov 7, 2019 at 6:56 PM vino yang <ya...@gmail.com> wrote:
> >
> > Hi Lei Nie,
> >
> > You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the job id.
> >
> > Best,
> > Vino
> >
> > Lei Nie <ly...@gmail.com> 于2019年11月8日周五 上午8:38写道：
> >>
> >> Hello,
> >> I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id).
> >>
> >> Thanks
Re: Retrieving Flink job ID/YARN Id programmatically
Posted by Lei Nie <ly...@gmail.com>.
This does not get the correct id:
StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
eea5abc21dd8743a4090f4a3a660f9e8
Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a
On Thu, Nov 7, 2019 at 6:56 PM vino yang <ya...@gmail.com> wrote:
>
> Hi Lei Nie,
>
> You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the job id.
>
> Best,
> Vino
>
> Lei Nie <ly...@gmail.com> 于2019年11月8日周五 上午8:38写道：
>>
>> Hello,
>> I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id).
>>
>> Thanks
Re: Retrieving Flink job ID/YARN Id programmatically
Posted by vino yang <ya...@gmail.com>.
Hi Lei Nie,
You can use
`StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the
job id.
Best,
Vino
Lei Nie <ly...@gmail.com> 于2019年11月8日周五 上午8:38写道：
> Hello,
> I am currently executing streaming jobs via StreamExecutionEnvironment. Is
> it possible to retrieve the Flink job ID/YARN ID within the context of a
> job? I'd like to be able to automatically register the job such that
> monitoring jobs can run (REST api requires for example job id).
>
> Thanks
>