You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Debasish Ghosh <gh...@gmail.com> on 2019/06/03 09:32:16 UTC
Is there any way to get the ExecutionGraph of a Job
Hello -
I am trying to build an API that can start, control and stop a Flink Job
programmatically.
When I do an executionEnv.execute() where executionEnv is an
StreamExecutionEnvironment, I get back a JobExecutionResult. I find no way
to stop the job (say based on some timeout) from a JobExecutionResult.
One class that can help me is ExecutionGraph which has APIs like stop() and
scheduleForExecution(). But how do I get an ExecutionGraph from
executionEnv.execute(). All I want is to submit the job for execution and
then be able to stop it after some timeout. Or maybe if the job fails I
would like to do some cleanups.
What is the idiomatic way to design such APIs in Flink ?
regards.
--
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh
Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg
Re: Is there any way to get the ExecutionGraph of a Job
Posted by Debasish Ghosh <gh...@gmail.com>.
Thanks a lot for the clarification.
On Tue, Jun 4, 2019 at 8:37 AM Yun Gao <yu...@aliyun.com> wrote:
> Hi Debasish,
>
> You cannot get ExecutionGraph since it resides in the JobMaster,
> which is not in the same process with Client.
>
> In my opinion, currently you may not be able to stop the job or query
> the job status using the Client API. The community is currently trying to
> enhance the Client API[1] so it will be available in the future.
>
> An alternative option may be using the REST API to monitor the status
> of the job, and canceling job by calling the undocumented REST API
> <JM_ADDR>/jobs/<job id>/yarn-cancel (May be removed in the future).
>
>
> [1]
> https://lists.apache.org/list.html?dev@flink.apache.org:lte=1M:Flink%20client%20api%20enhancement%20for%20downstream%20project
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/rest_api.html
>
>
> Best
> Yun Gao
>
> ------------------------------------------------------------------
> From:Debasish Ghosh <gh...@gmail.com>
> Send Time:2019 Jun. 3 (Mon.) 17:32
> To:user <us...@flink.apache.org>
> Subject:Is there any way to get the ExecutionGraph of a Job
>
> Hello -
>
> I am trying to build an API that can start, control and stop a Flink Job
> programmatically.
>
> When I do an executionEnv.execute() where executionEnv is an
> StreamExecutionEnvironment, I get back a JobExecutionResult. I find no way
> to stop the job (say based on some timeout) from a JobExecutionResult.
>
> One class that can help me is ExecutionGraph which has APIs like stop()
> and scheduleForExecution(). But how do I get an ExecutionGraph from
> executionEnv.execute(). All I want is to submit the job for execution and
> then be able to stop it after some timeout. Or maybe if the job fails I
> would like to do some cleanups.
>
> What is the idiomatic way to design such APIs in Flink ?
>
> regards.
> --
> Debasish Ghosh
> http://manning.com/ghosh2
> http://manning.com/ghosh
>
> Twttr: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: http://github.com/debasishg
>
>
>
--
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh
Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg
Re: Is there any way to get the ExecutionGraph of a Job
Posted by Yun Gao <yu...@aliyun.com>.
Hi Debasish,
You cannot get ExecutionGraph since it resides in the JobMaster, which is not in the same process with Client.
In my opinion, currently you may not be able to stop the job or query the job status using the Client API. The community is currently trying to enhance the Client API[1] so it will be available in the future.
An alternative option may be using the REST API to monitor the status of the job, and canceling job by calling the undocumented REST API <JM_ADDR>/jobs/<job id>/yarn-cancel (May be removed in the future).
[1] https://lists.apache.org/list.html?dev@flink.apache.org:lte=1M:Flink%20client%20api%20enhancement%20for%20downstream%20project
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/rest_api.html
Best
Yun Gao
------------------------------------------------------------------
From:Debasish Ghosh <gh...@gmail.com>
Send Time:2019 Jun. 3 (Mon.) 17:32
To:user <us...@flink.apache.org>
Subject:Is there any way to get the ExecutionGraph of a Job
Hello -
I am trying to build an API that can start, control and stop a Flink Job programmatically.
When I do an executionEnv.execute() where executionEnv is an StreamExecutionEnvironment, I get back a JobExecutionResult. I find no way to stop the job (say based on some timeout) from a JobExecutionResult.
One class that can help me is ExecutionGraph which has APIs like stop() and scheduleForExecution(). But how do I get an ExecutionGraph from executionEnv.execute(). All I want is to submit the job for execution and then be able to stop it after some timeout. Or maybe if the job fails I would like to do some cleanups.
What is the idiomatic way to design such APIs in Flink ?
regards.
--
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh
Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg