You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Leon Xu <lx...@attentivemobile.com> on 2022/05/21 06:24:13 UTC
Application mode deployment through API call
Hi Flink community,
I am looking to deploy my flink job through *Application Mode *in my Java
program. Ideally I'd like my java code to just call an API to achieve this.
Does flink provide a rest API to support this? I don't seem to find any
documentation or code on that.
If I need to build this on my own, is there any example I can follow?
Should I just follow how CliFrontend.java does ? (I believe the command
line calls this internally)? Or is there any other better examples?
Thanks
Leon
Re: Application mode deployment through API call
Posted by Shengkai Fang <fs...@gmail.com>.
Hi, Peter.
I am not sure whether this doc is enough or not. The doc[1] lists all the
available REST API in the Flink runtime now. You can use the RestClient[2]
to send request to the JM for later usage.
Best,
Shengkai
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/
[2]
https://github.com/apache/flink/blob/646ff2d36f40704f5dca017b8fffed78bd51b307/flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java
Peter Schrott <pe...@bluerootlabs.io> 于2022年5月24日周二 19:52写道：
> Hi Vikash,
>
> Could you be more precise about the shared libraries? Is there any
> documentation about this?
>
> Thanks, Peter
>
> On Tue, May 24, 2022 at 1:23 PM Vikash Dat <da...@gmail.com> wrote:
>
>> Similar to agent Biao, Application mode is okay if you only have a single
>> app, but when running multiple apps session mode is better for control. In
>> my experience, the CLIFrontend is not as robust as the REST API, or you
>> will end up having to rebuild a very similar Rest API. For the meta space
>> issue, have you tried adding shared libraries to the flink lib folder?
>>
>> On Mon, May 23, 2022 at 23:31 Shengkai Fang <fs...@gmail.com> wrote:
>>
>>> Hi, all.
>>>
>>> > is there any plan in the Flink community to provide an easier way of
>>> deploying Flink with application mode on YARN
>>>
>>> Yes. Jark has already opened a ticket about how to use the sql client to
>>> submit the SQL in application mode[1]. What's more, in FLIP-222 we are able
>>> to manage the jobs in SQL, which will list all submitted jobs and their web
>>> UI[2].
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-26541
>>> [2]
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-222%3A+Support+full+query+lifecycle+statements+in+SQL+client
>>>
>>>
>>>
Re: Application mode deployment through API call
Posted by Peter Schrott <pe...@bluerootlabs.io>.
Hi Vikash,
Could you be more precise about the shared libraries? Is there any
documentation about this?
Thanks, Peter
On Tue, May 24, 2022 at 1:23 PM Vikash Dat <da...@gmail.com> wrote:
> Similar to agent Biao, Application mode is okay if you only have a single
> app, but when running multiple apps session mode is better for control. In
> my experience, the CLIFrontend is not as robust as the REST API, or you
> will end up having to rebuild a very similar Rest API. For the meta space
> issue, have you tried adding shared libraries to the flink lib folder?
>
> On Mon, May 23, 2022 at 23:31 Shengkai Fang <fs...@gmail.com> wrote:
>
>> Hi, all.
>>
>> > is there any plan in the Flink community to provide an easier way of
>> deploying Flink with application mode on YARN
>>
>> Yes. Jark has already opened a ticket about how to use the sql client to
>> submit the SQL in application mode[1]. What's more, in FLIP-222 we are able
>> to manage the jobs in SQL, which will list all submitted jobs and their web
>> UI[2].
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-26541
>> [2]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-222%3A+Support+full+query+lifecycle+statements+in+SQL+client
>>
>>
>>
Re: Application mode deployment through API call
Posted by Vikash Dat <da...@gmail.com>.
Similar to agent Biao, Application mode is okay if you only have a single
app, but when running multiple apps session mode is better for control. In
my experience, the CLIFrontend is not as robust as the REST API, or you
will end up having to rebuild a very similar Rest API. For the meta space
issue, have you tried adding shared libraries to the flink lib folder?
On Mon, May 23, 2022 at 23:31 Shengkai Fang <fs...@gmail.com> wrote:
> Hi, all.
>
> > is there any plan in the Flink community to provide an easier way of
> deploying Flink with application mode on YARN
>
> Yes. Jark has already opened a ticket about how to use the sql client to
> submit the SQL in application mode[1]. What's more, in FLIP-222 we are able
> to manage the jobs in SQL, which will list all submitted jobs and their web
> UI[2].
>
> [1] https://issues.apache.org/jira/browse/FLINK-26541
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-222%3A+Support+full+query+lifecycle+statements+in+SQL+client
>
>
>
Re: Application mode deployment through API call
Posted by Shengkai Fang <fs...@gmail.com>.
Hi, all.
> is there any plan in the Flink community to provide an easier way of
deploying Flink with application mode on YARN
Yes. Jark has already opened a ticket about how to use the sql client to
submit the SQL in application mode[1]. What's more, in FLIP-222 we are able
to manage the jobs in SQL, which will list all submitted jobs and their web
UI[2].
[1] https://issues.apache.org/jira/browse/FLINK-26541
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-222%3A+Support+full+query+lifecycle+statements+in+SQL+client
Re: Application mode deployment through API call
Posted by Xiao Ma <xi...@geotab.com>.
Also, is there any plan to open restful api call on the flink operator
side, to support the application / session cluster creation, instead of the
kubectl apply command?
Thank you very much.
Best,
*Xiao Ma*
On Sun, May 22, 2022 at 7:27 PM Leon Xu <lx...@attentivemobile.com> wrote:
> Additional question, is there any plan in the Flink community to provide
> an easier way of deploying Flink with application mode on YARN? Overall I
> feel application mode would be most widely used when it comes to running
> multiple long-running flink jobs.
>
> Thanks
> Leon
>
> On Sun, May 22, 2022 at 3:49 PM Leon Xu <lx...@attentivemobile.com> wrote:
>
>> Thanks Geng, I will take a look at the YARNApplicationITCase
>> implementation.
>> And yeah I am switching from session mode to application mode, as
>> application mode can offer better resource isolation.
>> Besides that with application mode we don't need to worry about the issue
>> of Metaspace not getting GCed.
>>
>> Leon
>>
>> On Sat, May 21, 2022 at 8:11 PM Geng Biao <bi...@gmail.com> wrote:
>>
>>> Hi Leon,
>>>
>>>
>>>
>>> If you are using application mode through Flink on YARN, the codes in
>>> YARNApplicationITCase may be helpful, which has similar function calls with
>>> CliFrontend but may be more straightforward.
>>>
>>>
>>>
>>> Side notes: in my own experience of using Flink, it is not a very good
>>> idea to run a flink jar using java codes through application mode.
>>>
>>> If you are using session modes, there could be some RESTful interfaces
>>> which makes life much easier. But when using application mode, the cluster
>>> is launched after running commands like `flink run-application -t
>>> yarn-application ….` so REST services are not initially available. As a
>>> result, you have to fight with some deployment details like setting
>>> environment variables, making classloaders work correctly and so on. All
>>> details shaded by flink have to be handled by hand.
>>>
>>>
>>>
>>> Best,
>>>
>>> Biao Geng
>>>
>>>
>>>
>>> *发件人**:* Leon Xu <lx...@attentivemobile.com>
>>> *日期**:* 星期六, 2022年5月21日 下午2:24
>>> *收件人**:* user <us...@flink.apache.org>
>>> *主题**:* Application mode deployment through API call
>>>
>>> Hi Flink community,
>>>
>>>
>>>
>>> I am looking to deploy my flink job through *Application Mode *in my
>>> Java program. Ideally I'd like my java code to just call an API to achieve
>>> this. Does flink provide a rest API to support this? I don't seem to find
>>> any documentation or code on that.
>>>
>>> If I need to build this on my own, is there any example I can follow?
>>>
>>> Should I just follow how CliFrontend.java does ? (I believe the command
>>> line calls this internally)? Or is there any other better examples?
>>>
>>>
>>>
>>>
>>>
>>> Thanks
>>>
>>> Leon
>>>
>>
Re: Application mode deployment through API call
Posted by Leon Xu <lx...@attentivemobile.com>.
Additional question, is there any plan in the Flink community to provide an
easier way of deploying Flink with application mode on YARN? Overall I feel
application mode would be most widely used when it comes to running
multiple long-running flink jobs.
Thanks
Leon
On Sun, May 22, 2022 at 3:49 PM Leon Xu <lx...@attentivemobile.com> wrote:
> Thanks Geng, I will take a look at the YARNApplicationITCase
> implementation.
> And yeah I am switching from session mode to application mode, as
> application mode can offer better resource isolation.
> Besides that with application mode we don't need to worry about the issue
> of Metaspace not getting GCed.
>
> Leon
>
> On Sat, May 21, 2022 at 8:11 PM Geng Biao <bi...@gmail.com> wrote:
>
>> Hi Leon,
>>
>>
>>
>> If you are using application mode through Flink on YARN, the codes in
>> YARNApplicationITCase may be helpful, which has similar function calls with
>> CliFrontend but may be more straightforward.
>>
>>
>>
>> Side notes: in my own experience of using Flink, it is not a very good
>> idea to run a flink jar using java codes through application mode.
>>
>> If you are using session modes, there could be some RESTful interfaces
>> which makes life much easier. But when using application mode, the cluster
>> is launched after running commands like `flink run-application -t
>> yarn-application ….` so REST services are not initially available. As a
>> result, you have to fight with some deployment details like setting
>> environment variables, making classloaders work correctly and so on. All
>> details shaded by flink have to be handled by hand.
>>
>>
>>
>> Best,
>>
>> Biao Geng
>>
>>
>>
>> *发件人**:* Leon Xu <lx...@attentivemobile.com>
>> *日期**:* 星期六, 2022年5月21日 下午2:24
>> *收件人**:* user <us...@flink.apache.org>
>> *主题**:* Application mode deployment through API call
>>
>> Hi Flink community,
>>
>>
>>
>> I am looking to deploy my flink job through *Application Mode *in my
>> Java program. Ideally I'd like my java code to just call an API to achieve
>> this. Does flink provide a rest API to support this? I don't seem to find
>> any documentation or code on that.
>>
>> If I need to build this on my own, is there any example I can follow?
>>
>> Should I just follow how CliFrontend.java does ? (I believe the command
>> line calls this internally)? Or is there any other better examples?
>>
>>
>>
>>
>>
>> Thanks
>>
>> Leon
>>
>
Re: Application mode deployment through API call
Posted by Leon Xu <lx...@attentivemobile.com>.
Thanks Geng, I will take a look at the YARNApplicationITCase implementation.
And yeah I am switching from session mode to application mode, as
application mode can offer better resource isolation.
Besides that with application mode we don't need to worry about the issue
of Metaspace not getting GCed.
Leon
On Sat, May 21, 2022 at 8:11 PM Geng Biao <bi...@gmail.com> wrote:
> Hi Leon,
>
>
>
> If you are using application mode through Flink on YARN, the codes in
> YARNApplicationITCase may be helpful, which has similar function calls with
> CliFrontend but may be more straightforward.
>
>
>
> Side notes: in my own experience of using Flink, it is not a very good
> idea to run a flink jar using java codes through application mode.
>
> If you are using session modes, there could be some RESTful interfaces
> which makes life much easier. But when using application mode, the cluster
> is launched after running commands like `flink run-application -t
> yarn-application ….` so REST services are not initially available. As a
> result, you have to fight with some deployment details like setting
> environment variables, making classloaders work correctly and so on. All
> details shaded by flink have to be handled by hand.
>
>
>
> Best,
>
> Biao Geng
>
>
>
> *发件人**:* Leon Xu <lx...@attentivemobile.com>
> *日期**:* 星期六, 2022年5月21日 下午2:24
> *收件人**:* user <us...@flink.apache.org>
> *主题**:* Application mode deployment through API call
>
> Hi Flink community,
>
>
>
> I am looking to deploy my flink job through *Application Mode *in my Java
> program. Ideally I'd like my java code to just call an API to achieve this.
> Does flink provide a rest API to support this? I don't seem to find any
> documentation or code on that.
>
> If I need to build this on my own, is there any example I can follow?
>
> Should I just follow how CliFrontend.java does ? (I believe the command
> line calls this internally)? Or is there any other better examples?
>
>
>
>
>
> Thanks
>
> Leon
>
答复: Application mode deployment through API call
Posted by Geng Biao <bi...@gmail.com>.
Hi Leon,
If you are using application mode through Flink on YARN, the codes in YARNApplicationITCase may be helpful, which has similar function calls with CliFrontend but may be more straightforward.
Side notes: in my own experience of using Flink, it is not a very good idea to run a flink jar using java codes through application mode.
If you are using session modes, there could be some RESTful interfaces which makes life much easier. But when using application mode, the cluster is launched after running commands like `flink run-application -t yarn-application ….` so REST services are not initially available. As a result, you have to fight with some deployment details like setting environment variables, making classloaders work correctly and so on. All details shaded by flink have to be handled by hand.
Best,
Biao Geng
发件人: Leon Xu <lx...@attentivemobile.com>
日期: 星期六, 2022年5月21日 下午2:24
收件人: user <us...@flink.apache.org>
主题: Application mode deployment through API call
Hi Flink community,
I am looking to deploy my flink job through Application Mode in my Java program. Ideally I'd like my java code to just call an API to achieve this. Does flink provide a rest API to support this? I don't seem to find any documentation or code on that.
If I need to build this on my own, is there any example I can follow?
Should I just follow how CliFrontend.java does ? (I believe the command line calls this internally)? Or is there any other better examples?
Thanks
Leon