Running a flink job with Yarn per-job mode from application code.

[Kamil ty <ka...@gmail.com>]

Hello all,

I'm looking for a way to submit a Yarn job from another flink jobs
application code. I can see that you can access a cluster and submit jobs
with a RestClusterClient, but it seems a Yarn per-job mode is not supported
with it.

Any suggestions would be appreciated.

Best Regards
Kamil

Re: Running a flink job with Yarn per-job mode from application code.

[Kamil ty <ka...@gmail.com>]

Sorry for that. Will remember to add the mailing list in responses.

The REST API approach will probably be sufficient. One more question
regarding this. Is it possible to get the address/port of the rest api
endpoint from the job? I see that when deploying a job to yarn the
rest.address and rest.port keys are not set inside the configuration
obtained with env.getConfiguration().toString();.

On Thu, 9 Dec 2021 at 03:29, Caizhi Weng <ts...@gmail.com> wrote:

> Hi!
>
> Please make sure to always reply to the user mailing list so that everyone
> can see the discussion.
>
> You can't get the execution environment for an already running job but if
> you want to operate on that job you can try to get its JobClient instead.
> However this is somewhat complicated to get with Java code. If you're
> interested see ClusterClientJobClientAdapter and its usage.
>
> I would also recommend using REST API to operate on an existing job. See
> [1] for the APIs.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/
>
> Kamil ty <ka...@gmail.com> 于2021年12月8日周三 21:15写道：
>
>> Thank you. I guess I will try this solution. One thing I'm unsure about
>> is if I will be able to get the execution environment in a function from an
>> already running job. Will get back with an response if this works.
>>
>> Best regards
>> Kamil
>>
>> On Wed, 8 Dec 2021, 03:20 Caizhi Weng, <ts...@gmail.com> wrote:
>>
>>> Hi!
>>>
>>> So you would like to submit a yarn job with Java code, not using
>>> /bin/flink run?
>>>
>>> If that is the case, you'll need to set 'execution.target' config option
>>> to 'yarn-per-job'. Set this in the configuration of ExecutionEnvironment
>>> and execute the job with Flink API as normal.
>>>
>>> Kamil ty <ka...@gmail.com> 于2021年12月7日周二 19:16写道：
>>>
>>>> Hello all,
>>>>
>>>> I'm looking for a way to submit a Yarn job from another flink jobs
>>>> application code. I can see that you can access a cluster and submit jobs
>>>> with a RestClusterClient, but it seems a Yarn per-job mode is not supported
>>>> with it.
>>>>
>>>> Any suggestions would be appreciated.
>>>>
>>>> Best Regards
>>>> Kamil
>>>>
>>>

Re: Running a flink job with Yarn per-job mode from application code.

[Caizhi Weng <ts...@gmail.com>]

Hi!

Please make sure to always reply to the user mailing list so that everyone
can see the discussion.

You can't get the execution environment for an already running job but if
you want to operate on that job you can try to get its JobClient instead.
However this is somewhat complicated to get with Java code. If you're
interested see ClusterClientJobClientAdapter and its usage.

I would also recommend using REST API to operate on an existing job. See
[1] for the APIs.

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/

Kamil ty <ka...@gmail.com> 于2021年12月8日周三 21:15写道：

> Thank you. I guess I will try this solution. One thing I'm unsure about is
> if I will be able to get the execution environment in a function from an
> already running job. Will get back with an response if this works.
>
> Best regards
> Kamil
>
> On Wed, 8 Dec 2021, 03:20 Caizhi Weng, <ts...@gmail.com> wrote:
>
>> Hi!
>>
>> So you would like to submit a yarn job with Java code, not using
>> /bin/flink run?
>>
>> If that is the case, you'll need to set 'execution.target' config option
>> to 'yarn-per-job'. Set this in the configuration of ExecutionEnvironment
>> and execute the job with Flink API as normal.
>>
>> Kamil ty <ka...@gmail.com> 于2021年12月7日周二 19:16写道：
>>
>>> Hello all,
>>>
>>> I'm looking for a way to submit a Yarn job from another flink jobs
>>> application code. I can see that you can access a cluster and submit jobs
>>> with a RestClusterClient, but it seems a Yarn per-job mode is not supported
>>> with it.
>>>
>>> Any suggestions would be appreciated.
>>>
>>> Best Regards
>>> Kamil
>>>
>>

Re: Running a flink job with Yarn per-job mode from application code.

[Caizhi Weng <ts...@gmail.com>]

Hi!

So you would like to submit a yarn job with Java code, not using /bin/flink
run?

If that is the case, you'll need to set 'execution.target' config option to
'yarn-per-job'. Set this in the configuration of ExecutionEnvironment and
execute the job with Flink API as normal.

Kamil ty <ka...@gmail.com> 于2021年12月7日周二 19:16写道：

> Hello all,
>
> I'm looking for a way to submit a Yarn job from another flink jobs
> application code. I can see that you can access a cluster and submit jobs
> with a RestClusterClient, but it seems a Yarn per-job mode is not supported
> with it.
>
> Any suggestions would be appreciated.
>
> Best Regards
> Kamil
>