You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Si-li Liu <un...@gmail.com> on 2023/03/21 03:54:59 UTC

How to auto scale yarn session job

I use this command to launch a flink yarn session.

yarn-session.sh -s 6 -jm 2048 -tm 4096 -nm sql-client-session -m
yarn-cluster -d

And all my flink sql job has 2 parallelism, and I found that my yarn
session can only have 3 pipelines. If my session doesn't have free slot,
submit to this session will cause exists pipeline failed. Can the yarn
session auto allocate new TM when it runs out its free slots?
-- 
Best regards

Sili Liu

Re: How to auto scale yarn session job

Posted by Si-li Liu <un...@gmail.com>.
Sorry, it seems related our own bug, after fixed the defect it can scale
automatically.

Thanks for your reply.

Weihua Hu <hu...@gmail.com> 于2023年3月21日周二 12:02写道：

> Hi,
>
> Yarn session clusters should auto allocate new task managers if slots are
> not enough.
> And the new submitting should not affect running jobs.
>
> Could you provide the failure log?
>
> Best,
> Weihua
>
>
> On Tue, Mar 21, 2023 at 11:57 AM Si-li Liu <un...@gmail.com> wrote:
>
>> I use this command to launch a flink yarn session.
>>
>> yarn-session.sh -s 6 -jm 2048 -tm 4096 -nm sql-client-session -m
>> yarn-cluster -d
>>
>> And all my flink sql job has 2 parallelism, and I found that my yarn
>> session can only have 3 pipelines. If my session doesn't have free slot,
>> submit to this session will cause exists pipeline failed. Can the yarn
>> session auto allocate new TM when it runs out its free slots?
>> --
>> Best regards
>>
>> Sili Liu
>>
>

-- 
Best regards

Sili Liu

Re: How to auto scale yarn session job

Posted by Weihua Hu <hu...@gmail.com>.
Hi,

Yarn session clusters should auto allocate new task managers if slots are
not enough.
And the new submitting should not affect running jobs.

Could you provide the failure log?

Best,
Weihua


On Tue, Mar 21, 2023 at 11:57 AM Si-li Liu <un...@gmail.com> wrote:

> I use this command to launch a flink yarn session.
>
> yarn-session.sh -s 6 -jm 2048 -tm 4096 -nm sql-client-session -m
> yarn-cluster -d
>
> And all my flink sql job has 2 parallelism, and I found that my yarn
> session can only have 3 pipelines. If my session doesn't have free slot,
> submit to this session will cause exists pipeline failed. Can the yarn
> session auto allocate new TM when it runs out its free slots?
> --
> Best regards
>
> Sili Liu
>