Flink production configuration

[Sigalit Eliazov <e....@gmail.com>]

Hello
New to flink and I am struggling with defining the correct configuration.
Currently we decided we have only one job manger.
We have 5 different pipelines.
We are not sure how many task managers needs to be defined, how many slots.

Can you please send reference or is there any calculator that assist in
such decsions?

Thanks


---------- Forwarded message ---------
מאת: Sigalit B <e....@gmail.com>
‪Date: יום ה׳, 7 באוק׳ 2021, 12:40‬
Subject: Flink production configuration
To: <us...@flink.apache.org>


Hello
New to flink and I am struggling with defining the correct configuration.
Currently we decided we have only one job manger.
We have 5 different pipelines.
We are not sure how many task managers needs to be defined, how many slots.

Can you please send reference or is there any calculator that assist in
such decsions?

Thanks

Re: Flink production configuration

[Caizhi Weng <ts...@gmail.com>]

Hi!

The number of task slots needed is the total number of parallelisms of all
the jobs running at the same time (if all the jobs are streaming jobs). The
number of task managers is the total number of task slots divide the number
of task slots per task manager (taskmanager.numberOfTaskSlots). So it
really depends on the jobs you're running.

If you're struggling with determining the number of parallelisms a job
needs, try to start with a relatively small number like 8 or 16 and see if
that meets your needs, if not then double it and try again. There are also
many optimizations which helps you increase the performance if you're using
the Table / SQL API, see
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/tuning/

Sigalit Eliazov <e....@gmail.com> 于2021年10月8日周五 下午9:44写道：

>
> Hello
> New to flink and I am struggling with defining the correct configuration.
> Currently we decided we have only one job manger.
> We have 5 different pipelines.
> We are not sure how many task managers needs to be defined, how many slots.
>
> Can you please send reference or is there any calculator that assist in
> such decsions?
>
> Thanks
>
>
> ---------- Forwarded message ---------
> מאת: Sigalit B <e....@gmail.com>
> ‪Date: יום ה׳, 7 באוק׳ 2021, 12:40‬
> Subject: Flink production configuration
> To: <us...@flink.apache.org>
>
>
> Hello
> New to flink and I am struggling with defining the correct configuration.
> Currently we decided we have only one job manger.
> We have 5 different pipelines.
> We are not sure how many task managers needs to be defined, how many slots.
>
> Can you please send reference or is there any calculator that assist in
> such decsions?
>
> Thanks
>