You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Lee Parayno <le...@gmail.com> on 2023/01/20 04:31:37 UTC
Kubernetes JobManager and TaskManager minimum/maximum resources
For application mode FlinkDeployments (maybe even session mode) in Kubernetes from the Flink Kubernetes Operator what is the absolute minimum amount of CPU and RAM that is required to run the JobManager and TaskManager processes?
Some of the example deployment yaml examples have CPU set at 1 full vCPU and memory at 2GB (2048 MB). If you factor in JobManager HA, and 1 or more TaskManagers (not sure what is the bounding limit for these processes), you can be at 3 vCPU and 6 GB memory used just by the “Flink Infrastructure” not counting the Job pods.
Has anyone seen a need to have more resources dedicated to these processes for some reason? Has anyone run it leaner than this (like with 0.5 vCPU and less than 1GB memory) in production?
Comparing this to Google Cloud Platform and the Dataflow Runner, AFAIK the only resources utilized (that customers pay for) are the Job instances.
Lee Parayno
Sent from my iPhone
Re: Kubernetes JobManager and TaskManager minimum/maximum resources
Posted by Gyula Fóra <gy...@gmail.com>.
But of course the actual memory requirement will largely depend on the type
of job, statebackend , number of task slots etc
Production TM/JMs usually have much more resources allocated than 2gb/1cpu
as you never want to run out of it :)
Gyula
On Sat, 21 Jan 2023 at 11:17, Gyula Fóra <gy...@gmail.com> wrote:
> Hi!
>
> I think the examples allocate too many resources by default and we should
> reduce it in the yamls.
>
> 1gb memory and 0.5 cpu should be more than enough , we could probably get
> away with even less for example purposes.
>
> Would you have time trying this out and maybe contributing this
> improvement? :)
>
> Thanks
> Gyula
>
>
> On Fri, 20 Jan 2023 at 05:32, Lee Parayno <le...@gmail.com> wrote:
>
>> For application mode FlinkDeployments (maybe even session mode) in
>> Kubernetes from the Flink Kubernetes Operator what is the absolute minimum
>> amount of CPU and RAM that is required to run the JobManager and
>> TaskManager processes?
>>
>> Some of the example deployment yaml examples have CPU set at 1 full vCPU
>> and memory at 2GB (2048 MB). If you factor in JobManager HA, and 1 or more
>> TaskManagers (not sure what is the bounding limit for these processes), you
>> can be at 3 vCPU and 6 GB memory used just by the “Flink Infrastructure”
>> not counting the Job pods.
>>
>> Has anyone seen a need to have more resources dedicated to these
>> processes for some reason? Has anyone run it leaner than this (like with
>> 0.5 vCPU and less than 1GB memory) in production?
>>
>> Comparing this to Google Cloud Platform and the Dataflow Runner, AFAIK
>> the only resources utilized (that customers pay for) are the Job instances.
>>
>> Lee Parayno
>> Sent from my iPhone
>
>
Re: Kubernetes JobManager and TaskManager minimum/maximum resources
Posted by Gyula Fóra <gy...@gmail.com>.
Hi!
I think the examples allocate too many resources by default and we should
reduce it in the yamls.
1gb memory and 0.5 cpu should be more than enough , we could probably get
away with even less for example purposes.
Would you have time trying this out and maybe contributing this
improvement? :)
Thanks
Gyula
On Fri, 20 Jan 2023 at 05:32, Lee Parayno <le...@gmail.com> wrote:
> For application mode FlinkDeployments (maybe even session mode) in
> Kubernetes from the Flink Kubernetes Operator what is the absolute minimum
> amount of CPU and RAM that is required to run the JobManager and
> TaskManager processes?
>
> Some of the example deployment yaml examples have CPU set at 1 full vCPU
> and memory at 2GB (2048 MB). If you factor in JobManager HA, and 1 or more
> TaskManagers (not sure what is the bounding limit for these processes), you
> can be at 3 vCPU and 6 GB memory used just by the “Flink Infrastructure”
> not counting the Job pods.
>
> Has anyone seen a need to have more resources dedicated to these processes
> for some reason? Has anyone run it leaner than this (like with 0.5 vCPU
> and less than 1GB memory) in production?
>
> Comparing this to Google Cloud Platform and the Dataflow Runner, AFAIK the
> only resources utilized (that customers pay for) are the Job instances.
>
> Lee Parayno
> Sent from my iPhone