Increasing number of task slots in the task manager

[Vishwas Siravara <vs...@gmail.com>]

Hi guys,
I get java heap space error when I have 1 GB of TM memory and 4 slots(we
have 4 cores in our lower environment) per TM , each slot has 1/4GB of
managed memory.
From the flink doc
https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html#task-slots-and-resources
I
see that slots are allocated memory statically . When I change the TM
memory to 8GB , my job works fine without any heap issues with 4 slots. So
here each slot gets around 2GB of heap . In another environment we have 60
cores. Does it make sense for me to have 60 slots in the task manager for
8GB of TM heap ? I assume that I will get heap space error since each slot
will have 8/60 GB of memory . Is my assumption correct ?

Thanks,
Vishwas

Re: Increasing number of task slots in the task manager

[Fabian Hueske <fh...@gmail.com>]

Hi Vishwas,

First of all, 8 GB for 60 cores is not a lot.
You might not be able to utilize all cores when running Flink.

However, the memory usage depends on several things.
Assuming your are using Flink for stream processing, the type of the state
backend is important. If you use the FSStateBackend, all state is stored on
the JVM heap which is certainly not want you want.
However, also the RocksDBStateBackend which stores data on the hard disk
has memory requirements. IIRC a few hundred MBs per keyed state and task.
So 60 tasks with keyed state might not possible due to the memory
requirements of RocksDB.

In addition to the state backend, you need to configure a sufficient number
of network buffers.
These also depend on the number of slots that the TaskManager process
offers.

Besides the state backend and network buffers, there are some fixed memory
costs per TaskManager process. A slot itself (without any user code
running) does not take much memory.
Note that Flink does not enforce any memory budgets for slots. Every slot
can take as much JVM memory as it wants. However, if there's not enough
memory, the whole TaskManager crashes with an OutOfMemoryError.

This is not a definitive answer to your question, but hopefully helps you
to configure the cluster for your use case.

Best,
Fabian

Am Di., 1. Okt. 2019 um 20:59 Uhr schrieb Vishwas Siravara <
vsiravara@gmail.com>:

> Hi guys,
> I get java heap space error when I have 1 GB of TM memory and 4 slots(we
> have 4 cores in our lower environment) per TM , each slot has 1/4GB of
> managed memory.
> From the flink doc
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html#task-slots-and-resources I
> see that slots are allocated memory statically . When I change the TM
> memory to 8GB , my job works fine without any heap issues with 4 slots. So
> here each slot gets around 2GB of heap . In another environment we have 60
> cores. Does it make sense for me to have 60 slots in the task manager for
> 8GB of TM heap ? I assume that I will get heap space error since each slot
> will have 8/60 GB of memory . Is my assumption correct ?
>
> Thanks,
> Vishwas
>