Re: Not enough free slots to run the job

[David Dreyfus <dd...@gmail.com>]

Hello,

I know this is an older thread, but ...

If some slots are left empty it doesn't necessarily mean that machine
resources are wasted. Some managed memory might be unavailable, but CPU,
heap memory, network, and disk are shared across slots. To the extent there
are multiple operators executing within a slot, multiple threads are
executing consuming those resources. It's not clear what the actual
performance degradation would be, if any. Correct?

David 



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Not enough free slots to run the job

[Fabian Hueske <fh...@gmail.com>]

Hi David,

that's correct. A TM is a single process. A slot is just a virtual concept
in the TM process and runs its program slice in multiple threads.
Besides managed memory (which is split into chunks add assigned to slots)
all other resources (CPU, heap, network, disk) are not isolated and free to
use for all threads.

The DataSet API operators operate almost exclusively on managed memory.
Heap memory is only used for in-flight data and not to store larger amounts.
So having unused slots leaves some of the configured memory unused.

Best, Fabian

2017-10-27 3:44 GMT+02:00 David Dreyfus <dd...@gmail.com>:

> Hello,
>
> I know this is an older thread, but ...
>
> If some slots are left empty it doesn't necessarily mean that machine
> resources are wasted. Some managed memory might be unavailable, but CPU,
> heap memory, network, and disk are shared across slots. To the extent there
> are multiple operators executing within a slot, multiple threads are
> executing consuming those resources. It's not clear what the actual
> performance degradation would be, if any. Correct?
>
> David
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/
>