Support for multiple slots per task manager in Flink 1.5

[Pawel Bartoszek <pa...@gmail.com>]

Hi,


According to the Flink 1.5 release docs multiple slots per task manager are
"not fully supported yet". Can you provide more information about what are
the risks of running more than one slot per tm?
We are running Flink on EMR on YARN. Previously we run 4 task task managers
with 8 slot each now we are running 32 with 1 slot each. However, since we
upgraded to Flink 1.5 we started seeing timeouts taking a connection from
AWS S3 SDK pool etc.

Thanks.
Pawel

Re: Support for multiple slots per task manager in Flink 1.5

[罗齐 <lu...@bytedance.com>]

Hi Pawel,

Flink 1.5 supports multiple sets per TM. “Not fully supported yet” refers to an issue [1] when requesting TM from SlotManager, which will allocate more TM than actual (but idle TMs will be released later).

So I suggest you to check further in your timeout logs to identify other issues. You can also set the slots per TM to 1 to verify this.

[1] https://issues.apache.org/jira/browse/FLINK-9455?jql=text%20~%20%22slot%22

> On Dec 5, 2018, at 7:41 PM, Pawel Bartoszek <pa...@gmail.com> wrote:
> 
> Hi,
> 
> 
> According to the Flink 1.5 release docs multiple slots per task manager are "not fully supported yet". Can you provide more information about what are the risks of running more than one slot per tm?
> We are running Flink on EMR on YARN. Previously we run 4 task task managers with 8 slot each now we are running 32 with 1 slot each. However, since we upgraded to Flink 1.5 we started seeing timeouts taking a connection from AWS S3 SDK pool etc. 
> 
> Thanks.
> Pawel