Flink parallel subtask affinity of taskmanager

[Andrew Roberts <ar...@fuze.com>]

Hello,

We run flink as a standalone cluster. When moving from flink 1.3 to 1.6, we noticed a change in the scheduling behavior. Where previously parallel subtasks of a job seemed to be round-robin allocated around our cluster, flink 1.6 appears to want to deploy as many subtasks to the same host as possible. This model is not as good for our use case, and is causing memory issues for us.

Is there any way to configure flink 1.6 to deploy in the old style? Or if we upgraded to 1.7, does that open any doors back towards the model we want?

Thanks,

Andrew
-- 
*Confidentiality Notice: The information contained in this e-mail and any

attachments may be confidential. If you are not an intended recipient, you

are hereby notified that any dissemination, distribution or copying of this

e-mail is strictly prohibited. If you have received this e-mail in error,

please notify the sender and permanently delete the e-mail and any

attachments immediately. You should not retain, copy or use this e-mail or

any attachment for any purpose, nor disclose all or any part of the

contents to any other person. Thank you.*

Re: Flink parallel subtask affinity of taskmanager

[Piotr Nowojski <pi...@ververica.com>]

Hi,

You should be able to use legacy mode for this:
https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#legacy

However note that this option will disappear in the near future and there is a JIRA ticket to address this issue:

https://issues.apache.org/jira/browse/FLINK-11815

Piotrek

> On 4 Mar 2019, at 15:00, Andrew Roberts <ar...@fuze.com> wrote:
> 
> Hello,
> 
> We run flink as a standalone cluster. When moving from flink 1.3 to 1.6, we noticed a change in the scheduling behavior. Where previously parallel subtasks of a job seemed to be round-robin allocated around our cluster, flink 1.6 appears to want to deploy as many subtasks to the same host as possible. This model is not as good for our use case, and is causing memory issues for us.
> 
> Is there any way to configure flink 1.6 to deploy in the old style? Or if we upgraded to 1.7, does that open any doors back towards the model we want?
> 
> Thanks,
> 
> Andrew
> -- 
> *Confidentiality Notice: The information contained in this e-mail and any
> 
> attachments may be confidential. If you are not an intended recipient, you
> 
> are hereby notified that any dissemination, distribution or copying of this
> 
> e-mail is strictly prohibited. If you have received this e-mail in error,
> 
> please notify the sender and permanently delete the e-mail and any
> 
> attachments immediately. You should not retain, copy or use this e-mail or
> 
> any attachment for any purpose, nor disclose all or any part of the
> 
> contents to any other person. Thank you.*