Balancing operation across machines

[Le Xu <sh...@gmail.com>]

Hello!

Just wondering if there is a way to load-balancing all executors on one
operator across the machine. I have a reduce operator that has less
parallelism level than the total number of slots in the cluster. Currently
flink tries to compact executors in an operator and schedule them to some
of the machines in the cluster. Just wondering if there is a way to make
Flink schedule these executor evenly across the machines (e.g. one executor
per machine.)?

Thanks!

Le

Re: Balancing operation across machines

[Tony Wei <to...@gmail.com>]

Hi,

I think this patch[1] is what you need and it might be merged in 1.4.0
release.
AFAIK, there is no way to change the behavior of scheduler from public API.
One work around that I could come up with is to give each TM one available
slot, make all operators in the same SlotSharingGroup and set the
parallelism as the number of TMs.
Hope this will help you.

Best,
Tony Wei

[1] https://issues.apache.org/jira/browse/FLINK-7851

2017-10-22 10:35 GMT+08:00 Le Xu <sh...@gmail.com>:

> Hello!
>
> Just wondering if there is a way to load-balancing all executors on one
> operator across the machine. I have a reduce operator that has less
> parallelism level than the total number of slots in the cluster. Currently
> flink tries to compact executors in an operator and schedule them to some
> of the machines in the cluster. Just wondering if there is a way to make
> Flink schedule these executor evenly across the machines (e.g. one executor
> per machine.)?
>
> Thanks!
>
> Le
>