You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by zheng faaron <fa...@gmail.com> on 2020/09/16 02:58:03 UTC

Why setAllVerticesInSameSlotSharingGroupByDefault is set to false in batch mode

Hi All,

I find we set AllVerticesInSameSlotSharingGroupByDefault to false in flink 1.10. It will make batch job request lots of containers. I'm not sure why we set it to false directly. I try to set it to true and find the batch job can run correctly with a small amount containers. Why  don't we add a configuration to let user configure it?

Best,
Faaron Zheng

Re: Why setAllVerticesInSameSlotSharingGroupByDefault is set to false in batch mode

Posted by Zhu Zhu <re...@gmail.com>.
Hi Zheng,

To divide managed memory for operators[1], we need to consider which tasks
will
run in the same slot. In batch jobs, vertices in different regions may not
run at
the same time. If we put them in the same slot sharing group, running tasks
may run slower with less managed memory, while managed memory reserved
for tasks that are not running yet will be wasted.

More slots can be requested as a result. However, it's not necessary to add
more
containers. One slot will serve fewer tasks, this means you can decrease
the slot
size (via increase "taskmanager.numberOfTaskSlots") so that the previous
number
of containers can be enough. This is because you are still running that
many tasks
at the same time, although they are spread into more slots.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management

Thanks,
Zhu

zheng faaron <fa...@gmail.com> 于2020年9月16日周三 上午10:58写道:

> Hi All,
>
> I find we set AllVerticesInSameSlotSharingGroupByDefault to false in
> flink 1.10. It will make batch job request lots of containers. I'm not sure
> why we set it to false directly. I try to set it to true and find the batch
> job can run correctly with a small amount containers. Why  don't we add a
> configuration to let user configure it?
>
> Best,
> Faaron Zheng
>