Change in sub-task id assignment from 1.9 to 1.10?

[Ken Krugler <kk...@transpac.com>]

Hi all,

Was there any change in how sub-tasks get allocated to TMs, from Flink 1.9 to 1.10?

Specifically for consecutively numbered sub-tasks (e.g. 0, 1, 2) did it become more or less likely that they’d be allocated to the same Task Manager?

Asking because a workflow that ran fine in 1.9 now has a “hot” TM that’s having trouble keeping up with a Kafka topic.

The most promising explanation is that now there are three sub-tasks on the same TM that are reading from that topic, versus previously they’d be scattered across multiple TMs.

But I don’t see significant changes in this area post 1.8

Thanks,

— Ken

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

Re: Change in sub-task id assignment from 1.9 to 1.10?

[Zhu Zhu <re...@gmail.com>]

Hi Ken,

There were no such changes in my mind.
And in Flink there was no designed logic to scatter subtasks of the same
operator into different taskmanagers.

One workaround to solve your problem could be to increase the parallelism
of
your source vertex to be no smaller than no other operator so that each
slot can contain a source task. With config cluster.evenly-spread-out-slots
set to true, slots can be evenly distributed in all available taskmanagers
in most cases.

Thanks,
Zhu Zhu

Ken Krugler <kk...@transpac.com> 于2020年8月7日周五 上午5:28写道：

> Hi all,
>
> Was there any change in how sub-tasks get allocated to TMs, from Flink 1.9
> to 1.10?
>
> Specifically for consecutively numbered sub-tasks (e.g. 0, 1, 2) did it
> become more or less likely that they’d be allocated to the same Task
> Manager?
>
> Asking because a workflow that ran fine in 1.9 now has a “hot” TM that’s
> having trouble keeping up with a Kafka topic.
>
> The most promising explanation is that now there are three sub-tasks on
> the same TM that are reading from that topic, versus previously they’d be
> scattered across multiple TMs.
>
> But I don’t see significant changes in this area post 1.8
>
> Thanks,
>
> — Ken
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
>