Parallelism under reactive scaling with slot sharing groups

[Allen Wang <al...@gmail.com>]

Hello,

Our job has operators of source -> sink -> global committer. We have
created two slot sharing groups, one for source and sink and one for global
committer. The global committer has specified max parallelism of 1. No max
parallelism set with the source/sink while there is a system level default
parallelism set to be the total number of task slots.

After we enabled reactive mode scaling, we found that the parallelism for
source/sink calculated by the adaptive scheduler is only half of the total
task slots. For example, if we have 8 slots, the derived parallelism is 4
meaning source/sink would use only 4 slots, and the global committer uses 1
slot. So we have 3 idle slots which is a waste.

Any ideas how we can avoid those idle slots in this setting?

Thanks,
Allen

Re: Parallelism under reactive scaling with slot sharing groups

[Allen Wang <al...@gmail.com>]

We were using Flink 1.16. I did some research and it looks like it is fixed
<https://issues.apache.org/jira/browse/FLINK-30895> in Flink 1.17.



On Thu, Jul 27, 2023 at 9:43 PM Weihua Hu <hu...@gmail.com> wrote:

> Hi,
>
> Which Flink version are you using? I haven't reproduced this issue by
> using the master branch.
>
>
> Best,
> Weihua
>
>
> On Tue, Jul 25, 2023 at 2:56 AM Allen Wang <al...@gmail.com> wrote:
>
>> Hello,
>>
>> Our job has operators of source -> sink -> global committer. We have
>> created two slot sharing groups, one for source and sink and one for global
>> committer. The global committer has specified max parallelism of 1. No max
>> parallelism set with the source/sink while there is a system level default
>> parallelism set to be the total number of task slots.
>>
>> After we enabled reactive mode scaling, we found that the parallelism for
>> source/sink calculated by the adaptive scheduler is only half of the total
>> task slots. For example, if we have 8 slots, the derived parallelism is 4
>> meaning source/sink would use only 4 slots, and the global committer uses 1
>> slot. So we have 3 idle slots which is a waste.
>>
>> Any ideas how we can avoid those idle slots in this setting?
>>
>> Thanks,
>> Allen
>>
>>
>>
>>

Re: Parallelism under reactive scaling with slot sharing groups

[Weihua Hu <hu...@gmail.com>]

Hi,

Which Flink version are you using? I haven't reproduced this issue by using
the master branch.


Best,
Weihua


On Tue, Jul 25, 2023 at 2:56 AM Allen Wang <al...@gmail.com> wrote:

> Hello,
>
> Our job has operators of source -> sink -> global committer. We have
> created two slot sharing groups, one for source and sink and one for global
> committer. The global committer has specified max parallelism of 1. No max
> parallelism set with the source/sink while there is a system level default
> parallelism set to be the total number of task slots.
>
> After we enabled reactive mode scaling, we found that the parallelism for
> source/sink calculated by the adaptive scheduler is only half of the total
> task slots. For example, if we have 8 slots, the derived parallelism is 4
> meaning source/sink would use only 4 slots, and the global committer uses 1
> slot. So we have 3 idle slots which is a waste.
>
> Any ideas how we can avoid those idle slots in this setting?
>
> Thanks,
> Allen
>
>
>
>