[jira] [Closed] (FLINK-31996) Chaining operators with different max parallelism prevents rescaling

["Chesnay Schepler (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-31996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chesnay Schepler closed FLINK-31996.
------------------------------------
    Fix Version/s: 1.18.0
       Resolution: Fixed

master: 58a849bcc497e17bf575e49946f106030f3e1a1a

> Chaining operators with different max parallelism prevents rescaling
> --------------------------------------------------------------------
>
>                 Key: FLINK-31996
>                 URL: https://issues.apache.org/jira/browse/FLINK-31996
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Coordination
>            Reporter: David Morávek
>            Assignee: David Morávek
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.18.0
>
>
> We might chain operators with different max parallelism together if they are set to have the same parallelism initially.
> When we decide to rescale the JobGraph vertices (using AdaptiveScheduler), we're gapped by the lowest maxParallelism of the operator chain. This is especially visible with things like CollectSink, TwoPhaseCommitSink, CDC, and a GlobalCommiter with maxParallelism set to 1.
>  
> An obvious solution would be to prevent the chaining of operators with different maxParallelism, but we need to double-check this doesn't introduce a breaking change.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)