[jira] [Commented] (FLINK-38323) Global Scale Factor Mode

["Gyula Fora (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-38323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18018091#comment-18018091 ] 

Gyula Fora commented on FLINK-38323:
------------------------------------

This goes a little against the design philosophy of the algorithm. Increasing the parallelism of all the vertices will be a much larger load increase than for a single vertex even if it doesn't add a new task manager so it may cause some issues.

Also we have to consider slot sharing groups here. So you may want to take the max within the slot sharing group not globally.

> Global Scale Factor Mode
> ------------------------
>
>                 Key: FLINK-38323
>                 URL: https://issues.apache.org/jira/browse/FLINK-38323
>             Project: Flink
>          Issue Type: New Feature
>          Components: Autoscaler
>    Affects Versions: 1.12.1
>            Reporter: Salva
>            Priority: Major
>         Attachments: TMs - CPU Usage.png
>
>
> As a user, I'd like to have an option to restrict to scaling decision altering the number of replicas. This could be achieved by using a global scale factor instead of individually adjusting the parallelism for the different operators. This would make the autoscaler work as when one adds/removes replicas by hand.
> Some users have reported running an internal version of the autoscaler with the one major difference being that they let the whole pipeline (except for the sources/sinks) to have the same parallelism to avoid uneven task distribution. The change should be relatively simple, i.e., just run the algorithm per vertex, and take the max of them.
> See the question/suggestion first posted [here|[https://lists.apache.org/thread/rf9fpr6cjnjjwl6rqhtbhfn380doghyl]].
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)