[jira] [Created] (FLINK-30773) Allow rescaling of jobs based on per-vertex parallelism overrides

["Maximilian Michels (Jira)" <ji...@apache.org>]

Maximilian Michels created FLINK-30773:
------------------------------------------

             Summary: Allow rescaling of jobs based on per-vertex parallelism overrides
                 Key: FLINK-30773
                 URL: https://issues.apache.org/jira/browse/FLINK-30773
             Project: Flink
          Issue Type: New Feature
          Components: Autoscaler, Runtime / Coordination, Runtime / REST
            Reporter: Maximilian Michels
            Assignee: Maximilian Michels


FLINK-29501 introduced a way to rescale jobs via a user-provided parallelism overrides map. This feature is already used today by the Autoscaler of the Flink Kubernetes operator. However, it requires a full restart of the Flink job and only supports the application deployment mode.

In a K8s environment, this is inefficient because all pods for a deployment will be surrendered. Upon restart, they have to be re-acquired. In addition to being slow, this can also lead to situations where resource constraints prevent a restart from executing properly.

Ideally, we would would want the following to happen on receiving a rescale request:
 # Rescale API receives a request with a parallelism overrides map (vertexId => parallelism) for a jobId
 # Compute the number of required task slots using the overrides and the current JobGraph
 ## If the total number of task slots for the cluster is less than the required number of task slots of the rescale, acquire the missing task slots. Otherwise, do nothing
 ## Wait for new task slots to become available
 ## Abort rescale request on timeout
 # Redeploy the JobGraph / Tasks with the updated parallelisms
 # Surrender any unused task slots in case of scaling down

 

There is an existing "Rescaling" API which is currently disabled. We have to evaluate whether reusing it makes sense.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)