[jira] [Created] (FLINK-33710) Autoscaler redeploys pipeline for a NOOP parallelism change

["Maximilian Michels (Jira)" <ji...@apache.org>]

Maximilian Michels created FLINK-33710:
------------------------------------------

             Summary: Autoscaler redeploys pipeline for a NOOP parallelism change
                 Key: FLINK-33710
                 URL: https://issues.apache.org/jira/browse/FLINK-33710
             Project: Flink
          Issue Type: Bug
          Components: Autoscaler, Kubernetes Operator
    Affects Versions: kubernetes-operator-1.7.0, kubernetes-operator-1.6.0
            Reporter: Maximilian Michels
            Assignee: Maximilian Michels
             Fix For: kubernetes-operator-1.8.0


The operator supports two modes to apply autoscaler changes:

# Use the internal Flink config {{pipeline.jobvertex-parallelism-overrides}} 
# Make use of Flink's Rescale API 

For (1), a string has to be generated for the Flink config with the actual overrides. This string has to be deterministic for a given map. But it is not.

Consider the following observed log:

{noformat}
  >>> Event  | Info    | SPECCHANGED     | SCALE change(s) detected (Diff: FlinkDeploymentSpec[flinkConfiguration.pipeline.jobvertex-parallelism-overrides : 92542d1280187bd464274368a5f86977:3,9f979ed859083299d29f281832cb5be0:1,84881d7bda0dc3d44026e37403420039:1,1652184ffd0522859c7840a24936847c:1 -> 9f979ed859083299d29f281832cb5be0:1,84881d7bda0dc3d44026e37403420039:1,92542d1280187bd464274368a5f86977:3,1652184ffd0522859c7840a24936847c:1]), starting reconciliation. 
{noformat}

The overrides are identical but the order is different which triggers a redeploy. This does not seem to happen often but some deterministic string generation (e.g. sorting by key) is required to prevent any NOOP updates.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)