[jira] [Closed] (FLINK-40531) K8s Operator Autoscaler Parallelism overrides serialized in standard-YAML format break the deployed job

["Gyula Fora (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gyula Fora closed FLINK-40531.
------------------------------
    Resolution: Duplicate

Already fixed

> K8s Operator Autoscaler Parallelism overrides serialized in standard-YAML format break the deployed job
> -------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-40531
>                 URL: https://issues.apache.org/jira/browse/FLINK-40531
>             Project: Flink
>          Issue Type: Bug
>          Components: Autoscaler, Kubernetes Operator
>            Reporter: Juliusz Nadberezny
>            Priority: Major
>              Labels: pull-request-available
>
> {{Affects unreleased 1.17.0 version}}
> {{}}
> Flink can write map-valued config settings in two styles: the old {{a:1,b:2}} and the newer {{{}{a: '1', b: '2'}{}}}. Every Flink version can read the old style; only newer parsers can read the new one.
> {{}}
> The autoscaler lets Flink choose the style automatically, and that choice depends on which config file the operator itself was started with. So if the operator runs with {{{}config.yaml{}}}, it writes parallelism overrides in the new style into every FlinkDeployment it scales. Any JobManager still reading a legacy {{flink-conf.yaml}} then fails to start, because it cannot parse the value. The operator itself reports no problem — only the job breaks.
> {{KubernetesScalingRealizer}} and the autoscaler state stores serialize the parallelism overrides map with {{{}ConfigurationUtils.convertValue(overrides, String.class){}}}. The two-arg overload picks its format from {{{}GlobalConfiguration.isStandardYaml(){}}}, which is derived from the config file the _operator's own JVM_ loaded.
> If the operator runs with {{config.yaml}} (standard YAML) instead of {{{}flink-conf.yaml{}}}, it writes {{pipeline.jobvertex-parallelism-overrides: \{a: '1', b: '2'}}} into every FlinkDeployment it scales. A JobManager loading a legacy {{flink-conf.yaml}} cannot parse that and fails on startup with {{{}Could not split string. Illegal quoting at position: 5{}}}. The operator logs nothing — only the deployed job breaks.
> The legacy {{a:1,b:2}} form is parseable by all Flink versions, including 2.x (which reads both forms even though it serializes the new one). The flow form is not. So serialization should be pinned to the portable format rather than inherited from ambient operator state.
> Made worse by a self-inflicted trap: {{getOverrideString}} reuses the previous string verbatim when the override map is unchanged, to avoid a no-op redeploy. That pins a broken string in place permanently, so an affected job cannot recover on its own.
> *Steps to reproduce*
>  # Deploy the operator with {{defaultConfiguration.config.yaml}} instead of {{{}flink-conf.yaml{}}}.
>  # Let the autoscaler scale a {{v1_20}} FlinkDeployment.
>  # Inspect {{spec.flinkConfiguration}} — the override string is {{{...}}} and the JobManager crash-loops.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)