[jira] [Updated] (FLINK-40380) Document the runtime configuration and stop jobs from overriding operator config through it

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated FLINK-40380:
-----------------------------------
    Labels: pull-request-available  (was: )

> Document the runtime configuration and stop jobs from overriding operator config through it
> -------------------------------------------------------------------------------------------
>
>                 Key: FLINK-40380
>                 URL: https://issues.apache.org/jira/browse/FLINK-40380
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kubernetes Operator
>            Reporter: Dennis-Mircea Ciupitu
>            Priority: Major
>              Labels: pull-request-available
>
> h1. Summary
> FLINK-35746 added runtime configuration observation, where the operator reads a job's effective settings from the Flink REST API and layers them over the observed configuration. Two follow-ups are needed: the operator documentation still describes the previous configuration model, and the global job parameters ingested from the REST response are not filtered.
> h1. Documentation is out of date
> The {{Internals -> Startup -> Configuration}} section still describes the model as it was before FLINK-35746, and three statements there are now wrong.
> h2. The configuration count and the Observe row
> The section opens with "Three configurations matter throughout the operator" and lists Default, Deploy and Observe. There is now a fourth source, the runtime configuration, fetched from the cluster and cached separately from the derived configurations.
> The Observe row describes that configuration as derived purely from the last reconciled spec. {{FlinkResourceContext#getObserveConfig}} now layers the runtime configuration on top, so REST-observed values win over the spec.
> h2. The warning is now false
> The section closes with:
> {quote}
> None of these is the running configuration. The operator derives its view from the spec and its own defaults, while the cluster can pick up settings the operator never sees: a config.yaml baked into the image, environment overrides, or properties the job sets programmatically. The configuration a pipeline actually runs with can therefore differ from everything the operator tracks.
> {quote}
> Reading back settings that the job sets programmatically is exactly what FLINK-35746 implemented, so this tells the reader the operator cannot do the thing it now does. A narrower version of the caution is still true and worth keeping, because the mapping covers only a subset of settings and only while the job runs.
> h2. The mechanism is undocumented
> Nothing describes when the runtime configuration is fetched (once per job by the job status observer, skipped for globally terminal jobs), how it is cached (per resource and job id, under the existing config cache limits), which settings it covers, or what happens when a fetch fails.
> h1. Global job parameters are ingested unfiltered
> {{FlinkRuntimeConfigurationUtils#mapJobConfiguration}} copies the job's global parameters wholesale:
> {code:java}
> jobConfig.putAll(execInfo.getGlobalJobParameters());
> {code}
> Every other value produced by that class is an explicit, curated {{ConfigOption}}. This one is unbounded and set by the job, and the result is written over the observed configuration through {{observeConfig::setString}}.
> A job that declares a global parameter matching an operator key, for example under {{kubernetes.operator.}} or {{job.autoscaler.}}, therefore changes how the operator manages that resource, with no indication in the logs. Keys in those namespaces have no legitimate reason to arrive from a job's global parameters, so they should be dropped.
> h1. Expected outcome
> The Configuration section describes the runtime configuration as a first-class source, states that it overrides the observed configuration, and replaces the warning with an accurate account of what the operator can and cannot observe.
> Global job parameters in the operator and autoscaler namespaces are ignored rather than applied, and the drop is logged so it is diagnosable.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)