[jira] [Created] (FLINK-41052) Helm chart: recreate the operator pod on ConfigMap changes and support annotations on the operator Deployment

["Gerk Elznik (Jira)" <ji...@apache.org>]

Gerk Elznik created FLINK-41052:
-----------------------------------

             Summary: Helm chart: recreate the operator pod on ConfigMap changes and support annotations on the operator Deployment
                 Key: FLINK-41052
                 URL: https://issues.apache.org/jira/browse/FLINK-41052
             Project: Flink
          Issue Type: Improvement
          Components: Kubernetes Operator
            Reporter: Gerk Elznik


h2. Problem

The chart renders the operator configuration into the {{flink-operator-config}} ConfigMap, but the pod template does not change when that ConfigMap does, so a configuration-only change applied through Helm, Argo CD or any other tool that renders the chart never restarts the operator.  The dynamic config watcher covers the reconcile-time options, but the options documented under the [System Configuration Startup|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#system-startup] and [Advanced System Configuration Startup|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#advanced-startup] sections, plus the [metrics reporters|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#system-metrics-configuration] and the logging config (unless {{monitorInterval}} is enabled), {*}keep the value read at startup until someone restarts the pod by hand{*}. The [Dynamic Operator Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#dynamic-operator-configuration] section already says so, so the gap is known; the chart just offers no way to close it.

Measured current behavior in a kind cluster on main (chart 1.17-SNAPSHOT, webhook enabled): a {{helm upgrade}} that adds {{kubernetes.operator.leader-election.enabled: true}} updates the ConfigMap, the pod is not recreated, and the operator keeps logging {{{}Operator leader election is disabled{}}}.

The usual Helm answer is a checksum of the rendered ConfigMap on the pod template, so the Deployment rolls exactly when the configuration changes, whichever tool applies the chart.  Many charts hardcode that.

Here my current thinking is to *make it opt-in* because the dynamic config feature is still there to avoid restarts and a recreation under the default {{Recreate}} strategy is a short outage during which the admission webhook ({{{}failurePolicy: Fail{}}}) rejects Flink resource changes, and a configuration the operator cannot load would crash-loop the new pod, whereas today the watcher logs the error and keeps the old configuration, so we'll want users to be well aware of what they're signing up for when they opt-in.
h2. Proposal
 # A new value {{operatorPod.restartOnConfigChange}} (default {{{}false{}}}) that adds {{checksum/config: <sha256 of the rendered controller/configmap.yaml>}} to the pod template annotations, only when the chart manages that ConfigMap ({{{}defaultConfiguration.create{}}}). Direct edits of the live ConfigMap are not tracked and keep going through the dynamic reload. Default off keeps existing installs byte-identical. If you would rather have the conventional always-on behavior, that is a one-line change and I am happy to make it. One property worth knowing: the ConfigMap render carries the chart labels, so a chart version bump also rolls the pod once; that usually coincides with an image change anyway, and hashing only the {{data}} block would avoid it if you prefer.
 # As a smaller companion, a new value {{operatorDeployment.annotations}} rendered on the Deployment object's own metadata. {{operatorPod.labels}} already lands on both the pod template and the Deployment (FLINK-35402), but annotations only reach the pod template. Deployment-level annotations are what controllers such as Stakater Reloader or Argo CD read, and they are also where ownership and alert-routing metadata usually lives. It is also the complement of the first item for people who bring their own ConfigMap: the chart cannot checksum what it does not render, so they need an external trigger, and Reloader reads its annotation from the Deployment. For clarity, FLINK-29391 and FLINK-37421 ask for labels on the JobManager and TaskManager Deployments that the operator creates; this ticket is only about the operator's own Deployment from the chart.

Both parts are additive and change nothing with default values.
h2. Verification so far (prototype, not yet pushed)

helm-unittest (71 pass on Helm 4.3, also with the 0.8.1 plugin version CI pins), ct lint (Helm 3.16 image, which also renders the toggle with the same checksum as Helm 4), and a kind cluster with the webhook enabled through cert-manager, operator image built from the branch:
 * toggle off, helm adds leader election to the config: ConfigMap updated, pod uid unchanged, operator still reports leader election disabled
 * toggle on, same values: pod recreated, {{{}Operator leader election is enabled{}}}, checksum present
 * toggle on, config changed again: pod recreated, checksum changed
 * toggle on, identical values re-applied: no pod churn, checksum unchanged
 * toggle on with {{{}defaultConfiguration.create: false{}}}: no checksum annotation rendered
 * {{operatorDeployment.annotations}} appears on the Deployment and not on the pod template
 * two running FlinkDeployments (v2_2) stayed RUNNING across the operator recreation, JobManager pods untouched, no operator errors after the restart

h2. Docs

Rows in the Helm installation table for both values (EN and ZH) and a short note in the [Dynamic Operator Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#dynamic-operator-configuration] section: what the toggle tracks, that live edits and bring-your-own ConfigMaps are not covered, and the outage trade-off under {{{}Recreate{}}}.
 #  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)