[jira] [Created] (FLINK-40330) [Kubernetes HA] Orphaned per-job leader entries accumulate in the cluster ConfigMap after interrupted cleanup or leadership loss

["Chaitanya Waikar (Jira)" <ji...@apache.org>]

Chaitanya Waikar created FLINK-40330:
----------------------------------------

             Summary: [Kubernetes HA] Orphaned per-job leader entries accumulate in the cluster ConfigMap after interrupted cleanup or leadership loss
                 Key: FLINK-40330
                 URL: https://issues.apache.org/jira/browse/FLINK-40330
             Project: Flink
          Issue Type: Bug
          Components: Deployment / Kubernetes, Runtime / Coordination
    Affects Versions: 2.1.3
         Environment: * *Flink version:* 2.1.3
 * *Flink release commit:* {{6cda56b084d5c337b36d2f8ed464bc92093b0a34}}
 * *Release commit date:* June 8, 2026
 * *Deployment type:* Kubernetes session cluster with HA enabled
 * *HA backend:* Kubernetes
 * *Observation date:* July 30–31, 2026
 * *Cluster age at observation:* approximately four months
 * *Cluster-level HA ConfigMap size:* approximately 290 KB
 * *Per-job leader entries:* 255
 * *Execution-plan entries:* 204
 * *Kubernetes leader transitions:* 193
            Reporter: Chaitanya Waikar


h2. Description

We observed a Kubernetes HA cluster in which old per-job leader-information entries accumulated in the cluster-level HA ConfigMap.

The affected ConfigMap contains *255* keys with this format:
Text
 
org.apache.flink.k8s.leader.job-<job-id>
Many of the corresponding JobIDs are no longer active, but their leader-information entries remain in the ConfigMap. The same ConfigMap also stores other HA metadata, so this accumulation causes it to grow continuously toward Kubernetes’ {*}1 MiB ConfigMap limit{*}.

Source inspection shows that Flink normally attempts to remove a job’s leader entry when the corresponding {{JobManagerRunner}} closes. However, this deletion is conditional on the process still holding Kubernetes leadership. In addition, the job-level HA cleanup method deletes the job-specific ConfigMap but does not perform a fallback deletion or reconciliation of the job’s leader entry in the cluster ConfigMap.

 

This means that {*}abnormal termination{*}, {*}leadership loss during cleanup{*}, or a failed ConfigMap update may leave an orphaned leader entry without a later reconciliation mechanism.
h2. Observed behavior

The cluster ConfigMap contains entries such as:
YAML
 
org.apache.flink.k8s.leader.job-04836a890e3bdbb2eb3c44e0332725cb:
  48cd444c-3df1-4963-91aa-0df8f0bcfdb5,pekko.tcp://flink@<pod-address>:6123/user/rpc/jobmanager_55


org.apache.flink.k8s.leader.job-048ea419dbc869d34139e6f277b0d1f0:
  92edfece-ac04-4300-91e3-40fbb57b6875,pekko.tcp://flink@<pod-address>:6123/user/rpc/jobmanager_185{{}}
{{}}
{{{}{}}}The corresponding JobIDs are no longer returned by the Dispatcher’s active-job list and are not expected to be recovered.
The entries remain in the ConfigMap after the jobs have disappeared from the active cluster state.

The cluster regularly stops and resubmits jobs from savepoints, generating new JobIDs. The cluster has also experienced JobManager process failures, including an OOMKill. At present, we cannot attribute every stale entry to a single cleanup path, but the production ConfigMap demonstrates that orphaned entries persist across multiple JobManager lifetimes.
h2. Expected behavior

After a job reaches a globally terminal state and its resources are cleaned up, the following leader-information key should eventually be absent:
 
org.apache.flink.k8s.leader.job-<terminated-job-id>
 

If immediate removal is not possible because the process has lost leadership or terminated unexpectedly, a later active leader should *reconcile and remove entries* for jobs that are no longer active or recoverable.

At minimum, stale per-job leader entries should not accumulate indefinitely in the shared cluster ConfigMap.
h2. Actual behavior

Some per-job leader entries survive job termination or JobManager failover and remain indefinitely in the shared cluster ConfigMap.

There does not appear to be a general reconciliation mechanism that compares persisted per-job leader keys with the set of active or recoverable jobs.

The ConfigMap therefore grows over time.

 

*Source Code Analysis*

We could not find a mechanism that periodically or during recovery:
 # reads all {{org.apache.flink.k8s.leader.job-*}} entries;
 # compares them with active or recoverable jobs;
 # removes entries for jobs that no longer exist.

The Kubernetes leader-election driver can parse all leader entries, but only reconciles information for currently registered contenders. An orphaned component ID has no registered contender that would cause its data to be corrected or removed.

 
h2. Suspected failure scenarios

The following scenarios appear capable of leaving orphaned keys.
 # Abrupt JobManager process termination (due to OOMKill)
 # Leadership loss during cleanup

h2.  
h2. Operational impact

The cluster-level ConfigMap is used for:
 * Kubernetes leader-election metadata;
 * Dispatcher, ResourceManager and REST endpoint leader information;
 * per-job JobMaster leader information;
 * execution-plan state handles;
 * potentially other cluster-level HA metadata.

Kubernetes limits ConfigMap data to approximately {*}1 MiB{*}.

Unbounded stale-entry accumulation can therefore eventually prevent new ConfigMap entries or updates from succeeding. Once the ConfigMap approaches that limit, operations that increase its size may fail with HTTP 422 validation errors.

This can affect:
 * new JobMaster leader publication;
 * execution-plan persistence;
 * other HA metadata writes;
 * Dispatcher operation;
 * recovery and failover reliability.

This is similar in impact to oversized HA ConfigMap incidents such as [FLINK-31135|https://issues.apache.org/jira/browse/FLINK-31135], although this report concerns a potentially different accumulation mechanism.

We are not claiming that the current ConfigMap size will immediately cause a cluster-wide outage. The concern is that the growth is unbounded and eventually reaches a hard Kubernetes limit.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)