You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Prashant Bhardwaj (Jira)" <ji...@apache.org> on 2026/05/18 21:22:00 UTC
[jira] [Created] (FLINK-39704) Kubernetes HA can recover a globally terminal FAILED application job after leadership revoke/reacquire
Prashant Bhardwaj created FLINK-39704:
-----------------------------------------
Summary: Kubernetes HA can recover a globally terminal FAILED application job after leadership revoke/reacquire
Key: FLINK-39704
URL: https://issues.apache.org/jira/browse/FLINK-39704
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 2.2.0, 2.4.0
Reporter: Prashant Bhardwaj
Attachments: jm-ha-reanimation-repro-current-2026-05-18.log, jm-ha-reanimation-repro-events-2026-05-18.txt, jm-ha-reanimation-repro-pod-describe-2026-05-18.txt, jm-ha-reanimation-repro-previous-2026-05-18.log
In a Kubernetes HA application cluster, a job that has already reached the globally terminal FAILED state can be recovered and restarted with the same JobID if Kubernetes leadership is revoked/reacquired immediately after the terminal transition.
Observed with apache/flink:2.2.0 and Kubernetes HA.
*Timeline from repro:*
{noformat}
20:52:51.075 Task failure after TaskManager deletion
20:52:51.119 Job e7ce38da0a5b4651ce64453d6ffaa25b switched RUNNING -> FAILING
20:52:51.122 Job e7ce38da0a5b4651ce64453d6ffaa25b switched FAILING -> FAILED
20:52:52.615 KubernetesLeaderElector observed empty leader holder
20:52:52.616 Leadership revoked
20:52:52.618 Dispatcher reported same job as terminal SUSPENDED
20:52:52.921 DefaultExecutionPlanStore released execution plan e7ce38da0a5b4651ce64453d6ffaa25b
20:52:52.926 Same job id was retrieved from KubernetesStateHandleStore
20:52:53.035 Same StreamGraph(jobId: e7ce38da0a5b4651ce64453d6ffaa25b) was recovered
20:53:11.340 Same job switched CREATED -> RUNNING
{noformat}
*Expected:*
Once a job reaches globally terminal FAILED, later leadership revocation/close should not overwrite or mask the globally terminal result as SUSPENDED. HA metadata should be cleaned up as a globally terminal job, and the same job should not be recovered.
*Actual:*
Leadership revocation closes the running JobMaster/Dispatcher path with synthetic SUSPENDED after the real FAILED result. The execution plan is released rather than permanently removed, so the same job id remains recoverable from Kubernetes HA storage and is started again.
A secondary issue is also visible in the same churn window:
DefaultLeaderElectionService receives a grant while issuedLeaderSessionID is already set and throws:
java.lang.IllegalStateException:
The leadership should have been granted while not having the leadership acquired.
This crashes the JobManager entrypoint, but the reanimation has already happened before the fatal error: the failed job was released/recovered from HA metadata.
*Reproduction outline:*
1. Run a Kubernetes HA application cluster with restart-strategy.type: none.
2. Use a persistent HA storage dir.
3. Delete the TaskManager so the job reaches FAILED.
4. Immediately after observing RUNNING -> FAILING, patch the cluster leader ConfigMap annotation holderIdentity to empty, forcing leadership loss/reacquire.
5. Observe FAILED followed by SUSPENDED/release/recovery of the same JobID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)