[jira] [Created] (FLINK-40226) Standalone session cluster JM recovery enters infinite MISSING loop and fails with HTTP 409 on TM Deployment when TaskManager pods still exist

["Aniruddh J (Jira)" <ji...@apache.org>]

Aniruddh J created FLINK-40226:
----------------------------------

             Summary: Standalone session cluster JM recovery enters infinite MISSING loop and fails with HTTP 409 on TM Deployment when TaskManager pods still exist
                 Key: FLINK-40226
                 URL: https://issues.apache.org/jira/browse/FLINK-40226
             Project: Flink
          Issue Type: Bug
          Components: Kubernetes Operator
    Affects Versions: 1.14.0, 1.11.0
         Environment: Operator version  1.11.0, 1.14.0 (the ones I have tested it on)

Flink version1.20.x (standalone mode)

Cluster typeStandalone Kubernetes session cluster ({{{}FlinkDeployment{}}} with no {{{}spec.job{}}})

Kubernetes reproduced on OpenShift and vanilla Kubernetes

HA Enabled 
            Reporter: Aniruddh J


When the *JobManager (JM) Deployment* of a standalone Kubernetes session cluster is deleted externally (e.g. by an admin) while the {*}TaskManager (TM) Deployment and its pods are still running or terminating{*}, the operator enters a state where it can never successfully recover the session cluster. Two distinct but related defects combine to produce this outcome:
 # *Bug 1 — Infinite MISSING status loop on submit failure.* {{recoverSession()}} in {{SessionReconciler}} does not set the {{JobManagerDeploymentStatus}} to {{DEPLOYING}} before calling {{{}submitSessionCluster(){}}}. If the submit call throws, the controller's error-handling path resets the status back to {{{}MISSING{}}}, causing {{shouldRecoverDeployment()}} to fire on every subsequent reconcile cycle — producing an unbound retry loop with no back-off.
 # *Bug 2 — HTTP 409 AlreadyExists on TM Deployment re-creation.* {{deployClusterInternal()}} in {{KubernetesStandaloneClusterDescriptor}} uses a bare Kubernetes {{CREATE}} (HTTP POST) for the TaskManager Deployment via {{{}Fabric8FlinkStandaloneKubeClient.createTaskManagerDeployment(){}}}. If the TM Deployment already exists — which is the normal state when only the JM was deleted — the API server returns {{{}HTTP 409 Conflict (AlreadyExists){}}}, causing every recovery attempt to fail. This failure feeds back into Bug 1.

 

Together, the operator logs a continuous stream of reconciliation errors and the session cluster remains permanently unrecoverable until manual operator intervention.

 

 
h2. Steps to Reproduce
 # Deploy a standalone Kubernetes session cluster (HA mode enabled) using {{{}FlinkDeployment{}}}. Wait for {{{}jobManagerDeploymentStatus: READY{}}}.
 # Manually delete only the *JM Deployment* from the namespace:
{{kubectl delete deployment <cluster-id> -n <namespace>}}
 # Do _not_ delete the TM Deployment or its pods — allow them to continue running.
 # Observe the operator logs. The operator detects {{MISSING}} and calls {{{}recoverSession(){}}}.
 # {{deployClusterInternal()}} attempts to re-create the TM Deployment via a bare HTTP POST. Because the TM Deployment still exists, the API server returns {{{}HTTP 409 AlreadyExists{}}}.
 # The 409 exception propagates through {{submitSessionCluster()}} → {{recoverSession()}} → controller error handler.
 # The controller error handler resets the status to {{MISSING}} from the in-memory cache (which was never updated to {{{}DEPLOYING{}}}).
 # On the next reconcile cycle (15 s default), {{shouldRecoverDeployment()}} evaluates to {{true}} again, and the loop repeats indefinitely.

 
h2. Expected Behaviour
 # When a JM-recovery submit fails for any transient reason (network error, 409, throttling), the {{jobManagerDeploymentStatus}} should be advanced to {{DEPLOYING}} _before_ the submit attempt, so that the error path does not restore {{MISSING}} and re-trigger an immediate retry.
 # A re-submit when only the JM Deployment is absent and the TM Deployment still exists should succeed without a 409. The desired TM spec is unchanged, so an idempotent create-or-replace is semantically correct.
 # A failed recovery attempt should not proactively delete or disturb still-running TM pods as part of its error rollback.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)