[jira] [Created] (FLINK-40470) Savepoint redeploy loops forever when the deployment attempt fails

["hyeon kim (Jira)" <ji...@apache.org>]

hyeon kim created FLINK-40470:
---------------------------------

             Summary: Savepoint redeploy loops forever when the deployment attempt fails
                 Key: FLINK-40470
                 URL: https://issues.apache.org/jira/browse/FLINK-40470
             Project: Flink
          Issue Type: Bug
          Components: Kubernetes Operator
    Affects Versions: kubernetes-operator-1.15.0
            Reporter: hyeon kim


When {{job.savepointRedeployNonce}} is changed, {{AbstractJobReconciler#redeployWithSavepoint}} cancels the running job (deleting the whole cluster) and redeploys from {{initialSavepointPath}}. Unlike every other deployment path (first deployment, suspend/resume, session deploy), the target spec is only recorded into the resource status *after* the deploy call succeeds:

{code:java}
cancelJob(ctx, SuspendMode.STATELESS);
...
deploy(...);                                             // <- if this throws...
ReconciliationUtils.updateStatusForDeployedSpec(...);    // <- ...the new nonce is never recorded
{code}

If {{deploy}} throws for any reason (transient API server error, or an error raised after the cluster resources were actually created — e.g. {{KubernetesClusterDescriptor#deployApplicationCluster}} fails resolving the rest endpoint of a freshly created LoadBalancer service on RBAC-restricted clusters where listing nodes is forbidden), the new nonce is never recorded in {{lastReconciledSpec}}. Every subsequent reconciliation classifies the diff as {{SAVEPOINT_REDEPLOY}} again and repeats the full delete + redeploy cycle indefinitely.

Observed in production on a managed Kubernetes platform where {{nodes}} list is forbidden for namespaced tenants: with {{kubernetes.rest-service.exposed.type: LoadBalancer}}, a savepoint redeploy looped delete/recreate for 23 minutes until manual intervention.

All other job deployment paths already call {{ReconciliationUtils.updateStatusBeforeDeploymentAttempt}} before deploying (see {{AbstractJobReconciler#reconcileSpecChange}} for the suspended->running transition and {{AbstractFlinkResourceReconciler#updateStatusBeforeFirstDeployment}}), which makes a failed attempt recover through the regular upgrading/observe flow instead of retriggering the redeploy.

*Proposed fix:* record the target spec with {{updateStatusBeforeDeploymentAttempt}} + {{statusRecorder.patchAndCacheStatus}} before the deploy call in {{redeployWithSavepoint}}, mirroring the suspend/resume path. A failed deploy then recovers on the next reconciliation by restoring from the recorded {{upgradeSavepointPath}} (which already points at {{initialSavepointPath}}), without repeating the delete + redeploy cycle.

*Open question:* with this fix, the retry recovers through the regular suspended->running restore path. That path only restores from {{upgradeSavepointPath}} when the effective upgrade mode is not STATELESS, so for a spec with {{upgradeMode: stateless}} the retry falls back to an empty-state restore instead of the requested savepoint (still strictly better than the current infinite delete/redeploy loop). Distinguishing "interrupted savepoint redeploy" from "user deliberately switched to a stateless resume" in {{reconcileSpecChange}} needs a semantic decision, so I left that corner as a follow-up — happy to address it in this ticket if there is a preferred approach.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)