[jira] [Assigned] (FLINK-40470) Savepoint redeploy loops forever when the deployment attempt fails

["Robert Metzger (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Metzger reassigned FLINK-40470:
--------------------------------------

    Assignee: hyeon kim

> Savepoint redeploy loops forever when the deployment attempt fails
> ------------------------------------------------------------------
>
>                 Key: FLINK-40470
>                 URL: https://issues.apache.org/jira/browse/FLINK-40470
>             Project: Flink
>          Issue Type: Bug
>          Components: Kubernetes Operator
>    Affects Versions: kubernetes-operator-1.15.0
>            Reporter: hyeon kim
>            Assignee: hyeon kim
>            Priority: Major
>
> When {{job.savepointRedeployNonce}} is changed, {{AbstractJobReconciler#redeployWithSavepoint}} cancels the running job (deleting the whole cluster) and redeploys from {{initialSavepointPath}}. Unlike every other deployment path (first deployment, suspend/resume, session deploy), the target spec is only recorded into the resource status *after* the deploy call succeeds:
> {code:java}
> cancelJob(ctx, SuspendMode.STATELESS);
> ...
> deploy(...);                                             // <- if this throws...
> ReconciliationUtils.updateStatusForDeployedSpec(...);    // <- ...the new nonce is never recorded
> {code}
> If {{deploy}} throws for any reason (transient API server error, or an error raised after the cluster resources were actually created — e.g. {{KubernetesClusterDescriptor#deployApplicationCluster}} fails resolving the rest endpoint of a freshly created LoadBalancer service on RBAC-restricted clusters where listing nodes is forbidden), the new nonce is never recorded in {{lastReconciledSpec}}. Every subsequent reconciliation classifies the diff as {{SAVEPOINT_REDEPLOY}} again and repeats the full delete + redeploy cycle indefinitely.
> Observed in production on a managed Kubernetes platform where {{nodes}} list is forbidden for namespaced tenants: with {{kubernetes.rest-service.exposed.type: LoadBalancer}}, a savepoint redeploy looped delete/recreate for 23 minutes until manual intervention.
> All other job deployment paths already call {{ReconciliationUtils.updateStatusBeforeDeploymentAttempt}} before deploying (see {{AbstractJobReconciler#reconcileSpecChange}} for the suspended->running transition and {{AbstractFlinkResourceReconciler#updateStatusBeforeFirstDeployment}}), which makes a failed attempt recover through the regular upgrading/observe flow instead of retriggering the redeploy.
> *Proposed fix:* record the target spec with {{updateStatusBeforeDeploymentAttempt}} + {{statusRecorder.patchAndCacheStatus}} before the deploy call in {{redeployWithSavepoint}}, mirroring the suspend/resume path. A failed deploy then recovers on the next reconciliation by restoring from the recorded {{upgradeSavepointPath}} (which already points at {{initialSavepointPath}}), without repeating the delete + redeploy cycle.
> *Open question:* with this fix, the retry recovers through the regular suspended->running restore path. That path only restores from {{upgradeSavepointPath}} when the effective upgrade mode is not STATELESS, so for a spec with {{upgradeMode: stateless}} the retry falls back to an empty-state restore instead of the requested savepoint (still strictly better than the current infinite delete/redeploy loop). Distinguishing "interrupted savepoint redeploy" from "user deliberately switched to a stateless resume" in {{reconcileSpecChange}} needs a semantic decision, so I left that corner as a follow-up — happy to address it in this ticket if there is a preferred approach.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)