[jira] [Created] (YUNIKORN-3355) Failed bind leaves a stale node assignment in the shim cache

["Dale Richardson (Jira)" <ji...@apache.org>]

Dale Richardson created YUNIKORN-3355:
-----------------------------------------

             Summary: Failed bind leaves a stale node assignment in the shim cache
                 Key: YUNIKORN-3355
                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3355
             Project: Apache YuniKorn
          Issue Type: Bug
          Components: core - scheduler, shim - kubernetes
            Reporter: Dale Richardson


If Pod bind calls fail while allocations are in flight (API server outage, restart, or sustained errors under load), YuniKorn can permanently strand the affected pods: they remain {{Pending}} in Kubernetes forever, while the core's queue books them as {*}allocated{*}. The scheduler logs "No outstanding apps found" and never retries. Pod metadata updates do not resurrect them; only deleting/recreating the pods or restarting the scheduler recovers. No WARN/ERROR is logged for the inconsistency.

This is very likely the root cause of YUNIKORN-3128 ("Yunikorn ignores pending pods after apiserver errors"): we reproduced the identical end state deterministically and traced the mechanism through the code.
h2. Reproduction (deterministic when timed right; reproduced on 1.9.0 and master a82e4f92)
 # kind cluster (K8s v1.36.1), 500 KWOK fake nodes, YuniKorn standard mode (Helm defaults, admission controller disabled).
 # Create 3,000 pods with {{schedulerName: yunikorn}} at high rate (~2,000 pods/s, client-go at concurrency 16).
 # ~3s in, while Binding POSTs are in flight, {{kill -9}} the kube-apiserver; kubelet restarts it ~10s later.
 # Wait for recovery and observe.

The kill must land while binds are in flight — killing before binding starts recovers cleanly. Any failure mode producing a burst of failed binds should trigger it (YUNIKORN-3128 saw it from apiserver errors under load, with no full outage).
h2. Observed end state (master build, 242 of 3,000 pods affected)
 * 242 pods {{Pending}} with empty {{{}spec.nodeName{}}}, permanently (observed >12 min, no recovery).
 * Queue REST API: {{root.default}} shows {{allocatedResource pods: 3000}} — the queue books all 3,000 {*}including the 242 never bound{*}; the applications list is empty (app completed and was removed).
 * Scheduler log: exactly 242 {{added existing allocation}} lines (1:1 with stuck pods), each with a {{{}targetNode{}}}, ~30-45s after the outage; later {{{}No outstanding apps found for a while{}}}.
 * New pods (a different app) schedule normally — the damage is silent quota leakage plus the stranded pods.

h2. Root cause (verified against master a82e4f92)

Four-step interaction between the shim cache's assumed-pod handling and the core's existing-allocation semantics:
 # *Assume stamps NodeName on the cached copy.* {{Context.AssumePod}} deep-copies the pod, sets {{{}assumedPod.Spec.NodeName = node{}}}, and stores it in the scheduler cache ({{{}pkg/cache/context.go:869-871{}}}). The cache records the assignment in {{{}assignedPods{}}}.
 # *ForgetPod does not undo the assignment.* On release after the failed bind, {{Context.ForgetPod}} fetches the pod from the cache — the assumed copy with NodeName set — and passes it to {{SchedulerCache.forgetPod}} ({{{}pkg/cache/context.go:879-884{}}}). {{forgetPod}} calls {{updatePod(pod)}} with that copy and deletes only the {{assumedPods}} marker ({{{}pkg/cache/external/scheduler_cache.go:494-506{}}}). Inside {{updatePod}} the copy still satisfies {{{}IsAssignedPod{}}}, so it is re-added to the node and re-recorded in {{{}assignedPods{}}}. Net effect: after a failed bind the cache permanently believes the pod is assigned.
 # *The stale assignment is copied onto the real pod object.* On the next informer event for the pod (watch reconnect / relist after the outage), {{updatePod}} sees the incoming pod has {{Spec.NodeName == ""}} while {{assignedPods}} has an entry, and executes {{pod.Spec.NodeName = nodeName}} ("use existing assignment", {{{}pkg/cache/external/scheduler_cache.go:~355-358{}}}). Note this also *mutates the shared informer-cache object* (pods are stored by reference via {{{}utils.Convert2Pod{}}}), poisoning every other consumer of that object.
 # *Task re-creation turns the stale assignment into a phantom placed allocation.* The bind-failure storm failed the tasks and completed/removed the application, so the informer event re-creates the app and task ({{{}ensureAppAndTaskCreated{}}}); {{Task.updateAllocation}} builds the allocation with {{NodeID: task.pod.Spec.NodeName}} ({{{}pkg/cache/task.go:319-322{}}}, {{{}pkg/common/si_helper.go:134{}}}) — now non-empty due to step 3. The core's {{PartitionContext.UpdateAllocation}} treats any allocation with a NodeID as already placed ("handling existing allocation" / "added existing allocation", {{{}pkg/scheduler/partition.go:~1240-1259{}}}) and books it against the queue without ever scheduling or binding it.

The pod is now invisible to the scheduler (no pending ask), unbound in Kubernetes, and counted against queue quota. When the re-created app later completes, the phantom allocations remain in queue accounting with no owning application.
h2. Impact
 * Pods stranded Pending indefinitely after any burst of bind failures; silent — scheduler appears healthy.
 * Queue quota silently leaked by phantom allocations; in quota-tight clusters this can starve a queue completely.
 * Shared informer objects mutated (step 3) — undefined behaviour for all other informer consumers.
 * Amplified by scheduler throughput: the faster the core allocates, the more binds are in flight during a blip, the larger the phantom set (relevant to the YUNIKORN-3350 throughput work).

h2. Suggested fixes (layered; any of the first three breaks the chain)
 # {{forgetPod}} must actually revert the assignment: clear {{Spec.NodeName}} on (a copy of) the pod before re-inserting, or restore the informer version, and remove the {{assignedPods}} entry.
 # {{updatePod}} must not copy a stale assignment onto an unassigned incoming pod when that assignment came from an assumed (never confirmed bound) pod — and must never mutate the incoming informer-owned object (copy-on-write instead).
 # The shim should only report {{NodeID}} to the core from the *informer's* view of {{spec.nodeName}} (ground truth of what is actually bound), never from cache-internal assumed state.
 # Defence in depth: bounded retry of transient bind failures (connection refused / timeout / 429) before failing the task; an assumed-pod expiry analogous to kube-scheduler's; a core-side invariant warning when queue allocated resources exist with no owning application.

h2. Environment

kind v0.32 / K8s v1.36.1 single control plane, 500 KWOK fake nodes; reproduced on YuniKorn 1.9.0 (Helm) and a master build (k8shim a82e4f92). Load generator and repro script available on request.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org