You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Dale Richardson (Jira)" <ji...@apache.org> on 2026/08/16 14:17:00 UTC
[jira] [Created] (YUNIKORN-3374) Phantom allocation when a node with an assumed pod is removed and re-added
Dale Richardson created YUNIKORN-3374:
-----------------------------------------
Summary: Phantom allocation when a node with an assumed pod is removed and re-added
Key: YUNIKORN-3374
URL: https://issues.apache.org/jira/browse/YUNIKORN-3374
Project: Apache YuniKorn
Issue Type: Bug
Components: shim - kubernetes
Reporter: Dale Richardson
When a node is removed while a pod is assumed on it (allocation made, bind not yet confirmed), the shim scheduler cache orphans the pod with the shim-invented {{Spec.NodeName}} still set. If the node re-appears (kubelet restart/re-registration, transient node object deletion and rejoin), the orphan adoption path re-assigns the never-bound pod to it and re-registers the task, and task recovery reports an existing allocation to the core for a pod that never ran: a phantom allocation that books queue and node capacity to nothing.
Found by the invariant property fuzzer of YUNIKORN-3373. Sibling of YUNIKORN-3355 (same phantom-allocation class, different trigger: a node flap instead of a failed bind).
*Mechanism* (all verified in code on master):
# {{SchedulerCache.removeNode}} ({{scheduler_cache.go}} ~198) iterates the NodeInfo's pods and orphans ALL of them: it deletes the {{assignedPods}} and {{assumedPods}} entries and puts the pod into {{orphanedPods}}. The assumed/bound distinction is erased, and an assumed pod's cached object keeps the {{Spec.NodeName}} stamped at assume time -- a value the shim invented, which the apiserver's pod object never had.
# {{updateNode}} on node (re-)add (~168) adopts any orphan whose {{Spec.NodeName}} matches the new node via {{updatePod}}: the phantom assignment is back.
# {{Context.addNode}} ({{context.go}} ~211-218) registers each adopted pod via {{updateYuniKornPod}} -> {{ensureAppAndTaskCreated}}. The original task is typically gone by then (the core released the node's allocations when the node was removed), so {{addTask}} re-creates it with metadata carrying the stale-NodeName pod, and the recovery path reports the pod as an existing allocation on the re-added node.
# The shim-cache half self-heals on the next informer update for the pod (it arrives unassigned), but the core-side allocation does not; it persists until released.
Note the asymmetry that makes this a bug: for a genuinely BOUND pod, orphan-and-adopt is correct -- its node name is real and the pod actually runs on the flapping node. Only the assumed-never-bound case adopts a fiction.
*Reachability*: requires a node delete + re-add around the assume-to-bind window of any pod. Node object flaps are routine at scale (kubelet re-registration, cloud node churn, node delete + rejoin), and the assume-to-bind window is always open somewhere on a busy cluster. In the YUNIKORN-3373 fuzzer, 50/50 seeds reach the state and observe the re-adoption (26,442 step observations over 250k steps; the fuzzer flaps nodes far more often than production, so this measures reachability, not frequency). A targeted regression test (assume, RemoveNode, UpdateNode, assert unassigned) fails on the unfixed code.
*Fix* (PR to follow): in {{removeNode}}, treat assumed pods the way the YUNIKORN-3355 fix treats them in {{forgetPod}}: revert instead of orphan -- drop the assumed/assigned state and re-store the pod unassigned (node name cleared on a copy). The pod is then simply rescheduled, which is the correct outcome as its target node is gone. Bound pods keep the existing orphan/adopt behaviour unchanged. The YUNIKORN-3373 fuzzer's invariants then guard the fix permanently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org