[jira] [Commented] (YUNIKORN-3374) Phantom allocation when a node with an assumed pod is removed and re-added

["Wilfred Spiegelenburg (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YUNIKORN-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105185#comment-18105185 ] 

Wilfred Spiegelenburg commented on YUNIKORN-3374:
-------------------------------------------------

This change by itself without proper handling of the bind failures would leave the system in a similar broken state. The cal to {{Assume()}} triggers an update of the task state which also triggers the async bind of the pod to the node. Without the proper handling of the volume and node binding errors that should happen in this case the task will be left in an allocated state in the k8shim and in a similar state on the core side.
This change cleans up the cache on the k8shim side it does not handle any of the changes on the core side that should be triggered. For that we need YUNIKORN-2884 as that reverts the core side and moves it back to scheduling via the call to {{rescheduleOnBindFailure()}}

> Phantom allocation when a node with an assumed pod is removed and re-added
> --------------------------------------------------------------------------
>
>                 Key: YUNIKORN-3374
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3374
>             Project: Apache YuniKorn
>          Issue Type: Bug
>          Components: shim - kubernetes
>            Reporter: Dale Richardson
>            Assignee: Dale Richardson
>            Priority: Major
>              Labels: pull-request-available
>
> When a node is removed while a pod is assumed on it (allocation made, bind not yet confirmed), the shim scheduler cache orphans the pod with the shim-invented {{Spec.NodeName}} still set. If the node re-appears (kubelet restart/re-registration, transient node object deletion and rejoin), the orphan adoption path re-assigns the never-bound pod to it and re-registers the task, and task recovery reports an existing allocation to the core for a pod that never ran: a phantom allocation that books queue and node capacity to nothing.
> Found by the invariant property fuzzer of YUNIKORN-3373. Sibling of YUNIKORN-3355 (same phantom-allocation class, different trigger: a node flap instead of a failed bind).
> *Mechanism* (all verified in code on master):
> # {{SchedulerCache.removeNode}} ({{scheduler_cache.go}} ~198) iterates the NodeInfo's pods and orphans ALL of them: it deletes the {{assignedPods}} and {{assumedPods}} entries and puts the pod into {{orphanedPods}}. The assumed/bound distinction is erased, and an assumed pod's cached object keeps the {{Spec.NodeName}} stamped at assume time -- a value the shim invented, which the apiserver's pod object never had.
> # {{updateNode}} on node (re-)add (~168) adopts any orphan whose {{Spec.NodeName}} matches the new node via {{updatePod}}: the phantom assignment is back.
> # {{Context.addNode}} ({{context.go}} ~211-218) registers each adopted pod via {{updateYuniKornPod}} -> {{ensureAppAndTaskCreated}}. The original task is typically gone by then (the core released the node's allocations when the node was removed), so {{addTask}} re-creates it with metadata carrying the stale-NodeName pod, and the recovery path reports the pod as an existing allocation on the re-added node.
> # The shim-cache half self-heals on the next informer update for the pod (it arrives unassigned), but the core-side allocation does not; it persists until released.
> Note the asymmetry that makes this a bug: for a genuinely BOUND pod, orphan-and-adopt is correct -- its node name is real and the pod actually runs on the flapping node. Only the assumed-never-bound case adopts a fiction.
> *Reachability*: requires a node delete + re-add around the assume-to-bind window of any pod. Node object flaps are routine at scale (kubelet re-registration, cloud node churn, node delete + rejoin), and the assume-to-bind window is always open somewhere on a busy cluster. In the YUNIKORN-3373 fuzzer, 50/50 seeds reach the state and observe the re-adoption (26,442 step observations over 250k steps; the fuzzer flaps nodes far more often than production, so this measures reachability, not frequency). A targeted regression test (assume, RemoveNode, UpdateNode, assert unassigned) fails on the unfixed code.
> *Fix* (PR to follow): in {{removeNode}}, treat assumed pods the way the YUNIKORN-3355 fix treats them in {{forgetPod}}: revert instead of orphan -- drop the assumed/assigned state and re-store the pod unassigned (node name cleared on a copy). The pod is then simply rescheduled, which is the correct outcome as its target node is gone. Bound pods keep the existing orphan/adopt behaviour unchanged. The YUNIKORN-3373 fuzzer's invariants then guard the fix permanently.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org