[jira] [Updated] (YUNIKORN-3449) [Core] Preemption falsely aborts and starves large asks by ignoring node available capacity in shortfall check

["Hedger Lai (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hedger Lai updated YUNIKORN-3449:
---------------------------------
        Parent: YUNIKORN-3186
    Issue Type: Sub-task  (was: Bug)

> [Core] Preemption falsely aborts and starves large asks by ignoring node available capacity in shortfall check
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: YUNIKORN-3449
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3449
>             Project: Apache YuniKorn
>          Issue Type: Sub-task
>          Components: core - scheduler
>    Affects Versions: 1.6.0
>            Reporter: Hedger Lai
>            Assignee: Hedger Lai
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.11.0
>
>
> h3. Problem Summary
> In {{Preemptor.TryPreemption()}}, when selecting preemption victims on a candidate node, upstream {{tryNodes()}} and {{calculateVictimsByNode()}} evaluate victims by adding their resources to {{nodeAvailable}}:
> {code:go}
> nodeCurrentAvailable.AddTo(victim.GetAllocatedResource())
> if nodeCurrentAvailable.FitIn(p.ask.GetAllocatedResource()) && index < 0 {
>     index = len(results)
> }
> {code}
> Upstream correctly stops collecting victims once {{nodeAvailable + victims >= ask}}. That is, upstream only selects victims to cover the node's residual *deficit* ({{ask - nodeAvailable}}).
> However, downstream in {{TryPreemption()}} (lines 670-678 on master), the shortfall validation compares {{victimsTotalResource}} directly against the full ask requirement ({{p.ask.GetAllocatedResource()}}):
> {code:go}
> hasShortfall := victimsTotalResource.IsEmpty()
> if !hasShortfall {
>     for k, victimVal := range victimsTotalResource.Resources {
>         if needVal, ok := p.ask.GetAllocatedResource().Resources[k]; ok && victimVal < needVal {
>             hasShortfall = true
>             break
>         }
>     }
> }
> if hasShortfall {
>     // there is shortfall, so preemption doesn't help
>     p.ask.LogAllocationFailure(common.PreemptionShortfall, true)
>     return nil, false
> }
> {code}
> Because {{victimsTotalResource}} only accumulates victim allocations without factoring in the node's existing available capacity ({{p.nodeAvailableMap[nodeID]}}), preemption is falsely aborted whenever a candidate node has partial available capacity!
> h3. Reproduction Scenario
> Consider a node with 4 vcores capacity:
> # *Node available capacity*: 2 vcores idle ({{nodeAvailable = 2}}).
> # *Running allocation*: Pod A using 2 vcores (candidate victim).
> # *Pending Ask*: Requires 4 vcores.
> * *Upstream behavior*:
> {{calculateVictimsByNode()}} starts with 2 free vcores. It selects Pod A (2 vcores). {{nodeCurrentAvailable}} becomes {{2 + 2 = 4 vcores}}, which fits the ask. Upstream returns {{nodeVictims = [Pod A]}} (2 vcores).
> * *Downstream failure*:
> {{victimsTotalResource}} = 2 vcores.
> Downstream evaluates {{victimVal (2) < needVal (4)}} -> {{hasShortfall = true}}.
> {{TryPreemption()}} aborts with {{PreemptionShortfall}}, even though preempting Pod A would free up the 4 vcores needed to schedule the ask!
> h3. Root Cause
> In {{TryPreemption()}}, the shortfall check assumes that the total resources of all selected victims alone must equal or exceed the ask's resource requirement. It completely ignores that the selected node already contributes {{p.nodeAvailableMap[nodeID]}} towards fulfilling the ask.
> h3. Proposed Solution
> When checking for preemption shortfall on the chosen node:
> # Factor in the node's existing available capacity:
> The required physical capacity from victims on the node is {{deficit = max(0, ask - nodeAvailable)}}.
> # Ensure the shortfall check verifies whether victims satisfy the node deficit (for physical node capacity) as well as any queue headroom requirements.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org