You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yunikorn.apache.org by "Hedger Lai (Jira)" <ji...@apache.org> on 2026/09/11 10:19:00 UTC

[jira] [Created] (YUNIKORN-3450) [Core] False preemption shortfall aborts queue preemption and starves large tasks despite sufficient guaranteed quota

Hedger Lai created YUNIKORN-3450:
------------------------------------

             Summary: [Core] False preemption shortfall aborts queue preemption and starves large tasks despite sufficient guaranteed quota
                 Key: YUNIKORN-3450
                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3450
             Project: Apache YuniKorn
          Issue Type: Bug
    Affects Versions: 1.6.0
            Reporter: Hedger Lai
            Assignee: Hedger Lai


h2. Relationship to YUNIKORN-3449
This issue is a companion/follow-up to YUNIKORN-3449:
* *YUNIKORN-3449* fixes the *Node physical capacity* dimension (where the node's existing available capacity was ignored during shortfall verification).
* *This issue* fixes the *Queue quota* dimension (where the ask queue's existing remaining guaranteed headroom is ignored during shortfall verification).
Together, these two issues resolve the false preemption shortfall defects in {{TryPreemption()}}.

h2. 1. The Symptom
In {{Preemptor.TryPreemption()}}, preemption for large tasks *falsely aborts with PreemptionShortfall and starves indefinitely*, even when preempting available victims has *already freed enough quota to bring the ask queue under its Guaranteed limit*.

h2. 2. Root Cause
In {{pkg/scheduler/objects/preemption.go}}:
{code:go}
for k, victimVal := range victimsTotalResource.Resources {
    if needVal, ok := p.ask.GetAllocatedResource().Resources[k]; ok && victimVal < needVal {
        hasShortfall = true
        break
    }
}
{code}
This check crudely demands {{victimVal >= needVal}} (victims alone must equal the entire ask demand).
It *completely ignores the ask queue's existing remaining guaranteed headroom*.

*Real-World Example (Pure extraVictims Scenario):*
* The target node is completely idle and has sufficient physical space (no node victims needed). Preemption is triggered purely to satisfy queue limits via {{calculateAdditionalVictims()}}.
* Parent Queue Max = 10. Currently used = 5. *Remaining Headroom = 5*.
* A large Ask needs 10.
* *Net Queue Deficit* = {{5 (used) + 10 (ask) - 10 (max) = 5}}.
* Upstream correctly preempts 1 victim (5 vcores), freeing enough quota so {{0 (used) + 10 (ask) = 10 <= 10}}.
* But downstream checks {{victimVal (5) < needVal (10)}}, falsely triggers {{PreemptionShortfall}}, and *aborts preemption!*

h2. 3. Deterministic Reproduction Unit Test
This bug reproduces deterministically in {{preemption_test.go}} (0.01s):

{code:go}
func TestTryPreemption_QueueResidualShortfallDeficit(t *testing.T) {
	appQueueMapping := NewAppQueueMapping()
	node1 := newNode(nodeID1, map[string]resources.Quantity{"first": 30})
	node2 := newNode(nodeID2, map[string]resources.Quantity{"first": 30})
	iterator := getNodeIteratorFn(node1, node2)
	rootQ, err := createRootQueue(map[string]string{"first": "60"})
	assert.NilError(t, err)

	// Parent Max = 10
	parentQ, err := createManagedQueueGuaranteed(rootQ, "parent", true, map[string]string{"first": "10"}, nil, appQueueMapping)
	assert.NilError(t, err)
	_, err = createManagedQueueGuaranteed(parentQ, "child1", false, nil, nil, appQueueMapping)
	assert.NilError(t, err)
	childQ2, err := createManagedQueueGuaranteed(parentQ, "child2", false, nil, map[string]string{"first": "20"}, appQueueMapping)
	assert.NilError(t, err)
	childQ3, err := createManagedQueueGuaranteed(parentQ, "child3", false, nil, nil, appQueueMapping)
	assert.NilError(t, err)

	// Victim alloc4 uses 5 on node2. Parent allocated = 5. Headroom = 10 - 5 = 5.
	app3 := newApplication(appID3, "default", "root.parent.child3")
	app3.SetQueue(childQ3)
	childQ3.AddApplication(app3)
	appQueueMapping.AddAppQueueMapping(app3.ApplicationID, childQ3)
	ask4 := newAllocationAsk("alloc4", appID3, resources.NewResourceFromMap(map[string]resources.Quantity{"first": 5}))
	assert.NilError(t, app3.AddAllocationAsk(ask4))
	alloc4 := newAllocationWithKey("alloc4", appID3, nodeID2, resources.NewResourceFromMap(map[string]resources.Quantity{"first": 5}))
	app3.AddAllocation(alloc4)
	assert.Check(t, node2.TryAddAllocation(alloc4), "node2 alloc4 failed")
	assert.NilError(t, childQ3.TryIncAllocatedResource(ask4.GetAllocatedResource()))

	// Preemptor ask needs 10. Deficit is 5. Preempting alloc4 (5) satisfies queue quota completely (0 + 10 <= 10).
	app2, ask3, err := creatApp2(childQ2, map[string]resources.Quantity{"first": 10}, "alloc3", appQueueMapping)
	assert.NilError(t, err)

	headRoom := resources.NewResourceFromMap(map[string]resources.Quantity{"first": 20})
	preemptor := NewPreemptor(app2, headRoom, 30*time.Second, ask3, iterator(), false)

	feasibleNodes := map[string]int{nodeID2: 1}
	plugin := mock.NewPreemptionPredicatePlugin(nil, feasibleNodes, false, false)
	plugins.RegisterSchedulerPlugin(plugin)
	defer plugins.UnregisterSchedulerPlugins()

	result, ok := preemptor.TryPreemption()
	assert.Assert(t, ok, "preemption should succeed: queue deficit is 5, victim is 5")
	assert.Assert(t, result != nil, "expected non-nil result")
	assert.Equal(t, "alloc3", result.Request.GetAllocationKey())
}
{code}

h2. 4. Terminal Failure Output (Evidence)
{code:text}
=== RUN   TestTryPreemption_QueueResidualShortfallDeficit
    preemption_test.go:2567: assertion failed: ok is false: preemption should succeed: queue deficit is 5, victim is 5
--- FAIL: TestTryPreemption_QueueResidualShortfallDeficit (0.01s)
FAIL
{code}

h2. 5. Proposed Solution
Update the queue preemption shortfall check to evaluate the net residual deficit:
Verify whether {{remainingGuaranteed + victimsTotalResource >= ask}}, rather than requiring {{victimsTotalResource >= ask}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org