[jira] [Created] (YUNIKORN-3481) Orphan allocation on node when a release races with allocation

["TingYao Huang (Jira)" <ji...@apache.org>]

TingYao Huang created YUNIKORN-3481:
---------------------------------------

             Summary: Orphan allocation on node when a release races with allocation
                 Key: YUNIKORN-3481
                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3481
             Project: Apache YuniKorn
          Issue Type: Bug
          Components: core - scheduler
            Reporter: TingYao Huang


{{Application.tryNode()}} adds the allocation to the node and the app, but its node ID is only set later in {{PartitionContext.allocate()}}, outside the app lock. A release or app removal handled in between sees an empty node ID and skips the node, so the allocation stays on it permanently.

*Impact:* node usage (and queue usage on the release path) leaks until scheduler restart; health check reports "Orphan allocation on node".

*Cause:* regression from YUNIKORN-2459 (1.6.0), which dropped {{NewAllocation(node.NodeID, ask)}}.

*Repro:* add to {{pkg/scheduler}} and run {{go test ./pkg/scheduler -run TestOrphanAllocationOnRelease}}. Fails on master, passes with the fix below.
{code:go}
func TestOrphanAllocationOnRelease(t *testing.T) {
	setupUGM()
	partition, err := newBasePartition()
	assert.NilError(t, err)
	defer partition.userGroupCache.Stop()
	node := setupNode(t, nodeID1, partition, resources.NewResourceFromMap(map[string]resources.Quantity{"vcore": 10}))
	app := newApplication(appID1, "default", defQueue)
	assert.NilError(t, partition.AddApplication(app))
	assert.NilError(t, app.AddAllocationAsk(newAllocationAsk(allocKey, appID1, resources.NewResourceFromMap(map[string]resources.Quantity{"vcore": 1}))))

	// step 1 of tryAllocate(): allocation is on the node and the app, node ID not set yet
	result := partition.root.TryAllocate(partition.GetNodeIterator, partition.GetFullNodeIterator, partition.GetNode, false)
	// RM event handler: pod deleted
	partition.removeAllocation(&si.AllocationRelease{PartitionName: "test", ApplicationID: appID1, AllocationKey: allocKey, TerminationType: si.TerminationType_STOPPED_BY_RM})
	// step 2 of tryAllocate()
	partition.allocate(result)

	assert.Equal(t, len(checkNodeAllocations(node, partition)), 0, "orphan allocation left on node")
}
{code}
Also seen in CI: user_group_limit e2e, [job|https://github.com/apache/yunikorn-k8shim/actions/runs/36432084339/job/108962971909]. Not flagged by {{-race}}: every access is locked.

*Fix:* call {{ask.SetNodeID(node.NodeID)}} in {{tryNode}} while holding the app lock; add the test above.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org