[jira] [Created] (YUNIKORN-3509) Rolling back a placeholder after a bind failure corrupts application accounting

["Dian-Xuan Yang (Jira)" <ji...@apache.org>]

Dian-Xuan Yang created YUNIKORN-3509:
----------------------------------------

             Summary: Rolling back a placeholder after a bind failure corrupts application accounting
                 Key: YUNIKORN-3509
                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3509
             Project: Apache YuniKorn
          Issue Type: Bug
          Components: core - scheduler
            Reporter: Dian-Xuan Yang
            Assignee: Dian-Xuan Yang


When the shim cannot bind a pod after all retries, {{Task.rescheduleOnBindFailure}} sends a {{SCHEDULING_FAILED_ON_RM}} release. Unlike the AssumePod failure path, which fails a placeholder task instead, this path does not check for placeholders. The core's {{PartitionContext.rollbackAllocation}} then treats a placeholder like a real allocation:
* {{Application.RollbackAllocation}} subtracts it from {{allocatedResource}}, where it was never added, and leaves {{allocatedPlaceholder}} untouched.
* The partition calls {{updateAllocationCount(-1)}} but never {{decPhAllocationCount}}.
* When the placeholder is allocated again, {{allocatedPlaceholder}} grows a second time.

Queue and node usage stay correct; the application's used/placeholder split does not.

*Impact:* (gang of two 1-vcore placeholders, one rollback, then normal replacement and release)
* REST: {{usedResource}} goes negative ({{vcore: -1}}), and {{placeholderUsedResource}} is inflated and keeps {{vcore: 1}} after every placeholder is gone. The partition placeholder count stays off by one.
* If the rollback lands before the last placeholder is allocated, the app moves to Running with a placeholder still missing.
* The app enters Completing while a real allocation is still running and never leaves it: the completing timer sees a non-zero {{allocatedPlaceholder}}, finds no placeholder allocation to release, clears the timer and never sends {{CompleteApplication}}. The application is never removed from the partition.

*Cause:* YUNIKORN-3316 (core) and YUNIKORN-2884 (k8shim), both in 1.10.0.

*Repro:* add to {{pkg/scheduler}} and run {{go test ./pkg/scheduler -run TestPlaceholderRollbackAccounting}}. All three checks fail on master.
{code:go}
func TestPlaceholderRollbackAccounting(t *testing.T) {
	setupUGM()
	partition, err := newBasePartition()
	assert.NilError(t, err)
	defer partition.userGroupCache.Stop()
	setupNode(t, nodeID1, partition, resources.NewResourceFromMap(map[string]resources.Quantity{"vcore": 10}))
	res := resources.NewResourceFromMap(map[string]resources.Quantity{"vcore": 1})
	app := newApplicationTG(appID1, "default", defQueue, resources.Multiply(res, 2))
	assert.NilError(t, partition.AddApplication(app))
	assert.NilError(t, app.AddAllocationAsk(newAllocationAskTG(phID, appID1, taskGroup, res, true)))
	assert.NilError(t, app.AddAllocationAsk(newAllocationAskTG(phID2, appID1, taskGroup, res, true)))

	result := partition.tryAllocate()
	assert.Assert(t, result != nil && result.Request != nil)
	// shim: Bind of the placeholder pod failed after all retries
	partition.removeAllocation(&si.AllocationRelease{PartitionName: "test", ApplicationID: appID1, AllocationKey: result.Request.GetAllocationKey(), TerminationType: si.TerminationType_SCHEDULING_FAILED_ON_RM})

	assert.Check(t, resources.IsZero(app.GetAllocatedResource()), "app allocated: %v", app.GetAllocatedResource())
	assert.Check(t, resources.IsZero(app.GetPlaceholderResource()), "app placeholder: %v", app.GetPlaceholderResource())
	assert.Check(t, partition.getPhAllocationCount() == 0, "partition placeholder count: %d", partition.getPhAllocationCount())
}
{code}

*Proposed fix:* in the core, have {{RollbackAllocation}} subtract a placeholder from {{allocatedPlaceholder}} (handling the placeholder timer when it reaches zero), and have the partition rollback call {{decPhAllocationCount(1)}} for a placeholder. A shim-only guard would leave the core open to the same release from any RM.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org