[jira] [Resolved] (YUNIKORN-3452) Node Preemption: Fix reversed penalty scores between originator and opted-out pods

["weichen lai (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

weichen lai resolved YUNIKORN-3452.
-----------------------------------
    Fix Version/s: 1.11.0
       Resolution: Fixed

> Node Preemption: Fix reversed penalty scores between originator and opted-out pods
> ----------------------------------------------------------------------------------
>
>                 Key: YUNIKORN-3452
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3452
>             Project: Apache YuniKorn
>          Issue Type: Bug
>          Components: core - scheduler
>            Reporter: weichen lai
>            Assignee: weichen lai
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.11.0
>
>
> h3. Problem
> Default preemption protects an opted-out pod more than an originator pod in two places on the node path:
> # {{sortVictimsForPreemption()}} sorts by {{allowPreemptSelf}} first and {{originator}} second, so an originator that allows preemption is preempted before an opted-out worker on the same node.
> # {{getSolutionScore()}}, used by {{checkPreemptionPredicates()}} to select a node, uses these penalties (lower is better):
> {code:go}scoreOriginator uint64 = 1 << 33
> scoreNoPreempt  uint64 = 1 << 34
> {code}
> A node that needs the originator is therefore preferred over a node that needs an opted-out worker.
> Preempting an originator (driver) fails the whole application; preempting an opted-out worker affects one task.
> h3. Expected order
> Regular > Opted-out > Originator.
> * [Preemption design|https://yunikorn.apache.org/docs/next/design/preemption/]: preempting an originator "should be prevented as much as possible"; the opt-out flag is "a strong suggestion, not a guarantee" (Law 1).
> * [Quota preemption design|https://yunikorn.apache.org/docs/next/design/quota_change_enforcement_through_preemption]: "Prefer Non originator pods over originator".
> h3. Impact
> Only when the regular pods on a node are not sufficient. When it happens, the originator is preempted and the whole application fails.
> h3. Proposed change
> * Swap the values of {{scoreOriginator}} and {{scoreNoPreempt}}.
> * In {{sortVictimsForPreemption()}}, sort by {{originator}} first and {{allowPreemptSelf}} second.
> h3. Reproducer / Unit Tests
> Both tests fail on current {{master}}:
> {code:go}func TestSolutionScoring_OriginatorVsOptedOut(t *testing.T) {
> 	allocMap := map[string][]*Allocation{
> 		"node-opted-out":  {allocForScore(false, false)},
> 		"node-originator": {allocForScore(true, true)},
> 	}
> 	solOptedOut := &predicateCheckResult{nodeID: "node-opted-out", success: true, index: 0}
> 	solOriginator := &predicateCheckResult{nodeID: "node-originator", success: true, index: 0}
> 	// An originator pod is the driver/core of an app and must be protected over an opted-out worker pod.
> 	// Therefore, a solution that sacrifices an opted-out pod MUST be betterThan a solution that sacrifices an originator pod.
> 	assert.Check(t, solOptedOut.betterThan(solOriginator, allocMap),
> 		"sacrificing opted-out pod should be a better solution than sacrificing originator pod")
> }
> func TestSortVictimsForPreemption(t *testing.T) {
> 	res := resources.NewResourceFromMap(map[string]resources.Quantity{"first": 10})
> 	regularPod := createAllocation("regularPod", "app1", nodeID1, true, false, 10, false, res)
> 	originatorPod := createAllocation("originatorPod", "app1", nodeID1, true, true, 10, false, res)
> 	optedOutPod := createAllocation("optedOutPod", "app1", nodeID1, false, false, 10, false, res)
> 	optedOutOriginatorPod := createAllocation("optedOutOriginatorPod", "app1", nodeID1, false, true, 10, false, res)
> 	allocationsByNode := map[string][]*Allocation{
> 		nodeID1: {optedOutOriginatorPod, originatorPod, optedOutPod, regularPod},
> 	}
> 	sortVictimsForPreemption(allocationsByNode)
> 	allocations := allocationsByNode[nodeID1]
> 	assert.Equal(t, allocations[0].GetAllocationKey(), "regularPod")
> 	assert.Equal(t, allocations[1].GetAllocationKey(), "optedOutPod")
> 	assert.Equal(t, allocations[2].GetAllocationKey(), "originatorPod")
> 	assert.Equal(t, allocations[3].GetAllocationKey(), "optedOutOriginatorPod")
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org