[jira] [Created] (YUNIKORN-3512) Resizing an allocated placeholder corrupts application accounting

["Dian-Xuan Yang (Jira)" <ji...@apache.org>]

Dian-Xuan Yang created YUNIKORN-3512:
----------------------------------------

             Summary: Resizing an allocated placeholder corrupts application accounting
                 Key: YUNIKORN-3512
                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3512
             Project: Apache YuniKorn
          Issue Type: Bug
          Components: core - scheduler
            Reporter: Dian-Xuan Yang
            Assignee: Dian-Xuan Yang


h3. Problem

When a placeholder pod is resized in place, the shim sends the new size to the core in the same way as for any other pod. {{Task.SetTaskPod}} detects the resource change, and {{Task.updateAllocation}} sends an update with {{Placeholder=true}} and the node set.

In the core, {{PartitionContext.UpdateAllocation}} passes the change to {{{}Application.UpdateAllocationResources{}}}. For an allocated ask, that method adds the delta to {{allocatedResource}} without checking whether the allocation is a placeholder. The delta should go to {{allocatedPlaceholder}} instead.

When the placeholder is later replaced or released, {{removeAllocationInternal}} subtracts its new size from {{{}allocatedPlaceholder{}}}. Both totals then end up wrong, in opposite directions.
h3. Incorrect accounting

Consider a gang application with two 2-vcore placeholders. Placeholder 1 is resized from 2 to 3 vcore after both placeholders are allocated.
||Accounting||Before resize||Expected||Current||
|{{allocatedResource}}|0|0|*1*|
|{{allocatedPlaceholder}}|4|5|*4*|

Shrinking placeholder 1 from 2 to 1 vcore instead leaves {{allocatedResource}} at *-1* and {{allocatedPlaceholder}} at {*}4{*}. The expected values are 0 and 3.

Queue, node and user usage remain correct, and {{allocatedResource}} + {{allocatedPlaceholder}} still equals the queue usage. The inconsistency is in the application's split between normal and placeholder resources.
h3. Impact
h3. Incorrect REST state
 * REST {{usedResource}} can become negative.
 * {{placeholderUsedResource}} can become negative, or keep a value after all placeholders are gone.

h3. Application lifecycle

Each scenario below was compared with the same run without the resize, which reaches {{{}Completed{}}}.
||Scenario||Outcome||Final {{allocatedResource}} / {{allocatedPlaceholder}}||
|Grow 2→3 vcore. Both placeholders are replaced, then both real allocations are released.|The application has no allocations left but {*}stays in {{Running}}{*}.|1 / -1|
|Shrink 2→1 vcore. Both placeholders are replaced by 1-vcore asks, then both are released.|The application enters {{Completing}} while a real allocation is still running, then {*}stays in {{Completing}}{*}.|-1 / 1|
|Grow 2→4 vcore. Only one real allocation arrives and replaces placeholder 1.|*Placeholder 2 never times out* and keeps its 2 vcore on the node and in the queue. The application stays in {{{}Running{}}}.|2 / 0|

In the {{Completing}} case, the completing timer has the same problem as in YUNIKORN-3509. It sees a non-zero {{{}allocatedPlaceholder{}}}, finds no placeholder allocation to release, and clears itself without sending {{{}CompleteApplication{}}}.

In the last case, replacing the resized placeholder subtracts 4 vcore from an {{allocatedPlaceholder}} of 4 vcore. The total reaches zero while placeholder 2 is still allocated. {{removeAllocationInternal}} then clears the placeholder timer, so the timeout that should release placeholder 2 never fires.
h3. Trigger

Any in-place resize of an allocated placeholder pod, for example:
{code:bash}
kubectl patch pod <placeholder-pod> --subresource resize \
  --patch '{"spec":{"containers":[{"name":"pause","resources":{"requests":{"cpu":"3"},"limits":{"cpu":"3"}}}]}}'

{code}
h3. Proposed fix

When the existing allocation is a placeholder, {{UpdateAllocationResources}} should apply the delta to {{allocatedPlaceholder}} instead of {{{}allocatedResource{}}}. The queue and user tracker updates stay as they are.

This needs one design decision. {{addAllocationInternal}} moves the application to {{Running}} when {{{}resources.Equals(allocatedPlaceholder, placeholderAsk){}}}. Today a resize does not change {{{}allocatedPlaceholder{}}}, so the gate still works when a placeholder is resized before the last one is allocated. Once the delta is booked into {{{}allocatedPlaceholder{}}}, the two totals are no longer equal and the gate never fires. The gate would need a check that does not depend on placeholder sizes, for example a count of allocated placeholders per task group.

Dropping placeholder resizes in the core or the shim would avoid the gate change. However, it would leave queue and node usage out of step with what the pod actually requests.

This is the same class of problem as YUNIKORN-3509, which covers the rollback path. Both were found while adding {{placeholderUsedResource}} in YUNIKORN-2133.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org