[jira] [Updated] (YUNIKORN-3512) Resizing an allocated placeholder corrupts application accounting

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated YUNIKORN-3512:
-------------------------------------
    Labels: pull-request-available  (was: )

> Resizing an allocated placeholder corrupts application accounting
> -----------------------------------------------------------------
>
>                 Key: YUNIKORN-3512
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3512
>             Project: Apache YuniKorn
>          Issue Type: Bug
>          Components: core - scheduler
>            Reporter: Dian-Xuan Yang
>            Assignee: Dian-Xuan Yang
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> When a placeholder pod is resized in place, the shim sends the new size to the core in the same way as for any other pod. {{Task.SetTaskPod}} detects the resource change, and {{Task.updateAllocation}} sends an update with {{Placeholder=true}} and the node set.
> In the core, {{PartitionContext.UpdateAllocation}} passes the change to {{{}Application.UpdateAllocationResources{}}}. For an allocated ask, that method adds the delta to {{allocatedResource}} without checking whether the allocation is a placeholder. The delta should go to {{allocatedPlaceholder}} instead.
> When the placeholder is later replaced or released, {{removeAllocationInternal}} subtracts its new size from {{{}allocatedPlaceholder{}}}. Both totals then end up wrong, in opposite directions.
> h3. Incorrect accounting
> Consider a gang application with two 2-vcore placeholders. Placeholder 1 is resized from 2 to 3 vcore after both placeholders are allocated.
> ||Accounting||Before resize||Expected||Current||
> |{{allocatedResource}}|0|0|*1*|
> |{{allocatedPlaceholder}}|4|5|*4*|
> Shrinking placeholder 1 from 2 to 1 vcore instead leaves {{allocatedResource}} at *-1* and {{allocatedPlaceholder}} at {*}4{*}. The expected values are 0 and 3.
> Queue, node and user usage remain correct, and {{allocatedResource}} + {{allocatedPlaceholder}} still equals the queue usage. The inconsistency is in the application's split between normal and placeholder resources.
> h3. Impact
> h3. Incorrect REST state
>  * REST {{usedResource}} can become negative.
>  * {{placeholderUsedResource}} can become negative, or keep a value after all placeholders are gone.
> h3. Application lifecycle
> Each scenario below was compared with the same run without the resize, which reaches {{{}Completed{}}}.
> ||Scenario||Outcome||Final {{allocatedResource}} / {{allocatedPlaceholder}}||
> |Grow 2→3 vcore. Both placeholders are replaced, then both real allocations are released.|The application has no allocations left but {*}stays in {{Running}}{*}.|1 / -1|
> |Shrink 2→1 vcore. Both placeholders are replaced by 1-vcore asks, then both are released.|The application enters {{Completing}} while a real allocation is still running, then {*}stays in {{Completing}}{*}.|-1 / 1|
> |Grow 2→4 vcore. Only one real allocation arrives and replaces placeholder 1.|*Placeholder 2 never times out* and keeps its 2 vcore on the node and in the queue. The application stays in {{{}Running{}}}.|2 / 0|
> In the {{Completing}} case, the completing timer has the same problem as in YUNIKORN-3509. It sees a non-zero {{{}allocatedPlaceholder{}}}, finds no placeholder allocation to release, and clears itself without sending {{{}CompleteApplication{}}}.
> In the last case, replacing the resized placeholder subtracts 4 vcore from an {{allocatedPlaceholder}} of 4 vcore. The total reaches zero while placeholder 2 is still allocated. {{removeAllocationInternal}} then clears the placeholder timer, so the timeout that should release placeholder 2 never fires.
> h3. Trigger
> Any in-place resize of an allocated placeholder pod, for example:
> {code:bash}
> kubectl patch pod <placeholder-pod> --subresource resize \
>   --patch '{"spec":{"containers":[{"name":"pause","resources":{"requests":{"cpu":"3"},"limits":{"cpu":"3"}}}]}}'
> {code}
> h3. Proposed fix
> When the existing allocation is a placeholder, {{UpdateAllocationResources}} should apply the delta to {{allocatedPlaceholder}} instead of {{{}allocatedResource{}}}. The queue and user tracker updates stay as they are.
> This needs one design decision. {{addAllocationInternal}} moves the application to {{Running}} when {{{}resources.Equals(allocatedPlaceholder, placeholderAsk){}}}. Today a resize does not change {{{}allocatedPlaceholder{}}}, so the gate still works when a placeholder is resized before the last one is allocated. Once the delta is booked into {{{}allocatedPlaceholder{}}}, the two totals are no longer equal and the gate never fires. The gate would need a check that does not depend on placeholder sizes, for example a count of allocated placeholders per task group.
> Dropping placeholder resizes in the core or the shim would avoid the gate change. However, it would leave queue and node usage out of step with what the pod actually requests.
> This is the same class of problem as YUNIKORN-3509, which covers the rollback path. Both were found while adding {{placeholderUsedResource}} in YUNIKORN-2133.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org