[jira] [Resolved] (YUNIKORN-3231) ForeignAllocations ignored / not accounted when pods use nodeSelector (works with nodeAffinity)

["Wilfred Spiegelenburg (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wilfred Spiegelenburg resolved YUNIKORN-3231.
---------------------------------------------
    Resolution: Duplicate

as mentioned closing as duplicate of YUNIKORN-3317

> ForeignAllocations ignored / not accounted when pods use nodeSelector (works with nodeAffinity)
> -----------------------------------------------------------------------------------------------
>
>                 Key: YUNIKORN-3231
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3231
>             Project: Apache YuniKorn
>          Issue Type: Bug
>         Environment: Kubernetes: 1.26
> YuniKorn scheduler: 1.8.0
>            Reporter: Alex Ovchenkov
>            Priority: Major
>
> h2. Description
> We observed that YuniKorn’s *ForeignAllocations* (pods not managed by YuniKorn / allocations coming from outside) may not be accounted correctly when such pods use {*}nodeSelector{*}. As a result, YuniKorn may overestimate node/queue available resources and make scheduling decisions as if those external pods do not exist (or are not tied to the expected nodes).
> When we replace {{nodeSelector}} with an equivalent {{{}nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution{}}}, the behavior becomes correct and ForeignAllocations are accounted as expected.
> h2. *Steps to reproduce*
> h5. 1) Create a pod that is scheduled by Kubernetes default scheduler (or another scheduler) and uses nodeSelector:
>  
> {code:java}
> apiVersion: v1
> kind: Pod
> metadata:
>   name: foreign-pod-selector
>   namespace: <ns>
> spec:
>   schedulerName: default-scheduler
>   nodeSelector:
>     dedicated: test
>   containers:
>     - name: c
>       image: busybox
>       command: ["sh", "-c", "sleep 360000"]
>       resources:
>         requests:
>           cpu: "1"
>           memory: "1Gi" {code}
> h5. 2) Observe YuniKorn resource accounting / foreign allocations
> Check YuniKorn UI / REST API / state dump and verify whether this pod appears as a foreign allocation and whether its resources are deducted from the node.
> h5. 3) Replace nodeSelector with equivalent nodeAffinity (requiredDuringScheduling)
> Delete the pod and apply the same pod but with nodeAffinity instead:
> {code:java}
> apiVersion: v1
> kind: Pod
> metadata:
>   name: foreign-pod-affinity
>   namespace: <ns>
> spec:
>   schedulerName: default-scheduler
>   affinity:
>     nodeAffinity:
>       requiredDuringSchedulingIgnoredDuringExecution:
>         nodeSelectorTerms:
>           - matchExpressions:
>               - key: dedicated
>                 operator: In
>                 values: ["test"]
>   containers:
>     - name: c
>       image: busybox
>       command: ["sh", "-c", "sleep 360000"]
>       resources:
>         requests:
>           cpu: "1"
>           memory: "1Gi" {code}
> Re-check YuniKorn accounting / foreign allocations.
> ----
> h2. Actual behavior
>  * With {*}nodeSelector{*}, the foreign pod is *not consistently* represented in ForeignAllocations / node accounting (or appears without correct node association / resources not deducted).
>  * This can lead to YuniKorn scheduling decisions that assume more resources are available on the affected node(s) than in reality.
> ----
> h2. Expected behavior
>  * Foreign pods should be accounted consistently regardless of whether the workload uses:
>  ** {{{}spec.nodeSelector{}}}, or
>  ** {{spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution}}
>  * If the pod is bound to a node (and consuming resources), YuniKorn should reflect it in foreign allocations / node resource usage.
> ----
> h2. Notes / hypothesis
> Based on initial investigation, the issue might be related to how foreign pods are processed in {{{}updateForeignPod{}}}:
> [https://github.com/apache/yunikorn-k8shim/blob/master/pkg/cache/context.go#L412]
> It appears that when a pod uses {{{}nodeSelector{}}}, some of its scheduling-related state (possibly node assignment or allocation state) may already be set earlier in the reconciliation flow, which could cause {{updateForeignPod}} to skip or treat it differently.
> In contrast, when equivalent constraints are expressed using {{{}nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution{}}}, the foreign allocation seems to be tracked and accounted correctly.
> This suggests there may be a difference in how {{nodeSelector}} and {{nodeAffinity}} are interpreted or propagated into the internal foreign allocation tracking logic.
> We may be missing some detail in the lifecycle of foreign pod updates, but the observable behavior indicates inconsistent accounting depending on whether {{nodeSelector}} or {{nodeAffinity}} is used.
> ----
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org