You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Paul Santa Clara (Jira)" <ji...@apache.org> on 2024/10/25 18:40:00 UTC
[jira] [Updated] (YUNIKORN-2950) Race condition in 1.5.2 causes queue usage to be incorrectly calculated
[ https://issues.apache.org/jira/browse/YUNIKORN-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Santa Clara updated YUNIKORN-2950:
---------------------------------------
Summary: Race condition in 1.5.2 causes queue usage to be incorrectly calculated (was: Race condition in 1.5.2 causes queue usage to be incorrectly calcuated)
> Race condition in 1.5.2 causes queue usage to be incorrectly calculated
> -----------------------------------------------------------------------
>
> Key: YUNIKORN-2950
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2950
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Paul Santa Clara
> Priority: Major
>
> We observed some of our larger clusters(with autoscaling via Karpenter) begin to mis-report their queue usages after upgrading to 1.5.2.
> As an example, given a leaf queue root.tiers.2, we observed the following state:
> {code:java}
> allocated capacity(root.tiers.2) : pods 200 memory 2.187347412109375 Tib vcore 0.2 k ephemeral-storage 4.294967295999999TB{code}
> but when we summed the allocations in the full-state dump, we found:
> {code:java}
> root.tiers.2 : pods 0 memory 0.0 Tib vcore 0.0 k ephemeral-storage 0.0 TB{code}
> Similarly, we examined the number of running pods in K8s, and we found 0. The queue allocations were clearly off.
> This was fixed by applying the following patch to remove the race condition:
> {code:java}
> func (sq *Queue) IncAllocatedResource(alloc *resources.Resource, nodeReported bool) error {
> // check this queue: failure stops checks if the allocation is not part of a node addition
> - fit, newAllocated := sq.allocatedResFits(alloc)
> + fit, _ := sq.allocatedResFits(alloc)
> if !nodeReported && !fit {
> return fmt.Errorf("allocation (%v) puts queue '%s' over maximum allocation (%v), current usage (%v)",
> alloc, sq.QueuePath, sq.maxResource, sq.allocatedResource)
> @@ -1058,6 +1058,7 @@ func (sq *Queue) IncAllocatedResource(alloc *resources.Resource, nodeReported bo
> sq.Lock()
> defer sq.Unlock()
> // all OK update this queue
> + newAllocated := resources.Add(sq.allocatedResource, alloc)
> sq.allocatedResource = newAllocated
> sq.updateAllocatedResourceMetrics()
> return nil {code}
>
> This appears to be [fixed|#L1041] in 1.6.0, although I have not confirmed it.
> It looks to me that the race condition was introduced [here|https://github.com/apache/yunikorn-core/pull/839/files#diff-27632d48eb925e150a33bc92370ceaa66c31048018d11ca7a53a0b50ab7250acL1033].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org