You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yunikorn.apache.org by "Craig Condit (Jira)" <ji...@apache.org> on 2023/05/11 20:50:00 UTC
[jira] [Resolved] (YUNIKORN-1712) Fix allocatedResource and availableResource should be updated at the sametime for ReplaceAllocation
[ https://issues.apache.org/jira/browse/YUNIKORN-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Craig Condit resolved YUNIKORN-1712.
------------------------------------
Fix Version/s: 1.3.0
Resolution: Fixed
Merged to master. Thanks [~zhuqi] for the contribution.
> Fix allocatedResource and availableResource should be updated at the sametime for ReplaceAllocation
> ---------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-1712
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1712
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Craig Condit
> Assignee: Qi Zhu
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.3.0
>
> Attachments: fullstatedump.json, logs.txt, logs_debug_level.txt
>
>
> Under some circumstances, it seems that placeholder allocations are being removed multiple times:
> {code:java}
> 2023-04-25T06:25:46.279Z INFO scheduler/partition.go:1233 replacing placeholder allocation {"appID": "spark-000000031tn2lgv2gar", "allocationId": "20a4cf77-7095-4635-b9e9-43a7564385c4"}
> ...
> 2023-04-25T06:25:46.299Z INFO scheduler/partition.go:1233 replacing placeholder allocation {"appID": "spark-000000031tn2lgv2gar", "allocationId": "20a4cf77-7095-4635-b9e9-43a7564385c4"}
> {code}
> This message only appears once in the codebase, in {{{}PartitionContext.removeAllocation(){}}}. Furthermore, it is guarded by a test for {{{}release.TerminationType == si.TerminationType_PLACEHOLDER_REPLACED{}}}. This would seem to indicate that {{removeAllocation()}} is somehow being called {*}twice{*}. I believe this would cause the used resources on the node to be subtracted twice for the same allocation. This quickly results in health checks failing:
> {code:java}
> 2023-04-25T06:26:10.632Z WARN scheduler/health_checker.go:176 Scheduler is not healthy {"health check values": [..., {"Name":"Consistency of data","Succeeded":false,"Description":"Check if node total resource = allocated resource + occupied resource + available resource","DiagnosisMessage":"Nodes with inconsistent data: [\"ip-10-0-112-148.eu-central-1.compute.internal\"]"}, ...]}
> {code}
> This was originally thought to be YUNIKORN-1615, but that seems related to occupied (rather than used) resources.
> Updated by [~zhuqi] :
> The release allocation called twice, but it not cause the node and other resource update twice.
> The title will be changed to:
> Fix allocatedResource and availableResource should be updated at the sametime for ReplaceAllocation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org