You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Qi Zhu (Jira)" <ji...@apache.org> on 2023/04/28 14:18:00 UTC
[jira] [Assigned] (YUNIKORN-1712) Placeholder allocations are being removed twice
[ https://issues.apache.org/jira/browse/YUNIKORN-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Qi Zhu reassigned YUNIKORN-1712:
--------------------------------
Assignee: Qi Zhu
> Placeholder allocations are being removed twice
> -----------------------------------------------
>
> Key: YUNIKORN-1712
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1712
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Craig Condit
> Assignee: Qi Zhu
> Priority: Major
> Attachments: logs.txt
>
>
> Under some circumstances, it seems that placeholder allocations are being removed multiple times:
> {code}
> 2023-04-25T06:25:46.279Z INFO scheduler/partition.go:1233 replacing placeholder allocation {"appID": "spark-000000031tn2lgv2gar", "allocationId": "20a4cf77-7095-4635-b9e9-43a7564385c4"}
> ...
> 2023-04-25T06:25:46.299Z INFO scheduler/partition.go:1233 replacing placeholder allocation {"appID": "spark-000000031tn2lgv2gar", "allocationId": "20a4cf77-7095-4635-b9e9-43a7564385c4"}
> {code}
> This message only appears once in the codebase, in {{PartitionContext.removeAllocation()}}. Furthermore, it is guarded by a test for {{release.TerminationType == si.TerminationType_PLACEHOLDER_REPLACED}}. This would seem to indicate that {{removeAllocation()}} is somehow being called *twice*. I believe this would cause the used resources on the node to be subtracted twice for the same allocation. This quickly results in health checks failing:
> {code}
> 2023-04-25T06:26:10.632Z WARN scheduler/health_checker.go:176 Scheduler is not healthy {"health check values": [..., {"Name":"Consistency of data","Succeeded":false,"Description":"Check if node total resource = allocated resource + occupied resource + available resource","DiagnosisMessage":"Nodes with inconsistent data: [\"ip-10-0-112-148.eu-central-1.compute.internal\"]"}, ...]}
> {code}
> This was originally thought to be YUNIKORN-1615, but that seems related to occupied (rather than used) resources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org