[jira] [Commented] (YUNIKORN-1712) Placeholder allocations are being removed twice

["Craig Condit (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YUNIKORN-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717386#comment-17717386 ] 

Craig Condit commented on YUNIKORN-1712:
----------------------------------------

Slack conversation where this issue was encountered:

*Abdullah:*
Hey team - Has anyone tried gang-scheduling with Spark pods having side-car containers?

*Sunil Govindan:*
Yes. We are using spark driver/exec running with sidecar.

*Wilfred Spiegelenburg:*
You do need to include the side car container size in the gang request you create.

*Abdullah:*
I did that but I am seeing wrong node capacities allocated + available < total and the scheduler is no more healthy. Not sure where this is coming from.

Placeholders are getting replaced by actual pods but the scheduler is showing wrong available capacity (lower than what it should be)which is not equal to the total capacity when the job finishes.

*Wilfred Spiegelenburg:*
In the logs we should show the difference between the real pod and the placeholder, if there is any. We account for that.
If the placeholder is smaller than the real allocation we cleanup all placeholders and allocate from scratch.
 If the placeholder is larger than the real one we update the tracked info in the node and queue.
We have tested that in lots of cases and we do the correct accounting.
I think that difference is still YUNIKORN-1615.

*Abdullah:*
I don’t think it is because it is happening always not occasionally  and with a clean helm installation.
I managed to get the scheduler logs (attached).

*Craig Condit*
Created YUNIKORN-1712 to track this. I now believe it is distinct from YUNIKORN-1615, as that was node occupied resources going negative (non-YuniKorn), whereas this is used resources.

> Placeholder allocations are being removed twice
> -----------------------------------------------
>
>                 Key: YUNIKORN-1712
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-1712
>             Project: Apache YuniKorn
>          Issue Type: Bug
>          Components: core - scheduler
>            Reporter: Craig Condit
>            Priority: Major
>
> Under some circumstances, it seems that placeholder allocations are being removed multiple times:
> {code}
> 2023-04-25T06:25:46.279Z	INFO	scheduler/partition.go:1233	replacing placeholder allocation {"appID": "spark-000000031tn2lgv2gar", "allocationId": "20a4cf77-7095-4635-b9e9-43a7564385c4"}
> ...
> 2023-04-25T06:25:46.299Z	INFO	scheduler/partition.go:1233	replacing placeholder allocation {"appID": "spark-000000031tn2lgv2gar", "allocationId": "20a4cf77-7095-4635-b9e9-43a7564385c4"}
> {code}
> This message only appears once in the codebase, in {{PartitionContext.removeAllocation()}}. Furthermore, it is guarded by a test for {{release.TerminationType == si.TerminationType_PLACEHOLDER_REPLACED}}. This would seem to indicate that {{removeAllocation()}} is somehow being called *twice*. I believe this would cause the used resources on the node to be subtracted twice for the same allocation. This quickly results in health checks failing:
> {code}
> 2023-04-25T06:26:10.632Z        WARN    scheduler/health_checker.go:176 Scheduler is not healthy        {"health check values": [..., {"Name":"Consistency of data","Succeeded":false,"Description":"Check if node total resource = allocated resource + occupied resource + available resource","DiagnosisMessage":"Nodes with inconsistent data: [\"ip-10-0-112-148.eu-central-1.compute.internal\"]"}, ...]}
> {code}
> This was originally thought to be YUNIKORN-1615, but that seems related to occupied (rather than used) resources.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org