You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Wilfred Spiegelenburg (Jira)" <ji...@apache.org> on 2023/09/20 05:58:00 UTC
[jira] [Commented] (YUNIKORN-1993) Race between allocation removal and Completed state change
[ https://issues.apache.org/jira/browse/YUNIKORN-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767003#comment-17767003 ]
Wilfred Spiegelenburg commented on YUNIKORN-1993:
-------------------------------------------------
Further detail on what happened based on the log and state dump analysis:
Our base was a Spark application that had defined one driver and one executor placeholder.
Both placeholders got allocated. The real driver replaced the placeholder and runs. There is never an executor pod created.
Now it becomes a little complex:
* placeholder timeout is triggered, removal of the unused placeholder is started
* driver exits normally and the allocation gets removed
* driver allocation removal triggers a change to completing state as it is the only _real_ allocation of the application
* ... time passes
* placeholder removal confirmation comes in from the shim and removal finishes and releases the application lock.
* completing state times out and gets the application lock
* no allocations or placeholders left which triggers a transition to Completed state
* on entry of Completed state the application queue link is set to nil
* placeholder removal has progressed to the point to update the queue allocations
When we try to retrieve the queue we get a nil. The queue is not set anymore and does not get updated…
We leak resources on the queue in the allocated accounting.
> Race between allocation removal and Completed state change
> ----------------------------------------------------------
>
> Key: YUNIKORN-1993
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1993
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Wilfred Spiegelenburg
> Assignee: Wilfred Spiegelenburg
> Priority: Critical
>
> A race between go routines exists that can leave allocation tracked on a queue. The end result could show a queue that has allocation without any running applications in the queue.
> Worst case scenario would be an exhausted root queue quota causing all scheduling to stop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org