[jira] [Updated] (YUNIKORN-3411) Application write lock held across the RM release round trip

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated YUNIKORN-3411:
-------------------------------------
    Labels: pull-request-available  (was: )

> Application write lock held across the RM release round trip
> ------------------------------------------------------------
>
>                 Key: YUNIKORN-3411
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3411
>             Project: Apache YuniKorn
>          Issue Type: Sub-task
>          Components: core - scheduler
>            Reporter: Dale Richardson
>            Assignee: Andy Huang
>            Priority: Major
>              Labels: pull-request-available
>
> {{Application.notifyRMAllocationReleased}} hands a release to the RM proxy and then waits on an unbuffered channel for the answer. Every path that reaches it holds the application write lock across that wait: the state timeout ({{timeoutStateTimer}}), placeholder timeout and placeholder replacement ({{timeoutPlaceholderProcessing}}, {{tryPlaceholderAllocate}}), and the whole preemption chain from {{tryAllocate}} through {{Preemptor.TryPreemption}} and the required-node preemptor.
> {code}	sa.rmEventHandler.HandleEvent(releaseEvent)
> 	// Wait from channel
> 	result := <-c        // no timeout; the application lock is held by every caller
> {code}
> The RM proxy never takes the application lock, so this is not a deadlock; it is an unbounded stall. How long the application is locked is however long the shim takes to process the release. It becomes permanent if the RM event queue is full and the event is dropped ({{HandleEvent}} drops on a full queue with a {{DPanic}} that is only a log line in production), or if the RM proxy goroutine is wedged for another reason. On the preemption chain the lock belongs to the scheduling goroutine, so all scheduling stops for the duration; on the timer paths that one application can neither schedule, complete nor be queried. This was measured as a stall during the July review with a test that delays the RM reply (not in the tree). YUNIKORN-1099 fixed a nil-channel deadlock in the same function; the lock-across-round-trip shape stayed.
> Fix: notify after unlocking. The timer paths can collect the releases under the lock, unlock, then notify. The preemption chain needs the victim list handed back up to {{Queue.TryAllocate}} (or a callback run after {{tryAllocate}} releases the lock). Bounding the wait is a fallback, not a fix.
> Marker: 11 {{+lockblockingignore}} sites across {{application.go}}, {{preemption.go}} and {{required_node_preemptor.go}} carry this JIRA; the fix removes them.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org