[jira] [Updated] (YUNIKORN-2548) Potential deadlock during concurrent bottom-up/top-down queue traversal

["Peter Bacsko (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Peter Bacsko updated YUNIKORN-2548:
-----------------------------------
    Summary: Potential deadlock during concurrent bottom-up/top-down queue traversal  (was: Potential deadlock during concurrent bottom-up and top-down queue traversal)

> Potential deadlock during concurrent bottom-up/top-down queue traversal
> -----------------------------------------------------------------------
>
>                 Key: YUNIKORN-2548
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-2548
>             Project: Apache YuniKorn
>          Issue Type: Sub-task
>          Components: core - scheduler
>            Reporter: Peter Bacsko
>            Assignee: Peter Bacsko
>            Priority: Major
>
> A real deadlock can occur if a queue is removed during config refresh and workloads are still running:
> Goroutine #1
> {noformat}
> objects.(*Queue).DecAllocatedResource { sq.Lock() } <<<<<
> objects.(*Queue).DecAllocatedResource { } }
> objects.(*Queue).DecAllocatedResource { if sq.parent != nil { }
> scheduler.(*PartitionContext).removeAllocation { if resources.StrictlyGreaterThanZero(total) { }
> scheduler.(*ClusterContext).processAllocationReleases { // notify the RM of the exact released allocations }
> scheduler.(*ClusterContext).handleRMUpdateAllocationEvent { } }
> scheduler.(*Scheduler).handleRMEvent { case ev := <-s.pendingEvents: }
> {noformat}
> Goroutine #2
> {noformat}
> objects.(*Queue).MarkQueueForRemoval { sq.Lock() } <<<<<
> objects.(*Queue).MarkQueueForRemoval { // need to lock for write as we don't want to add a queue while marking for removal }
> objects.(*Queue).MarkQueueForRemoval { if len(sq.children) > 0 { }
> scheduler.(*partitionManager).remove { // remove applications: we do not care about return values or issues }
> scheduler.(*partitionManager).Stop { manager.remove() }
> scheduler.(*ClusterContext).Stop { part.partitionManager.Stop() }
> scheduler.(*Scheduler).Stop { s.clusterContext.Stop() }
> entrypoint.(*ServiceContext).StopAll { s.Scheduler.Stop() }
> tests.TestBasicScheduler.deferwrap1 { ms := &mockScheduler{} }
> tests.TestBasicScheduler { waitForPendingAppResource(t, app, 0, 1000) }
> {noformat}
> MarkQueueForRemoval() performs a top-down traversal, while DecAllocatedResource() does the opposite (from the leaf).
> We must reduce the scope of the locks. In both cases the methods are written with this approach:
> {noformat}
> func (t *Type) topDown() {
>   t.Lock()
>   defer t.Unlock()
>   range _, ch := range t.children {   
>     ch.topDown()   // lock must not be hold here
>   }
> }
> func (t *Type) bottomUp() {
>   t.Lock()
>   defer t.Unlock()
>   if t.parent != nil {
>     t.parent.bottumUp()  // lock must not be hold here
>   }
>  }
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org