[jira] [Resolved] (YUNIKORN-3365) Shutdown ordering: scheduler allocation notify can block on an RM reply the proxy never drains

["TingYao Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

TingYao Huang resolved YUNIKORN-3365.
-------------------------------------
     Fix Version/s: 1.10.0
    Target Version: 1.10.0
        Resolution: Fixed

> Shutdown ordering: scheduler allocation notify can block on an RM reply the proxy never drains
> ----------------------------------------------------------------------------------------------
>
>                 Key: YUNIKORN-3365
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3365
>             Project: Apache YuniKorn
>          Issue Type: Bug
>          Components: core - scheduler
>            Reporter: Dale Richardson
>            Assignee: weichen lai
>            Priority: Minor
>              Labels: newbie, pull-request-available
>             Fix For: 1.10.0
>
>
> {{ServiceContext.StopAll}} stops the scheduler before the RM proxy. {{Scheduler.handleAllocEvent}} can select a queued allocation event over the just-closed stop channel; the chain {{handleAllocEvent -> handleRMUpdateAllocationEvent -> processAllocations -> notifyRMNewAllocation}} then posts to the RM proxy and blocks on an unbuffered reply channel ({{context.go:803}}), but {{rmproxy.handleRMEvents}} returns on its own stop without draining {{pendingRMEvents}}, so the reply never arrives and the goroutine parks.
> Reproduced in ~43% of full {{pkg/scheduler/tests}} runs (6/14): {{go test -race -count=14 ./pkg/scheduler/tests/}}.
> Severity is limited under the current lifecycle. As noted on PR #1124, {{StopAll}} today runs as the process exits — the reply is moot and the queued events are dropped anyway — so the parked goroutine is harmless in normal operation. It matters only if (a) the parked goroutine blocks {{StopAll}} from completing cleanly, or (b) the scheduler is stopped without the process exiting, i.e. under in-process restart (YUNIKORN-3370). {{notifyRMAllocationReleased}} has the same unbuffered-reply shape.
> Proposed fix: drain or explicitly abandon in-flight replies on shutdown (or invert the stop order), then delete the {{notifyRMNewAllocation}} exemption in {{pkg/common/leakcheck/leakcheck.go}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org