[jira] [Created] (YUNIKORN-3365) Shutdown race: scheduler allocation notify blocks forever because the RM proxy stops without draining pending events

["Dale Richardson (Jira)" <ji...@apache.org>]

Dale Richardson created YUNIKORN-3365:
-----------------------------------------

             Summary: Shutdown race: scheduler allocation notify blocks forever because the RM proxy stops without draining pending events
                 Key: YUNIKORN-3365
                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3365
             Project: Apache YuniKorn
          Issue Type: Bug
          Components: core - scheduler
            Reporter: Dale Richardson


Follow-up to YUNIKORN-3357 (PR #1124); burns down the leakcheck exemption
scheduler.(*ClusterContext).notifyRMNewAllocation.

{{ServiceContext.StopAll}} stops the scheduler before the RM proxy. {{Scheduler.handleAllocEvent}} can select a queued allocation event over the just-closed stop channel; the chain {{handleAllocEvent -> handleRMUpdateAllocationEvent -> processAllocations -> notifyRMNewAllocation}} then posts to the RM proxy and blocks on an unbuffered reply channel ({{context.go:803}}), but {{rmproxy.handleRMEvents}} returns on its own stop without draining {{pendingRMEvents}}, so the reply never arrives and the goroutine is stuck permanently.

Reproduced in ~43% of full {{pkg/scheduler/tests}} runs (6/14):
{code}
go test -race -count=14 ./pkg/scheduler/tests/
{code}

{{notifyRMAllocationReleased}} has the same unbuffered-reply shape and would leak under a different top frame; it has not been observed yet, so it is deliberately not exempted — but the fix should cover it too.

Proposed fix: drain or explicitly abandon in-flight replies on shutdown (or invert the stop order), then delete the exemption in {{pkg/common/leakcheck/leakcheck.go}}. Argue both the deadlock and data-integrity directions preemptively (cf. the YUNIKORN-2629 -> YUNIKORN-2910 regression).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org