You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Dale Richardson (Jira)" <ji...@apache.org> on 2026/10/23 23:25:00 UTC
[jira] [Comment Edited] (YUNIKORN-3370) yunikorn-core services are not restartable in-process, so the shim cannot stop the core it starts without intermittent failures
[ https://issues.apache.org/jira/browse/YUNIKORN-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131207#comment-18131207 ]
Dale Richardson edited comment on YUNIKORN-3370 at 10/23/26, 11:24 PM:
-----------------------------------------------------------------------
Re-probed on current core master, 2026-10-24: I disabled every leakcheck exemption on the #1124 branch and ran the full suite ({{-race -tags deadlock}}). All five core exemptions are still live, and they are the only goroutines left that outlive a test binary.
||Goroutine||Leaked instances||Packages||
|{{events.(*EventSystemImpl).StartServiceWithPublisher.func1}}|17|events, scheduler, scheduler/objects, webservice|
|{{events.(*eventPublisher).start.func1}}|6|events, webservice|
|{{scheduler.(*partitionManager).cleanRoot}}|13|scheduler, webservice|
|{{scheduler.(*partitionManager).cleanExpiredApps}}|13|scheduler, webservice|
|{{common/security.(*UserGroupCache).run}}|2|common/security, scheduler|
The other two core exemptions (the event stream forwarder and {{notifyRMNewAllocation}}) were removed after YUNIKORN-3364, YUNIKORN-3436 and YUNIKORN-3365 landed. The fix here is unchanged: {{defer Stop()}} / {{StopAll()}} in the tests in those five packages, then delete the matching entry from {{pkg/common/leakcheck/leakcheck.go}}.
was (Author: tigerquoll):
Re-probed on current core master, 2026-10-24: I disabled every leakcheck exemption on the #1124 branch and ran the full suite ({{-race -tags deadlock}}). All five core exemptions are still live, and they are the only goroutines left that outlive a test binary.
||Goroutine||Leaked instances||Packages||
|{{events.(*EventSystemImpl).StartServiceWithPublisher.func1}}|17|events, scheduler/objects, webservice|
|{{events.(*eventPublisher).start.func1}}|6|events, webservice|
|{{scheduler.(*partitionManager).cleanRoot}}|13|scheduler, webservice|
|{{scheduler.(*partitionManager).cleanExpiredApps}}|13|scheduler, webservice|
|{{common/security.(*UserGroupCache).run}}|2|common/security|
The other two core exemptions (the event stream forwarder and {{notifyRMNewAllocation}}) were removed after YUNIKORN-3364, YUNIKORN-3436 and YUNIKORN-3365 landed. The fix here is unchanged: {{defer Stop()}} / {{StopAll()}} in the tests in those five packages, then delete the matching entry from {{pkg/common/leakcheck/leakcheck.go}}.
> yunikorn-core services are not restartable in-process, so the shim cannot stop the core it starts without intermittent failures
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3370
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3370
> Project: Apache YuniKorn
> Issue Type: Test
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Assignee: weichen lai
> Priority: Minor
> Labels: newbie
>
> Follow-up to YUNIKORN-3357. Supersedes YUNIKORN-3366 (closed as a duplicate of this issue) and replaces the original description of this issue, which claimed the core services could not be restarted in-process. That claim was wrong: {{EventSystemImpl.Stop()}} / {{StartServiceWithPublisher()}} have been CAS-gated since YUNIKORN-3336, {{UserGroupCache.Stop()}} resets its singleton, and core's own {{pkg/scheduler/tests}} mock scheduler already starts and stops the full service set per test in one binary. The A/B in the shim (adding {{coreContext.StopAll()}} to {{{}MockScheduler.stop(){}}}) that motivated the original description no longer reproduces on master (0/15 failures either way).
> What is actually wrong is test hygiene: tests in both repos start core services and never stop them, so their goroutines outlive the test binary and the goleak check has to exempt them.
> Core (#1124), 5 exemptions:
> * {{events.(*EventSystemImpl).StartServiceWithPublisher.func1}} and {{{}events.(*eventPublisher).start.func1{}}}: tests call {{StartService()}} without {{Stop()}}
> * {{scheduler.(*partitionManager).cleanRoot}} and {{{}cleanExpiredApps{}}}: tests build a {{ClusterContext}} / partition without {{Stop()}}
> * {{{}common/security.(*UserGroupCache).run{}}}: tests resolve users without stopping the cache
> Shim (#1061), 17 exemptions, all core-service goroutines: {{MockScheduler.stop()}} stops the shim but not the core it started. Adding {{fc.coreContext.StopAll()}} lets all 17 be deleted; verified on the #1061 branch with the exemption list emptied.
> Repro: delete the exemption in {{pkg/common/leakcheck/leakcheck.go}} and run the package; goleak's {{VerifyTestMain}} fails the binary with the leaked stack.
> Fix: {{defer <service>.Stop()}} / {{StopAll()}} in the affected tests, then delete the exemptions. Can be split per package (events / partition cleaners / UserGroupCache / shim mock).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org