[jira] [Resolved] (YUNIKORN-3417) RM event replies are unbuffered sends made under the ClusterContext write lock

["TingYao Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

TingYao Huang resolved YUNIKORN-3417.
-------------------------------------
     Fix Version/s: 1.11.0
    Target Version: 1.11.0
        Resolution: Fixed

> RM event replies are unbuffered sends made under the ClusterContext write lock
> ------------------------------------------------------------------------------
>
>                 Key: YUNIKORN-3417
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3417
>             Project: Apache YuniKorn
>          Issue Type: Sub-task
>          Components: core - scheduler
>            Reporter: Dale Richardson
>            Assignee: Andy Huang
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 1.11.0
>
>
> The three infrastructure event handlers in {{context.go}} ({{processRMRegistrationEvent}}, {{processRMConfigUpdateEvent}}, {{removePartitionsByRMID}}) reply to the RM proxy with {{event.Channel <- &rmevent.Result{...}}} while holding {{cc.Lock()}}. The channel is unbuffered and nothing guarantees the reply is collected. If the goroutine that posted the event is gone, or the reply is sent on a channel nobody reads, the send blocks for ever with the cluster context write lock held, and the scheduling cycle, the REST handlers and the health check stall with it.
> Today every producer parks unconditionally on the reply, so this needs the pairing to fail first; nothing reproduces it and no test exercises a dropped reply. It is the lock-side view of the plumbing in YUNIKORN-3365 (reply never drained on shutdown): there a parked goroutine is harmless, here a failed rendezvous is cluster-wide. Filed so the ten suppressed sends have a home rather than because it has bitten anyone.
> Fix: compute the result under the lock, release it, then send; or give the reply channels a buffer of one so the send cannot block.
> Marker: one commented site plus nine sibling sends in the same three handlers carry {{+lockblockingignore}} for this JIRA; the fix removes them.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org