[jira] [Created] (GUACAMOLE-2270) ABBA rwlock deadlock

["Masahji Stewart (Jira)" <ji...@apache.org>]

Masahji Stewart created GUACAMOLE-2270:
------------------------------------------

             Summary: ABBA rwlock deadlock
                 Key: GUACAMOLE-2270
                 URL: https://issues.apache.org/jira/browse/GUACAMOLE-2270
             Project: Guacamole
          Issue Type: Bug
          Components: guacamole-server
    Affects Versions: 1.6.0
            Reporter: Masahji Stewart
             Fix For: 1.6.1 [staging]


We've noticed an ABBA deadlock between two libguac functions that take {{display->last_frame.lock}} and {{display->pending_frame.lock}} in opposite orders:
||Function||Lock order||
|{{guac_display_dup()}}|last_frame (R) -> pending_frame (R)|
|{{guac_display_end_multiple_frames()}}|pending_frame (W) -> last_frame (W)|
h2. How it happens

When a new user is being promoted (the protocol's {{join_pending_handler}} calls {{{}guac_display_dup{}}}) at the same time the render loop is flushing a frame ({{{}guac_display_end_multiple_frames{}}}), the two threads wedge:
 * T1 holds {{last_frame.lock}} (R), blocks acquiring {{pending_frame.lock}} (R) — via {{guac_display_layer_get_bounds()}} called from inside {{{}guac_display_dup{}}}'s per-layer loop.
 * T2 holds {{pending_frame.lock}} (W), blocks acquiring {{last_frame.lock}} (W).

The default Linux glibc rwlock policy queues new readers behind a waiting writer, so once T2 has parked on {{last_frame.lock}} as a writer, T1's pending read acquire never makes progress, and T2 cannot acquire {{last_frame.lock}} until T1 releases. Neither does.
h2. Impact

Once wedged, the entire display pipeline starves: the protocol-client thread (e.g. VNC) blocks on {{pending_frame.lock}} trying to mutate a layer; per-user add/remove paths block on the client-level users-rwlock. Viewers see a black screen — no draws flow for the rest of the connection.

The race is intermittent and timing-dependent: it requires {{join_pending_handler}} firing while {{end_multiple_frames}} is past its {{defer_frame}} early-return and into the second lock acquire.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)