[jira] [Updated] (GUACAMOLE-2293) Active connection record leaks in memory when the guacd socket close throws during disconnect

["Nachum Barcohen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GUACAMOLE-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nachum Barcohen updated GUACAMOLE-2293:
---------------------------------------
    Summary: Active connection record leaks in memory when the guacd socket close throws during disconnect  (was: Active connection records leak in memory when a WebSocket client disconnects abruptly during socket teardown)

> Active connection record leaks in memory when the guacd socket close throws during disconnect
> ---------------------------------------------------------------------------------------------
>
>                 Key: GUACAMOLE-2293
>                 URL: https://issues.apache.org/jira/browse/GUACAMOLE-2293
>             Project: Guacamole
>          Issue Type: Bug
>          Components: guacamole-auth-jdbc, guacamole-common
>    Affects Versions: 1.6.0
>            Reporter: Nachum Barcohen
>            Priority: Major
>
> h3. Symptoms
>  * The Active Sessions admin view shows disconnected sessions that never clear.
>  * "Kill session" (REST PATCH .../activeConnections with op:remove) returns HTTP 200, but the record reappears on refresh - the in-memory ActiveConnectionRecord is never removed.
>  * guacd has no matching child process; the ghost is purely in guacamole-app memory.
>  * Ghosts accumulate indefinitely (observed 83 leaked records, oldest ~35 days) and count against concurrency limits. With a balancing group at max_connections_per_user=1, a single own-ghost locks the user out of the whole group. Only restarting guacamole-app flushes them.
>  * Log signature: java.lang.IllegalStateException: Message will not be sent because the WebSocket session has been closed, at GuacamoleWebSocketTunnelEndpoint.sendInstruction.
> h3. Environment
> Guacamole 1.6.0 (official Docker image), guacamole-auth-jdbc (MySQL) + LDAP + TOTP + ban, behind a reverse proxy / tunnel that does not always forward a WebSocket close frame to the origin (observed with a Cloudflare Zero Trust tunnel). The guacamole-app container is run with podman --security-opt=no-new-privileges. Predominantly RDP connections.
> h3. Root cause
> At disconnect there are two teardown paths, and cleanup can be skipped on either:
>  # *Socket-close path (the actual leak).* @OnClose fires normally -> tunnel.close() -> ManagedInetGuacamoleSocket.close() (or the SSL variant). That method calls super.close() and only afterwards runs socketClosedTask - the ConnectionCleanupTask that removes the record from the in-memory ActiveConnectionMultimap and releases the concurrency seat. The cleanup call is {*}not in a finally{*}. When the underlying guacd socket is being torn down concurrently by the read thread, JDK 21's NioSocketImpl fabricates java.io.IOException: No such file or directory on the second close() (the close() syscall itself succeeds - verified via strace, close(fd)=0; the ENOENT is a JVM NIO double-close artifact, not a kernel or permission error). super.close() throws, the cleanup task is skipped, and the ActiveConnectionRecord leaks.
>  # *Read-thread path (secondary symptom).* The transfer thread in GuacamoleWebSocketTunnelEndpoint catches only IOException. If the client WS closes mid-send, remote.sendText() throws an unchecked IllegalStateException (a RuntimeException); the thread dies before closeConnection() runs, so @OnClose never fires either.
> h3. Conclusion
> It is a timing race between the read side closing the guacd socket and the
> @OnClose close. Running guacamole-app under podman no-new-privileges shifts JVM
> thread scheduling enough to tip the race toward the throwing order (cap-drop=ALL
> alone does not; only the NoNewPrivs bit matters). RDP's heavy frame bursts keep
> the writer busy at disconnect and widen the window; light VNC streams stay clean.
> This is scheduling, not a blocked syscall - strace during teardown shows zero
> EPERM/EACCES and zero failed socket calls, and Seccomp=2 is identical between
> clean and ghosting configs.
> h3. Steps to reproduce
>  # Run stock guacamole/guacamole:1.6.0 + guacd + MySQL, with guacamole-app started under --security-opt=no-new-privileges.
>  # Configure at least 2 RDP connections to reachable Windows hosts.
>  # Open 3 concurrent browser tabs to those RDP connections; let them connect (~15–30s).
>  # Close all tabs near-simultaneously (close-storm).
>  # Wait ~60s, then looks at the active connections list. Connections still there. You can also query the in-memory tracker as an admin: GET /api/session/data/mysql/activeConnections.
> A single sequential connect/disconnect does not leak; the leak needs concurrent
> live RDP sessions plus a near-simultaneous close-storm.
> h3. Fix
> Make cleanup order-independent so the record is always released:
>  # ManagedInetGuacamoleSocket.close() and ManagedSSLGuacamoleSocket.close() - run socketClosedTask in a finally, so cleanup runs even when super.close() throws on an already-torn-down socket. (ConnectionCleanupTask is already guarded to run once via an AtomicBoolean, so this is safe.)
>  # ActiveConnectionService.deleteObject() - drop the tunnel.isOpen() guard before tunnel.close(). For a connection whose peer already vanished isOpen() returns false, so "Kill session" was a no-op; always call close() so an admin can reap an existing  host. The guard doesn't make sense, if the tunnel.isOpen is false, there is no reason  to keep the record in memory.
>  # GuacamoleWebSocketTunnelEndpoint (hardening) - catch RuntimeException/Error in the transfer thread and close the WS session in a finally, guaranteeing @OnClose  fires regardless of how the thread terminates.
> Fixes 1-2 are the necessary fix; fix 3 is defensive hardening for the parallel
> read-thread symptom. I have validated on my production setup and the issue is gone.
> {*}Mailing-list context{*}: https://lists.apache.org/thread/fwppogp2rl6n6hcn2wq33t3fz0obn0wq



--
This message was sent by Atlassian Jira
(v8.20.10#820010)