[jira] [Created] (GUACAMOLE-1717) Cursor Improperly Initialized when Connecting to Gnome Remote Desktop via RDP

["Mike Powers (Jira)" <ji...@apache.org>]

Mike Powers created GUACAMOLE-1717:
--------------------------------------

             Summary: Cursor Improperly Initialized when Connecting to Gnome Remote Desktop via RDP
                 Key: GUACAMOLE-1717
                 URL: https://issues.apache.org/jira/browse/GUACAMOLE-1717
             Project: Guacamole
          Issue Type: Bug
          Components: RDP
    Affects Versions: 1.4.0
            Reporter: Mike Powers


When connecting to a [Gnome Remote Desktop|https://gitlab.gnome.org/GNOME/gnome-remote-desktop] RDP instance the mouse cursor is created but the image data is never copied into the cursor's pixel buffer resulting in a garbled bitmap containing uninitialized data:
!image-2022-11-20-19-59-57-478.png|width=664,height=311!

This appears to happen because of this guard condition in src/protocols/rdp/pointer.c:
```c
    if (pointer->andMaskData && pointer->xorMaskData)
```

It appears when Gnome Remote Desktop [initializes cursors|https://gitlab.gnome.org/GNOME/gnome-remote-desktop/-/blob/master/src/grd-session-rdp.c#L591] when starting the session it sets `andMaskData` to NULL with a length of zero:
```c
      pointer_color->lengthAndMask = 0;
      pointer_color->lengthXorMask = xor_mask_length;
      pointer_color->andMaskData = NULL;
```

This causes the guard condition in the RDP plugin to fail to copy bitmap data from the server. It appears as if FreeRDP [correctly handles|https://github.com/FreeRDP/FreeRDP/blob/eb7adaec46eceaa4b247dc664dcce2d65a5edead/libfreerdp/codec/color.c#L322] the image data in `freerdp_image_copy_from_pointer_data` if `lengthAndMask` == 0:
```c
	if (!xorMask || (xorMaskLength == 0))
		return FALSE;
	if (!andMask || (andMaskLength == 0))
		return FALSE;
```

I believe this guard condition is probably not necessary; removing it resolves the issue for me. I tested against both Gnome Remote Desktop and a Windows 11 Professional instance.

Please let me know if you'd like any additional information or testing done. This was tested with Guacamole 1.4.0 from Docker Hub (guacamole/guacamole:1.4.0 and guacamole/guacd:1.4.0). Testing against a fresh install of Fedora 37/Gnome 43.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)