Handling of simultaneous key-down and mouse-button events

[Felix Wolfheimer <f....@googlemail.com>]

I'm using an application on a remote desktop served by Guacamole 0.9.14
(Remote side: CentOS 7.5, MATE Desktop, TurboVNC 2.2, VirtualGL 2.6). The
application uses the following workflow:
If pressing the Spacebar (and holding it down) it shows an overlay on an
OpenGL window it renders. This overlay has three buttons. Each button
selects a different action for the 3D view (rotate, zoom, pan). So while
the spacebar is pressed, I'm moving the mouse to this overlay, press the
left mouse button to select the action, and then move the mouse (spacebar
and left mouse button still pressed at the same time) to perform the action
on the 3D view.

While a normal TurboVNC+VirtualGL connection works for this workflow, the
behavior gets weird when guacamole is on top of it. The overlay flickers
once I start moving the mouse (mouse button and spacebar pressed) and the
3D view doesn't perform the expected action. I suppose that guacamole has
some difficulty passing the events to the remote session. It seems like it
sends a series of key-down and key-release events instead of a single
key-down event at the start and a key-release event at the end (at least
the behavior of the remote application looks like this). Has anyone seen
such a behavior?

Re: Handling of simultaneous key-down and mouse-button events

[Felix Wolfheimer <f....@online.de>]

Hi Mike,

found a perfect way to get what I want from Guacamole and just wanted
to share what I did in case anyone runs into a similar issue.

In guacamole-common-js/src/main/webapp/modules/Keyboard.js 

there's a list (no_repeat) of keys for which the key-down event is not
repeatedly send to the server if a user presses the key and and holds
it down. I just added the spacebar key to this list and rebuild the
webapp. While the application on the remote side is now working as
expected, the drawback is that something like a text box of a remote
application also won't get multiple key-down events if the user holds
down the space-bar, i.e., if someone wants to type multiple spaces in
such a window he/she needs to press the spacebar for each space he/she
wnats to type on the remote side. For me that's perfectly fine. :-)
Thanks for your help, Mike!  

Re: Handling of simultaneous key-down and mouse-button events

[Felix Wolfheimer <f....@online.de>]

Hi Mike

thanks a lot for your input. I thought a bit about it and it seems that
the heuristics which makes sense here is:

Use the keyboard behavior as is if no mouse button is pressed.
If a mouse button is pressed, don't send repeated key events while the
mouse button is pressed.

Many CAD programs use combined key-down + mouse button-down + mouse
move events to navigate in their 3D views, so I think that this
behavior makes sense for at least this class of applications. I looked
at the Guacamole.Keyboard as well as Guacamole.Mouse implementations.
As I'm not a Javascript programmer I'm not completely sure whether I
understand all of this stuff correctly, but it seems to me that I might
get the described behavior by changing a function 
in guacamole/src/main/webapp/app/client/directives/guacClient.js:

// Translate local keydown events to remote keydown events if keyboard
is enabled
$scope.$on('guacKeydown', function keydownListener(event, keysym,
keyboard) {
     if ($scope.client.clientProperties.keyboardEnabled &&
!event.defaultPrevented && 
                  !($scope.client.mouse.currentState.left ||
$scope.client.mouse.currentState.middle ||
$scope.client.mouse.currentState.right)) 
                {
                    client.sendKeyEvent(1, keysym);
                    event.preventDefault();
                }
});

The additional "!($scope.client.mouse.currentState.left ||
$scope.client.mouse.currentState.middle ||
$scope.client.mouse.currentState.right)" should tell the client to not
send repeated key-down events if any mouse button is clicked. Does this
sound right?

Re: Handling of simultaneous key-down and mouse-button events

[Mike Jumper <mj...@apache.org>]

On Mon, Oct 22, 2018 at 1:47 PM Felix Wolfheimer <f....@online.de>
wrote:

> Hi Mike,
>
> thanks for confirming that this is a kind of known behavior. Is there a
> way to tweak the auto-repeat timer in Guacamole? I suppose that setting
> it to a high value might give the behavior I'd need for the application
> although this might lead to unexpected behavior (?) for other
> applications.


Yes. You'd be better off looking into developing some sort of autorepeat
debouncing heuristic within Guacamole.Keyboard and contributing that, as it
is something which would make sense to have even though the general case
should probably assume client-side autorepeat. The main issue is making
things work identically across the board (all browsers/platforms) and
avoiding breaking the expectations of applications and remote desktop
servers.

I wouldn't recommend just setting the timer interval to a high value.
Besides not working across all platforms, it's a nasty hack. ;)

- Mike

Re: Handling of simultaneous key-down and mouse-button events

[Felix Wolfheimer <f....@online.de>]

Hi Mike,

thanks for confirming that this is a kind of known behavior. Is there a
way to tweak the auto-repeat timer in Guacamole? I suppose that setting
it to a high value might give the behavior I'd need for the application
although this might lead to unexpected behavior (?) for other
applications. 

Re: Handling of simultaneous key-down and mouse-button events

[Mike Jumper <mj...@apache.org>]

On Mon, Oct 22, 2018, 08:02 Felix Wolfheimer <f....@googlemail.com>
wrote:

> I'm using an application on a remote desktop served by Guacamole 0.9.14
> (Remote side: CentOS 7.5, MATE Desktop, TurboVNC 2.2, VirtualGL 2.6). The
> application uses the following workflow:
> If pressing the Spacebar (and holding it down) it shows an overlay on an
> OpenGL window it renders. This overlay has three buttons. Each button
> selects a different action for the 3D view (rotate, zoom, pan). So while
> the spacebar is pressed, I'm moving the mouse to this overlay, press the
> left mouse button to select the action, and then move the mouse (spacebar
> and left mouse button still pressed at the same time) to perform the action
> on the 3D view.
>
> While a normal TurboVNC+VirtualGL connection works for this workflow, the
> behavior gets weird when guacamole is on top of it. The overlay flickers
> once I start moving the mouse (mouse button and spacebar pressed) and the
> 3D view doesn't perform the expected action. I suppose that guacamole has
> some difficulty passing the events to the remote session. It seems like it
> sends a series of key-down and key-release events instead of a single
> key-down event at the start and a key-release event at the end (at least
> the behavior of the remote application looks like this). Has anyone seen
> such a behavior?
>

Yes. Unfortunately, this isn't a bug per se, but a platform- and
browser-specific aspect of how key events are fired. The key events of
printable keys may autorepeat while held.

To unify things across platforms, as not all do this (or even reliably
report keyup), Guacamole also has its own autorepeat timer within the
keyboard handling. Automatically debouncing with client-side heuristics
would be another approach which could conceivably be implemented (and that
would be compatible with your intentions), but exactly how to do this
without breaking the expectations of protocols and applications which rely
on automatic key repeat would be tricky.

- Mike