Dirty Rectangles - By FreeRDP or Guacamole

[Palaash Jain <pa...@gmail.com>]

Hi,

I've recently been exploring the images stored in Guacamole's recording
files, particularly those sent to the client as base64. By slightly
modifying the guacenc source code, I was able to save all the images to
disk and analyze the screen changes they represent. This has helped me
understand how screen updates are handled and recorded.

While reviewing the source code, I was curious to know whether Guacamole
simply accepts the image stream from the underlying FreeRDP library and
translates it directly into the Guacamole protocol, or if it performs
additional optimizations—such as identifying redundant or duplicate regions
and replacing them with copy or move instructions or techniques like bitmap
caching, etc.—to minimize recording size and improve streaming efficiency.

My ultimate goal is to fully understand Guacamole’s core architecture and
design a C#-based streaming solution that transmits dirty rectangles from a
target machine to any connected client.

This is my first experience diving into such a large-scale open-source
project, and I must say I’m very impressed by how clean and well-structured
Guacamole’s C code is. I’m really excited to continue learning from it.

This thread will really help the future beginners to be able to understand
the behaviour and techniques used by guacamole that makes it so much more
efficient and a light-weight clientless streaming solution.

Best regards,
Palaash Jain

Re: Dirty Rectangles - By FreeRDP or Guacamole

[Palaash Jain <pa...@gmail.com>]

Hi Mike,
Thanks a lot for your reply.

I am not sure if this is the right place to ask but I wanted to understand
the flow of code in drawing operations.

1) Guacamole subscribes to specific updates using OrderSupport (however I
have no idea what these updates are - perhaps 64x64 screen changes?)
2) All drawing operations are received in the
*guac_rdp_gdi_begin_paint* function
after which guacamole takes charge.
3) It stores all the dirty rectangles till invoking of
*guac_rdp_gdi_end_paint*.
4) The *guac_display_render_loop *along with
*guac_display_end_multiple_frames *listens to this and starts the magic.
i.e., the different passes.
5) Then it identifies empty color changes, 64x64 minimal dirty rects,
combining them and then rendering so that we don't have many threads that
can overload our server.

I would be really grateful if you can share as png images what exactly do
you receive from FreeRDP server and then what guacamole optimizes it into
for just a single sample screen.
I have set up FreeRDP on Windows solely for this and editing the SDL2
source code, I was able to get the changes but not able to get the ones
guacamole gets because for some reason the OrderSupport code isn't working.

FreeRDP:
[image: image.png]

Guacamole:
[image: image.png]

I am able to get updates like above from FreeRDP. If somehow, you can share
the png's of changes guacamole receives and what it transforms to (i.e.,
the final png only for a single frame to keep it easy), it would really be
a great help.

Regards,
Palaash Jain

On Sun, 20 Apr 2025 at 00:04, Michael Jumper <mj...@apache.org> wrote:

> On 4/19/25 2:10 AM, Palaash Jain wrote:
> > Hi,
> >
> > I've recently been exploring the images stored in Guacamole's recording
> > files, particularly those sent to the client as base64. By slightly
> > modifying the guacenc source code, I was able to save all the images to
> > disk and analyze the screen changes they represent. This has helped me
> > understand how screen updates are handled and recorded.
> >
> > While reviewing the source code, I was curious to know whether Guacamole
> > simply accepts the image stream from the underlying FreeRDP library and
> > translates it directly into the Guacamole protocol, or if it performs
> > additional optimizations—such as identifying redundant or duplicate
> > regions and replacing them with copy or move instructions or techniques
> > like bitmap caching, etc.—to minimize recording size and improve
> > streaming efficiency.
> >
>
> Good question - it does optimizations!
>
> The updates that come from the RDP server (or VNC server) are not always
> very efficient. Sometimes, the updates we receive are even broken up
> into individual single-pixel lines, and those really need to be combined
> before being sent for compression and decoding to be efficient. Just
> sending things directly as-is is much slower and uses more bandwidth.
>
> The optimizer used by guacd uses a cost estimation heuristic to
> determine whether to combine updates. It estimates the computational and
> network cost of sending updates as they stand vs. combining them, and
> then chooses whichever route appears to have the least cost. There is
> also continuous tracking of the performance of the client vs. server,
> to make sure that the server doesn't generate data faster than the
> client can consume.
>
> This has since been further improved. The optimizer in the upcoming
> 1.6.0 release is capable of recognizing copies/scrolling in real time,
> as well as contiguous regions of solid color, and optimizes things
> further. It also improves on the heuristics used for combining updates
> and does a better job of making reasonable choices.
>
> Here's a couple videos demonstrating the different versions of the
> optimizer:
>
> https://vimeo.com/1047655714 (before)
> https://vimeo.com/1047659972 (after)
>
> This is all pretty tricky, since these optimizations have to be done
> quickly enough that they don't contribute appreciable latency and defeat
> their own purpose.
>
> > My ultimate goal is to fully understand Guacamole’s core architecture
> > and design a C#-based streaming solution that transmits dirty rectangles
> > from a target machine to any connected client.
>
> Guacamole is intentionally designed with separate, reusable core
> components (libguac, guacamole-common, guacamole-common-js). You _can_
> incorporate Guacamole components into your own application.
>
> It would be better to leverage that and build off the established base,
> contribute back to those core components, etc. than to attempt to
> reimplement everything.
>
> >
> > This is my first experience diving into such a large-scale open-source
> > project, and I must say I’m very impressed by how clean and well-
> > structured Guacamole’s C code is. I’m really excited to continue
> > learning from it.
> >
>
> Thanks! This is a point of pride for us, and important for keeping
> things maintainable and understandable.
>
> - Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@guacamole.apache.org
> For additional commands, e-mail: user-help@guacamole.apache.org
>
>

Re: Dirty Rectangles - By FreeRDP or Guacamole

[Michael Jumper <mj...@apache.org>]

On 4/19/25 2:10 AM, Palaash Jain wrote:
> Hi,
> 
> I've recently been exploring the images stored in Guacamole's recording 
> files, particularly those sent to the client as base64. By slightly 
> modifying the guacenc source code, I was able to save all the images to 
> disk and analyze the screen changes they represent. This has helped me 
> understand how screen updates are handled and recorded.
> 
> While reviewing the source code, I was curious to know whether Guacamole 
> simply accepts the image stream from the underlying FreeRDP library and 
> translates it directly into the Guacamole protocol, or if it performs 
> additional optimizations—such as identifying redundant or duplicate 
> regions and replacing them with copy or move instructions or techniques 
> like bitmap caching, etc.—to minimize recording size and improve 
> streaming efficiency.
> 

Good question - it does optimizations!

The updates that come from the RDP server (or VNC server) are not always 
very efficient. Sometimes, the updates we receive are even broken up 
into individual single-pixel lines, and those really need to be combined 
before being sent for compression and decoding to be efficient. Just 
sending things directly as-is is much slower and uses more bandwidth.

The optimizer used by guacd uses a cost estimation heuristic to 
determine whether to combine updates. It estimates the computational and 
network cost of sending updates as they stand vs. combining them, and 
then chooses whichever route appears to have the least cost. There is 
also continuous tracking of the performance of the client vs. server, 
to make sure that the server doesn't generate data faster than the 
client can consume.

This has since been further improved. The optimizer in the upcoming 
1.6.0 release is capable of recognizing copies/scrolling in real time, 
as well as contiguous regions of solid color, and optimizes things 
further. It also improves on the heuristics used for combining updates 
and does a better job of making reasonable choices.

Here's a couple videos demonstrating the different versions of the 
optimizer:

https://vimeo.com/1047655714 (before)
https://vimeo.com/1047659972 (after)

This is all pretty tricky, since these optimizations have to be done 
quickly enough that they don't contribute appreciable latency and defeat 
their own purpose.

> My ultimate goal is to fully understand Guacamole’s core architecture 
> and design a C#-based streaming solution that transmits dirty rectangles 
> from a target machine to any connected client.

Guacamole is intentionally designed with separate, reusable core 
components (libguac, guacamole-common, guacamole-common-js). You _can_ 
incorporate Guacamole components into your own application.

It would be better to leverage that and build off the established base, 
contribute back to those core components, etc. than to attempt to 
reimplement everything.

> 
> This is my first experience diving into such a large-scale open-source 
> project, and I must say I’m very impressed by how clean and well- 
> structured Guacamole’s C code is. I’m really excited to continue 
> learning from it.
> 

Thanks! This is a point of pride for us, and important for keeping 
things maintainable and understandable.

- Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@guacamole.apache.org
For additional commands, e-mail: user-help@guacamole.apache.org