Window-Based Flow Control Performance Question

["Steigerwald, Aaron" <as...@brandesassociates.com.INVALID>]

Hello,

Does it make sense that setting consumerWindowSize=0 on the client's connection URL in order to disable client message buffering would improve message throughput performance given the following scenario:

Artemis version: 2.3.0
Number of queue consumers: 8 from 8 different systems, some of which are geographically separated from the broker but have low latency network connections.
Message type: byte
Average message size: less than 10 kilobytes

Thank you,
Aaron Steigerwald

Re: [EXTERNAL]:Re: Window-Based Flow Control Performance Question

["Steigerwald, Aaron" <as...@brandesassociates.com.INVALID>]

Hello Justin,

Thank you very much for all the info. I think the team feels it must have been consumer starvation. Whatever the reason things are much better now.

Thanks again,
Aaron

________________________________
From: Justin Bertram <jb...@apache.org>
Sent: Sunday, March 20, 2022 10:18 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: [EXTERNAL]:Re: Window-Based Flow Control Performance Question

[CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.]
________________________________


Window-based flow-control for consumers was designed to decrease network
round-trips and therefore increase performance. The classic example is a
consumer receiving lots of small messages over a relatively slow network
and processing those messages very quickly. In this kind of use-case
window-based flow-control can provide a significant performance boost by
reducing network round-trips. However, as the network speeds up or as the
messages get larger or as it takes longer to process those messages the
performance boost decreases. On the flip side, it's also possible to starve
consumers which will obviously hurt performance. There is a sweet-spot for
every use-case where optimal throughput can be achieved. The only way to
find that sweet-spot is with careful tuning.

As far as the "administrative cost" goes, I would say it's negligible. Do
you have a clear indication that the broker is using either more CPU or
more heap when the consumers buffer messages?

Lastly, I would recommend upgrading to 2.20.0 if you can support Java 11+
on the broker or 2.19.1 if you're stuck on Java 8. Version 2.3.0 is over 3
years old at this point.


Justin

On Fri, Mar 18, 2022 at 5:28 PM Steigerwald, Aaron
<as...@brandesassociates.com.invalid> wrote:

> Hello Justin,
>
> I mean overall performance. All 8 consumers are clones of one another
> (Docker containers) so, in theory, they should all perform more or less the
> same. We are trying to find ways to reduce message buildup in the queue
> when a surge of messages enter it. Setting consumerWindowSize=0 seems to
> have improved overall message throughput and I am trying to understand why.
> From what we can tell, all 8 systems process roughly the same number of
> messages per unit time.
>
> One of our theories is that the broker pays an administrative cost because
> it has to track which messages have been buffered (and not ACKed) for each
> consumer and disabling buffering eliminates such a cost.
>
> Thank you,
> Aaron
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Friday, March 18, 2022 6:04 PM
> To: asteigerwald@brandesassociates.com.invalid
> Cc: users@activemq.apache.org
> Subject: [EXTERNAL]:Re: Window-Based Flow Control Performance Question
>
> [CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.] ________________________________
>
>
> By "improve message throughput performance" are you talking about
> *overall* message throughput or throughput for a few selected consumers?
>
> Generally speaking, reducing consumerWindowSize will help when messages
> are accumulating on one consumer and other consumers have none (i.e.
> consumer starvation). This typically happens when you have a consumer that
> is slower than the others. The speed of the network between the consumer
> and the broker isn't as much of an issue as the speed at which the consumer
> is processing the messages.
>
>
> Justin
>
> On Fri, Mar 18, 2022 at 4:56 PM Steigerwald, Aaron
> <as...@brandesassociates.com.invalid> wrote:
>
> > Hello,
> >
> > Does it make sense that setting consumerWindowSize=0 on the client's
> > connection URL in order to disable client message buffering would
> > improve message throughput performance given the following scenario:
> >
> > Artemis version: 2.3.0
> > Number of queue consumers: 8 from 8 different systems, some of which
> > are geographically separated from the broker but have low latency
> > network connections.
> > Message type: byte
> > Average message size: less than 10 kilobytes
> >
> > Thank you,
> > Aaron Steigerwald
> >
>

Re: [EXTERNAL]:Re: Window-Based Flow Control Performance Question

[Justin Bertram <jb...@apache.org>]

Window-based flow-control for consumers was designed to decrease network
round-trips and therefore increase performance. The classic example is a
consumer receiving lots of small messages over a relatively slow network
and processing those messages very quickly. In this kind of use-case
window-based flow-control can provide a significant performance boost by
reducing network round-trips. However, as the network speeds up or as the
messages get larger or as it takes longer to process those messages the
performance boost decreases. On the flip side, it's also possible to starve
consumers which will obviously hurt performance. There is a sweet-spot for
every use-case where optimal throughput can be achieved. The only way to
find that sweet-spot is with careful tuning.

As far as the "administrative cost" goes, I would say it's negligible. Do
you have a clear indication that the broker is using either more CPU or
more heap when the consumers buffer messages?

Lastly, I would recommend upgrading to 2.20.0 if you can support Java 11+
on the broker or 2.19.1 if you're stuck on Java 8. Version 2.3.0 is over 3
years old at this point.


Justin

On Fri, Mar 18, 2022 at 5:28 PM Steigerwald, Aaron
<as...@brandesassociates.com.invalid> wrote:

> Hello Justin,
>
> I mean overall performance. All 8 consumers are clones of one another
> (Docker containers) so, in theory, they should all perform more or less the
> same. We are trying to find ways to reduce message buildup in the queue
> when a surge of messages enter it. Setting consumerWindowSize=0 seems to
> have improved overall message throughput and I am trying to understand why.
> From what we can tell, all 8 systems process roughly the same number of
> messages per unit time.
>
> One of our theories is that the broker pays an administrative cost because
> it has to track which messages have been buffered (and not ACKed) for each
> consumer and disabling buffering eliminates such a cost.
>
> Thank you,
> Aaron
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Friday, March 18, 2022 6:04 PM
> To: asteigerwald@brandesassociates.com.invalid
> Cc: users@activemq.apache.org
> Subject: [EXTERNAL]:Re: Window-Based Flow Control Performance Question
>
> [CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.] ________________________________
>
>
> By "improve message throughput performance" are you talking about
> *overall* message throughput or throughput for a few selected consumers?
>
> Generally speaking, reducing consumerWindowSize will help when messages
> are accumulating on one consumer and other consumers have none (i.e.
> consumer starvation). This typically happens when you have a consumer that
> is slower than the others. The speed of the network between the consumer
> and the broker isn't as much of an issue as the speed at which the consumer
> is processing the messages.
>
>
> Justin
>
> On Fri, Mar 18, 2022 at 4:56 PM Steigerwald, Aaron
> <as...@brandesassociates.com.invalid> wrote:
>
> > Hello,
> >
> > Does it make sense that setting consumerWindowSize=0 on the client's
> > connection URL in order to disable client message buffering would
> > improve message throughput performance given the following scenario:
> >
> > Artemis version: 2.3.0
> > Number of queue consumers: 8 from 8 different systems, some of which
> > are geographically separated from the broker but have low latency
> > network connections.
> > Message type: byte
> > Average message size: less than 10 kilobytes
> >
> > Thank you,
> > Aaron Steigerwald
> >
>

RE: [EXTERNAL]:Re: Window-Based Flow Control Performance Question

["Steigerwald, Aaron" <as...@brandesassociates.com.INVALID>]

Hello Justin,

I mean overall performance. All 8 consumers are clones of one another (Docker containers) so, in theory, they should all perform more or less the same. We are trying to find ways to reduce message buildup in the queue when a surge of messages enter it. Setting consumerWindowSize=0 seems to have improved overall message throughput and I am trying to understand why. From what we can tell, all 8 systems process roughly the same number of messages per unit time.

One of our theories is that the broker pays an administrative cost because it has to track which messages have been buffered (and not ACKed) for each consumer and disabling buffering eliminates such a cost.

Thank you,
Aaron

-----Original Message-----
From: Justin Bertram <jb...@apache.org> 
Sent: Friday, March 18, 2022 6:04 PM
To: asteigerwald@brandesassociates.com.invalid
Cc: users@activemq.apache.org
Subject: [EXTERNAL]:Re: Window-Based Flow Control Performance Question

[CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.] ________________________________


By "improve message throughput performance" are you talking about *overall* message throughput or throughput for a few selected consumers?

Generally speaking, reducing consumerWindowSize will help when messages are accumulating on one consumer and other consumers have none (i.e. consumer starvation). This typically happens when you have a consumer that is slower than the others. The speed of the network between the consumer and the broker isn't as much of an issue as the speed at which the consumer is processing the messages.


Justin

On Fri, Mar 18, 2022 at 4:56 PM Steigerwald, Aaron <as...@brandesassociates.com.invalid> wrote:

> Hello,
>
> Does it make sense that setting consumerWindowSize=0 on the client's 
> connection URL in order to disable client message buffering would 
> improve message throughput performance given the following scenario:
>
> Artemis version: 2.3.0
> Number of queue consumers: 8 from 8 different systems, some of which 
> are geographically separated from the broker but have low latency 
> network connections.
> Message type: byte
> Average message size: less than 10 kilobytes
>
> Thank you,
> Aaron Steigerwald
>

Re: Window-Based Flow Control Performance Question

[Justin Bertram <jb...@apache.org>]

By "improve message throughput performance" are you talking about *overall*
message throughput or throughput for a few selected consumers?

Generally speaking, reducing consumerWindowSize will help when messages are
accumulating on one consumer and other consumers have none (i.e. consumer
starvation). This typically happens when you have a consumer that is slower
than the others. The speed of the network between the consumer and the
broker isn't as much of an issue as the speed at which the consumer is
processing the messages.


Justin

On Fri, Mar 18, 2022 at 4:56 PM Steigerwald, Aaron
<as...@brandesassociates.com.invalid> wrote:

> Hello,
>
> Does it make sense that setting consumerWindowSize=0 on the client's
> connection URL in order to disable client message buffering would improve
> message throughput performance given the following scenario:
>
> Artemis version: 2.3.0
> Number of queue consumers: 8 from 8 different systems, some of which are
> geographically separated from the broker but have low latency network
> connections.
> Message type: byte
> Average message size: less than 10 kilobytes
>
> Thank you,
> Aaron Steigerwald
>