artemis performance question

["Dondorp, Erwin" <er...@cgi.com>]

Hello,

We are performing a performance test by just putting a lot of messages through one queue.
With producer and consumer on one node we get around 2000 msg/sec.
But when producer and consumer are each on a separate cluster-node, it drops to 350 msg/sec.
This is strange because each node still essentially has to process each message once only.
And we see no obvious limits being violated (cpu, memory, network, etc).
Any clues/hints on why the difference is so big?

thx,
Erwin

RE: artemis performance question

["Dondorp, Erwin" <er...@cgi.com>]

Justin,

Thanks for your response!

Just a moment ago our infrastructure-guy told us that the cause is likely in the network between the nodes.
I have good hope that this is true because it is consistent with most/all symptoms.
(so I'll skip creating the reproducer-project for now)

> With performance testing the devil is in the details as they say
So true!

> do you mean that there is one cluster node with a consumer and another cluster node with a producer and messages are
> flowing from the producer to its cluster node and then to the other cluster node and ultimately to the consumer?
Yes; and the intention is that both "sides" of the cluster can grow as needed. But we first reduced our problem to just 2 nodes in total to make analysis easier.
To give a tiny bit of context: The "left" side behaves as DMZ. The "right" side is the normal/internal part. The user-list, group-list and address-permissions on each side are also a bit different.

Erwin

-----Oorspronkelijk bericht-----
Van: Justin Bertram <jb...@apache.org> 
Verzonden: woensdag 2 juni 2021 18:53
Aan: users@activemq.apache.org
Onderwerp: Re: artemis performance question


EXTERNAL SENDER:   Do not click any links or open any attachments unless you trust the sender and know the content is safe.
EXPÉDITEUR EXTERNE:    Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe à moins qu’ils ne proviennent d’un expéditeur fiable, ou que vous ayez l'assurance que le contenu provient d'une source sûre.

With performance testing the devil is in the details as they say. It would be ideal to have something to run which would reproduce the (relative) numbers you're seeing. Could you drop a project on GitHub or something to this end?

At the very least can you elaborate on your test setup? When you say that "producer and consumer are each on a separate cluster-node" do you mean that there is one cluster node with a consumer and another cluster node with a producer and messages are flowing from the producer to its cluster node and then to the other cluster node and ultimately to the consumer? If so, I would expect a sizable performance drop.


Justin

On Wed, Jun 2, 2021 at 2:56 AM Dondorp, Erwin <er...@cgi.com> wrote:

> Hello,
>
> We are performing a performance test by just putting a lot of messages 
> through one queue.
> With producer and consumer on one node we get around 2000 msg/sec.
> But when producer and consumer are each on a separate cluster-node, it 
> drops to 350 msg/sec.
> This is strange because each node still essentially has to process 
> each message once only.
> And we see no obvious limits being violated (cpu, memory, network, etc).
> Any clues/hints on why the difference is so big?
>
> thx,
> Erwin
>

Re: artemis performance question

[Justin Bertram <jb...@apache.org>]

With performance testing the devil is in the details as they say. It would
be ideal to have something to run which would reproduce the (relative)
numbers you're seeing. Could you drop a project on GitHub or something to
this end?

At the very least can you elaborate on your test setup? When you say that
"producer and consumer are each on a separate cluster-node" do you mean
that there is one cluster node with a consumer and another cluster node
with a producer and messages are flowing from the producer to its cluster
node and then to the other cluster node and ultimately to the consumer? If
so, I would expect a sizable performance drop.


Justin

On Wed, Jun 2, 2021 at 2:56 AM Dondorp, Erwin <er...@cgi.com> wrote:

> Hello,
>
> We are performing a performance test by just putting a lot of messages
> through one queue.
> With producer and consumer on one node we get around 2000 msg/sec.
> But when producer and consumer are each on a separate cluster-node, it
> drops to 350 msg/sec.
> This is strange because each node still essentially has to process each
> message once only.
> And we see no obvious limits being violated (cpu, memory, network, etc).
> Any clues/hints on why the difference is so big?
>
> thx,
> Erwin
>