You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Daniel <da...@gmail.com> on 2017/07/18 19:59:58 UTC
Performance. Single producer and consumer is slow - 2 producers and
2 consumers is fast.
I've set up ActiveMQ on a host.
I run one producer and one consumer. The producer is simply pumping 10 000
messages into a queue using a pooled connection. The consumer simply fetches
messages.
This is slow. When I add another consumer process nothing changes. When I
add another producer process (now there are 2 producers and 2 consumers) the
producers suddenly start pumping messages significantly faster and the
consumers receive faster.
I can't work out why this is. What explains this behavior?
I'm using persistent messages. I see the same behavior with consumer
prefetch of 1 of 0.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Performance-Single-producer-and-consumer-is-slow-2-producers-and-2-consumers-is-fast-tp4728648.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Performance. Single producer and consumer is slow - 2 producers
and 2 consumers is fast.
Posted by Tim Bain <tb...@alumni.duke.edu>.
That sounds like a very plausible theory, and one I hadn't thought of. Nice
work.
Tim
On Jul 25, 2017 10:51 AM, "Daniel" <da...@gmail.com> wrote:
> I'm referring to this part of the code:
> https://github.com/apache/activemq/blob/master/activemq-
> broker/src/main/java/org/apache/activemq/broker/region/Queue.java#L882
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Performance-Single-producer-and-consumer-is-slow-
> 2-producers-and-2-consumers-is-fast-tp4728648p4728883.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Performance. Single producer and consumer is slow - 2 producers
and 2 consumers is fast.
Posted by Daniel <da...@gmail.com>.
I'm referring to this part of the code:
https://github.com/apache/activemq/blob/master/activemq-broker/src/main/java/org/apache/activemq/broker/region/Queue.java#L882
--
View this message in context: http://activemq.2283324.n4.nabble.com/Performance-Single-producer-and-consumer-is-slow-2-producers-and-2-consumers-is-fast-tp4728648p4728883.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Performance. Single producer and consumer is slow - 2 producers
and 2 consumers is fast.
Posted by Daniel <da...@gmail.com>.
Thanks for the responses Tim. I was getting emails to my spam so I didn't see
them.
After some digging it appears I know the reason for the different
performance we see.
ActiveMQ has a threadpool of 1 thread persisting messages received to a
queue to disk. In the situation where we have multiple producers and
multiple consumers ActiveMQ will deliver messages which are still queued to
consumers. This means that with multiple producers and consumers we see
higher performance because we skip the disk write.
With 1 producer only one message is ever sent at a time and so the
threadpool of 1 is never full and receives never get messages directly from
the queue before they are persisted to disk.
Does this assessment make sense to you?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Performance-Single-producer-and-consumer-is-slow-2-producers-and-2-consumers-is-fast-tp4728648p4728882.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Performance. Single producer and consumer is slow - 2 producers
and 2 consumers is fast.
Posted by Tim Bain <tb...@alumni.duke.edu>.
Sorry, I missed the part where you said both the producers and the
consumers went faster. Are the consumers simply limited by the speed of the
producers (i.e. when the producer goes faster, so does the consumer)?
And do you see the same behavior with no consumers at all?
Tim
On Jul 18, 2017 10:44 PM, Tim Bain <tb...@alumni.duke.edu> wrote:
> I've set up ActiveMQ on a host.
> I run one producer and one consumer. The producer is simply pumping 10 000
> messages into a queue using a pooled connection. The consumer simply
> fetches
> messages.
>
> This is slow. When I add another consumer process nothing changes. When I
> add another producer process (now there are 2 producers and 2 consumers)
> the
> producers suddenly start pumping messages significantly faster and the
> consumers receive faster.
>
> I can't work out why this is. What explains this behavior?
>
> I'm using persistent messages. I see the same behavior with consumer
> prefetch of 1 of 0.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Performance-Single-producer-and-consumer-is-slow-
> 2-producers-and-2-consumers-is-fast-tp4728648.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Performance. Single producer and consumer is slow - 2 producers
and 2 consumers is fast.
Posted by Daniel <da...@gmail.com>.
Extra info: I'm running version 5.15.0.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Performance-Single-producer-and-consumer-is-slow-2-producers-and-2-consumers-is-fast-tp4728648p4728649.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Performance. Single producer and consumer is slow - 2 producers
and 2 consumers is fast.
Posted by Tim Bain <tb...@alumni.duke.edu>.
Nothing you've described jumps out as a clear reason for the behavior
you're seeing, and I've never heard anyone describe similar behavior.
Do you see the same behavior with one consumer? (i.e. 1:1 is slow, 2:1 is
fast?) And is it just as fast for three (or four) producers as for two, or
does it get even faster with more producers?
Also, can you characterize in more detail what "fast" and "slow" mean here?
Do the producers go faster, or do the consumers? How many messages per
second per consumer/producer?
Are any resources (CPU, memory, disk I/O, disk space, network I/O) on any
of the hosts involved (broker, producers, consumers) pegged or close to
pegged?
Is the behavior the same for a larger prefetch buffer (e.g. 100 or 1000)?
Tim
On Jul 18, 2017 2:19 PM, "Daniel" <da...@gmail.com> wrote:
I've set up ActiveMQ on a host.
I run one producer and one consumer. The producer is simply pumping 10 000
messages into a queue using a pooled connection. The consumer simply fetches
messages.
This is slow. When I add another consumer process nothing changes. When I
add another producer process (now there are 2 producers and 2 consumers) the
producers suddenly start pumping messages significantly faster and the
consumers receive faster.
I can't work out why this is. What explains this behavior?
I'm using persistent messages. I see the same behavior with consumer
prefetch of 1 of 0.
--
View this message in context: http://activemq.2283324.n4.
nabble.com/Performance-Single-producer-and-consumer-is-slow-
2-producers-and-2-consumers-is-fast-tp4728648.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.