You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Brian Ramprasad <br...@cs.utoronto.ca> on 2019/09/12 23:07:49 UTC
Artemis Consumers
Hi,
I am trying to improve the throughput of queue Consumers. At present the Consumers are much slower than the Producers resulting in messages building up inside a queue. From what I can see(maybe I’m wrong) it looks like messages are sent to consumers of a queue one at a time in a round robin fashion which is not very efficient. Do you know if there is a way to batch messages together when sending to a consumer? For example if I have 2 consumers, I want to send them batches of 500 msgs each.
If I need to implement this feature myself, does anyone know which java class I need to modify?
Thanks!
Brian R
Re: Artemis Consumers
Posted by Justin Bertram <jb...@apache.org>.
First you need to identify what kind of consumer you're using (i.e. API &
protocol). The broker supports multiple protocols and each protocol
potentially has multiple client implementations.
Also, please elaborate on:
- how you made your observations and drew your conclusions
- the message volume your working with
- the number of producers and consumers
Justin
On Thu, Sep 12, 2019 at 6:14 PM Brian Ramprasad <br...@cs.utoronto.ca>
wrote:
> Hi,
>
> I am trying to improve the throughput of queue Consumers. At present the
> Consumers are much slower than the Producers resulting in messages building
> up inside a queue. From what I can see(maybe I’m wrong) it looks like
> messages are sent to consumers of a queue one at a time in a round robin
> fashion which is not very efficient. Do you know if there is a way to batch
> messages together when sending to a consumer? For example if I have 2
> consumers, I want to send them batches of 500 msgs each.
> If I need to implement this feature myself, does anyone know which java
> class I need to modify?
>
> Thanks!
> Brian R
>
>
>
Re: Artemis Consumers
Posted by Arthur Naseef <ar...@amlinv.com>.
Talking rates of 10K - 250K msg/sec (messages, not bytes) - those are
impressive numbers in general.
I would be very interested in the system statistics on the broker, and all
of the messaging clients (producers and consumers), as well as the network
links, while this is running. Some resource in the solution is going to be
the bottleneck. The question is which resource; the following are key
candidates:
- User-Mode CPU Utilization
- System CPU Utilization
- IO-Wait CPU Utilization
- Memory (most easily detected by measuring garbage-collection stats
from the JVM; total GC time is a good indicator to start)
- Network Bandwidth (byte per second over the ethernet connections)
- Disk I/O bytes-per-second (read and written)
- Disk IOPs (operations per second)
All of these on the broker and on each messaging client can be telling.
Also note that this may be a case of reaching the limits of this particular
use-case as it is written. One of the big reasons adding more consumers
increases aggregate messaging performance is that most consumers are doing
some amount of processing that take significant time.
Thinking about possibilities - a reason you may see decreased producer
performance when adding consumers is due to contention for broker resources
- CPU, disk, memory, or network.
Hope this helps. I would be very curious to see any measurements.
Art
On Thu, Sep 26, 2019 at 1:59 AM yw yw <wy...@gmail.com> wrote:
> > I tried doing this but it seems after 4 consumers the aggregate
> consumption throughput does not go up just starts spreading the load across
> more consumers.
> I have 3 quad core physical machines each on a 1gb link.
>
> In our experience, increasing number of consumers should increase
> consumption throughput. Did the bottleneck reside on client side? Did you
> try to deploy more consumers on different machines?
>
> Brian Ramprasad <br...@cs.utoronto.ca> 于2019年9月19日周四 上午2:48写道：
>
> > Hi Everyone
> >
> > Thanks for all the feedback. I have put my producer and consumer code and
> > broker.xml file on a public GitHub repo:
> > https://github.com/brianr82/artemistesting.git
> > Please see my replies below.
> >
> >
> >
> > My reply to Arthur's questions:
> >
> > > - First, reproduce the problem outside of production if not already
> > done
> > I am running in an isolated test environment
> >
> > > - Increase the number of consumers on the queue
> > I tried doing this but it seems after 4 consumers the aggregate
> > consumption throughput does not go up just starts spreading the load
> across
> > more consumers.
> > I have 3 quad core physical machines each on a 1gb link.
> > > - Attach consumers that do no processing on the messages — only
> > Consumers don’t do any processing, they receive and ignore
> > > consume, ack, and discard - this will give a baseline for just how
> > fast the
> > > consumers can go between the lowest level of the application and the
> > broker
> > > - Consume and produce from the same machine to isolate away potential
> > > differences in system, I/O, and network across multiple systems
> > > - Measure network and system I/O on the systems (especially the
> > consumer
> > > systems)
> > It seems like when the producers are running and no consumers are
> > consuming, the throughput is high and then when consumers are added the
> > producer throughput falls by about 50%. The memory fills up overtime
> > because the consumers are slower than producers.
> > >
> > > The following information on the messaging use-case would help further:
> > >
> > > - What QoS is used for the messages (persistent vs non-persistent,
> > etc)?
> > When I set to persistent the throughput on the producer side is 50k msg
> > per second, non persistent its 250k msg per second. The consumer side
> > throughput is much slower 10k and 50k respectively.
> > > - Are message groups, selectors, or other messaging features used?
> > No groups, selectors or other features
> > > - Are transactions in-use? If so, are they XA?
> > Yes, I create transactions. On the producer side I use transactions to
> > group 500 messages and then commit. On the consumer side I commit every
> 500
> > messages received. (See GitHub code)
> > > - How large are the messages?
> > The messages are small where the payload is 100 bytes per message
> >
> >
> >
> >
> > My reply to Justin's questions:
> >
> >
> > First you need to identify what kind of consumer you're using (i.e. API &
> > protocol). The broker supports multiple protocols and each protocol
> > potentially has multiple client implementations.
> > I believe I am using the CORE protocol and API. Please see my GitHub repo
> >
> > Also, please elaborate on:
> > - how you made your observations and drew your conclusions
> > I log the throughput on the producer side by calculating the number of
> > messages sent every second, the same is done on the consumer side.
> > - the message volume your working with
> > When I set to the queue to persistent the throughput on the producer side
> > is 50k msg per second, non persistent its 250k msg per second. The
> consumer
> > side throughput is much slower 10k and 50k respectively.
> >
> >
> > - the number of producers and consumers
> > I have 4 producers and 4 consumers each running on different machines.
> The
> > single broker is on its own machine.
> >
> > Justin
> >
> >
> >
> >
> >
> >
> >
> > > On Sep 17, 2019, at 1:30 AM, Arthur Naseef <ar...@amlinv.com> wrote:
> > >
> > > While it is possible to improve throughput with batching acks in
> > > transactions, I would recommend to check some other things first. Also
> > > note that transactions can introduce other issues, as I have
> experienced.
> > >
> > > Here are things I recommend to try (forgive me if these have already
> been
> > > tried):
> > >
> > > - First, reproduce the problem outside of production if not already
> > done
> > > - Increase the number of consumers on the queue
> > > - Attach consumers that do no processing on the messages -- only
> > > consume, ack, and discard - this will give a baseline for just how
> > fast the
> > > consumers can go between the lowest level of the application and the
> > broker
> > > - Consume and produce from the same machine to isolate away potential
> > > differences in system, I/O, and network across multiple systems
> > > - Measure network and system I/O on the systems (especially the
> > consumer
> > > systems)
> > >
> > > The following information on the messaging use-case would help further:
> > >
> > > - What QoS is used for the messages (persistent vs non-persistent,
> > etc)?
> > > - Are message groups, selectors, or other messaging features used?
> > > - Are transactions in-use? If so, are they XA?
> > > - How large are the messages?
> > >
> > > Hope this helps.
> > >
> > > Art
> > >
> > >
> > > On Fri, Sep 13, 2019 at 4:20 PM Clebert Suconic <
> > clebert.suconic@gmail.com>
> > > wrote:
> > >
> > >> You can batch acks in transactions.
> > >>
> > >> On Thu, Sep 12, 2019 at 7:14 PM Brian Ramprasad <
> brianr@cs.utoronto.ca>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> I am trying to improve the throughput of queue Consumers. At present
> > the
> > >>> Consumers are much slower than the Producers resulting in messages
> > >> building
> > >>> up inside a queue. From what I can see(maybe I’m wrong) it looks like
> > >>> messages are sent to consumers of a queue one at a time in a round
> > robin
> > >>> fashion which is not very efficient. Do you know if there is a way to
> > >> batch
> > >>> messages together when sending to a consumer? For example if I have 2
> > >>> consumers, I want to send them batches of 500 msgs each.
> > >>> If I need to implement this feature myself, does anyone know which
> java
> > >>> class I need to modify?
> > >>>
> > >>> Thanks!
> > >>> Brian R
> > >>>
> > >>>
> > >>> --
> > >> Clebert Suconic
> > >>
> >
> >
>
Re: Artemis Consumers
Posted by yw yw <wy...@gmail.com>.
> I tried doing this but it seems after 4 consumers the aggregate
consumption throughput does not go up just starts spreading the load across
more consumers.
I have 3 quad core physical machines each on a 1gb link.
In our experience, increasing number of consumers should increase
consumption throughput. Did the bottleneck reside on client side? Did you
try to deploy more consumers on different machines?
Brian Ramprasad <br...@cs.utoronto.ca> 于2019年9月19日周四 上午2:48写道：
> Hi Everyone
>
> Thanks for all the feedback. I have put my producer and consumer code and
> broker.xml file on a public GitHub repo:
> https://github.com/brianr82/artemistesting.git
> Please see my replies below.
>
>
>
> My reply to Arthur's questions:
>
> > - First, reproduce the problem outside of production if not already
> done
> I am running in an isolated test environment
>
> > - Increase the number of consumers on the queue
> I tried doing this but it seems after 4 consumers the aggregate
> consumption throughput does not go up just starts spreading the load across
> more consumers.
> I have 3 quad core physical machines each on a 1gb link.
> > - Attach consumers that do no processing on the messages — only
> Consumers don’t do any processing, they receive and ignore
> > consume, ack, and discard - this will give a baseline for just how
> fast the
> > consumers can go between the lowest level of the application and the
> broker
> > - Consume and produce from the same machine to isolate away potential
> > differences in system, I/O, and network across multiple systems
> > - Measure network and system I/O on the systems (especially the
> consumer
> > systems)
> It seems like when the producers are running and no consumers are
> consuming, the throughput is high and then when consumers are added the
> producer throughput falls by about 50%. The memory fills up overtime
> because the consumers are slower than producers.
> >
> > The following information on the messaging use-case would help further:
> >
> > - What QoS is used for the messages (persistent vs non-persistent,
> etc)?
> When I set to persistent the throughput on the producer side is 50k msg
> per second, non persistent its 250k msg per second. The consumer side
> throughput is much slower 10k and 50k respectively.
> > - Are message groups, selectors, or other messaging features used?
> No groups, selectors or other features
> > - Are transactions in-use? If so, are they XA?
> Yes, I create transactions. On the producer side I use transactions to
> group 500 messages and then commit. On the consumer side I commit every 500
> messages received. (See GitHub code)
> > - How large are the messages?
> The messages are small where the payload is 100 bytes per message
>
>
>
>
> My reply to Justin's questions:
>
>
> First you need to identify what kind of consumer you're using (i.e. API &
> protocol). The broker supports multiple protocols and each protocol
> potentially has multiple client implementations.
> I believe I am using the CORE protocol and API. Please see my GitHub repo
>
> Also, please elaborate on:
> - how you made your observations and drew your conclusions
> I log the throughput on the producer side by calculating the number of
> messages sent every second, the same is done on the consumer side.
> - the message volume your working with
> When I set to the queue to persistent the throughput on the producer side
> is 50k msg per second, non persistent its 250k msg per second. The consumer
> side throughput is much slower 10k and 50k respectively.
>
>
> - the number of producers and consumers
> I have 4 producers and 4 consumers each running on different machines. The
> single broker is on its own machine.
>
> Justin
>
>
>
>
>
>
>
> > On Sep 17, 2019, at 1:30 AM, Arthur Naseef <ar...@amlinv.com> wrote:
> >
> > While it is possible to improve throughput with batching acks in
> > transactions, I would recommend to check some other things first. Also
> > note that transactions can introduce other issues, as I have experienced.
> >
> > Here are things I recommend to try (forgive me if these have already been
> > tried):
> >
> > - First, reproduce the problem outside of production if not already
> done
> > - Increase the number of consumers on the queue
> > - Attach consumers that do no processing on the messages -- only
> > consume, ack, and discard - this will give a baseline for just how
> fast the
> > consumers can go between the lowest level of the application and the
> broker
> > - Consume and produce from the same machine to isolate away potential
> > differences in system, I/O, and network across multiple systems
> > - Measure network and system I/O on the systems (especially the
> consumer
> > systems)
> >
> > The following information on the messaging use-case would help further:
> >
> > - What QoS is used for the messages (persistent vs non-persistent,
> etc)?
> > - Are message groups, selectors, or other messaging features used?
> > - Are transactions in-use? If so, are they XA?
> > - How large are the messages?
> >
> > Hope this helps.
> >
> > Art
> >
> >
> > On Fri, Sep 13, 2019 at 4:20 PM Clebert Suconic <
> clebert.suconic@gmail.com>
> > wrote:
> >
> >> You can batch acks in transactions.
> >>
> >> On Thu, Sep 12, 2019 at 7:14 PM Brian Ramprasad <br...@cs.utoronto.ca>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am trying to improve the throughput of queue Consumers. At present
> the
> >>> Consumers are much slower than the Producers resulting in messages
> >> building
> >>> up inside a queue. From what I can see(maybe I’m wrong) it looks like
> >>> messages are sent to consumers of a queue one at a time in a round
> robin
> >>> fashion which is not very efficient. Do you know if there is a way to
> >> batch
> >>> messages together when sending to a consumer? For example if I have 2
> >>> consumers, I want to send them batches of 500 msgs each.
> >>> If I need to implement this feature myself, does anyone know which java
> >>> class I need to modify?
> >>>
> >>> Thanks!
> >>> Brian R
> >>>
> >>>
> >>> --
> >> Clebert Suconic
> >>
>
>
Re: Artemis Consumers
Posted by Brian Ramprasad <br...@cs.utoronto.ca>.
Hi Everyone
Thanks for all the feedback. I have put my producer and consumer code and broker.xml file on a public GitHub repo: https://github.com/brianr82/artemistesting.git
Please see my replies below.
My reply to Arthur's questions:
> - First, reproduce the problem outside of production if not already done
I am running in an isolated test environment
> - Increase the number of consumers on the queue
I tried doing this but it seems after 4 consumers the aggregate consumption throughput does not go up just starts spreading the load across more consumers.
I have 3 quad core physical machines each on a 1gb link.
> - Attach consumers that do no processing on the messages — only
Consumers don’t do any processing, they receive and ignore
> consume, ack, and discard - this will give a baseline for just how fast the
> consumers can go between the lowest level of the application and the broker
> - Consume and produce from the same machine to isolate away potential
> differences in system, I/O, and network across multiple systems
> - Measure network and system I/O on the systems (especially the consumer
> systems)
It seems like when the producers are running and no consumers are consuming, the throughput is high and then when consumers are added the producer throughput falls by about 50%. The memory fills up overtime because the consumers are slower than producers.
>
> The following information on the messaging use-case would help further:
>
> - What QoS is used for the messages (persistent vs non-persistent, etc)?
When I set to persistent the throughput on the producer side is 50k msg per second, non persistent its 250k msg per second. The consumer side throughput is much slower 10k and 50k respectively.
> - Are message groups, selectors, or other messaging features used?
No groups, selectors or other features
> - Are transactions in-use? If so, are they XA?
Yes, I create transactions. On the producer side I use transactions to group 500 messages and then commit. On the consumer side I commit every 500 messages received. (See GitHub code)
> - How large are the messages?
The messages are small where the payload is 100 bytes per message
My reply to Justin's questions:
First you need to identify what kind of consumer you're using (i.e. API &
protocol). The broker supports multiple protocols and each protocol
potentially has multiple client implementations.
I believe I am using the CORE protocol and API. Please see my GitHub repo
Also, please elaborate on:
- how you made your observations and drew your conclusions
I log the throughput on the producer side by calculating the number of messages sent every second, the same is done on the consumer side.
- the message volume your working with
When I set to the queue to persistent the throughput on the producer side is 50k msg per second, non persistent its 250k msg per second. The consumer side throughput is much slower 10k and 50k respectively.
- the number of producers and consumers
I have 4 producers and 4 consumers each running on different machines. The single broker is on its own machine.
Justin
> On Sep 17, 2019, at 1:30 AM, Arthur Naseef <ar...@amlinv.com> wrote:
>
> While it is possible to improve throughput with batching acks in
> transactions, I would recommend to check some other things first. Also
> note that transactions can introduce other issues, as I have experienced.
>
> Here are things I recommend to try (forgive me if these have already been
> tried):
>
> - First, reproduce the problem outside of production if not already done
> - Increase the number of consumers on the queue
> - Attach consumers that do no processing on the messages -- only
> consume, ack, and discard - this will give a baseline for just how fast the
> consumers can go between the lowest level of the application and the broker
> - Consume and produce from the same machine to isolate away potential
> differences in system, I/O, and network across multiple systems
> - Measure network and system I/O on the systems (especially the consumer
> systems)
>
> The following information on the messaging use-case would help further:
>
> - What QoS is used for the messages (persistent vs non-persistent, etc)?
> - Are message groups, selectors, or other messaging features used?
> - Are transactions in-use? If so, are they XA?
> - How large are the messages?
>
> Hope this helps.
>
> Art
>
>
> On Fri, Sep 13, 2019 at 4:20 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> You can batch acks in transactions.
>>
>> On Thu, Sep 12, 2019 at 7:14 PM Brian Ramprasad <br...@cs.utoronto.ca>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to improve the throughput of queue Consumers. At present the
>>> Consumers are much slower than the Producers resulting in messages
>> building
>>> up inside a queue. From what I can see(maybe I’m wrong) it looks like
>>> messages are sent to consumers of a queue one at a time in a round robin
>>> fashion which is not very efficient. Do you know if there is a way to
>> batch
>>> messages together when sending to a consumer? For example if I have 2
>>> consumers, I want to send them batches of 500 msgs each.
>>> If I need to implement this feature myself, does anyone know which java
>>> class I need to modify?
>>>
>>> Thanks!
>>> Brian R
>>>
>>>
>>> --
>> Clebert Suconic
>>
Re: Artemis Consumers
Posted by Arthur Naseef <ar...@amlinv.com>.
While it is possible to improve throughput with batching acks in
transactions, I would recommend to check some other things first. Also
note that transactions can introduce other issues, as I have experienced.
Here are things I recommend to try (forgive me if these have already been
tried):
- First, reproduce the problem outside of production if not already done
- Increase the number of consumers on the queue
- Attach consumers that do no processing on the messages -- only
consume, ack, and discard - this will give a baseline for just how fast the
consumers can go between the lowest level of the application and the broker
- Consume and produce from the same machine to isolate away potential
differences in system, I/O, and network across multiple systems
- Measure network and system I/O on the systems (especially the consumer
systems)
The following information on the messaging use-case would help further:
- What QoS is used for the messages (persistent vs non-persistent, etc)?
- Are message groups, selectors, or other messaging features used?
- Are transactions in-use? If so, are they XA?
- How large are the messages?
Hope this helps.
Art
On Fri, Sep 13, 2019 at 4:20 PM Clebert Suconic <cl...@gmail.com>
wrote:
> You can batch acks in transactions.
>
> On Thu, Sep 12, 2019 at 7:14 PM Brian Ramprasad <br...@cs.utoronto.ca>
> wrote:
>
> > Hi,
> >
> > I am trying to improve the throughput of queue Consumers. At present the
> > Consumers are much slower than the Producers resulting in messages
> building
> > up inside a queue. From what I can see(maybe I’m wrong) it looks like
> > messages are sent to consumers of a queue one at a time in a round robin
> > fashion which is not very efficient. Do you know if there is a way to
> batch
> > messages together when sending to a consumer? For example if I have 2
> > consumers, I want to send them batches of 500 msgs each.
> > If I need to implement this feature myself, does anyone know which java
> > class I need to modify?
> >
> > Thanks!
> > Brian R
> >
> >
> > --
> Clebert Suconic
>
Re: Artemis Consumers
Posted by Clebert Suconic <cl...@gmail.com>.
You can batch acks in transactions.
On Thu, Sep 12, 2019 at 7:14 PM Brian Ramprasad <br...@cs.utoronto.ca>
wrote:
> Hi,
>
> I am trying to improve the throughput of queue Consumers. At present the
> Consumers are much slower than the Producers resulting in messages building
> up inside a queue. From what I can see(maybe I’m wrong) it looks like
> messages are sent to consumers of a queue one at a time in a round robin
> fashion which is not very efficient. Do you know if there is a way to batch
> messages together when sending to a consumer? For example if I have 2
> consumers, I want to send them batches of 500 msgs each.
> If I need to implement this feature myself, does anyone know which java
> class I need to modify?
>
> Thanks!
> Brian R
>
>
> --
Clebert Suconic