You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by nkleinsch <nk...@lucidmedia.com> on 2009/10/07 00:01:58 UTC
Latency causing throughput problems in ActiveMQ 5.2
Hi all,
I'm having throughput issues on ActiveMQ 5.2 and was hoping for some tips to
improve performance. We're sending out messages on a topic at an average
rate of around 150/sec. The messages are object messages that vary in size,
but range up to 5 kilobytes. We're using a network of brokers between two
sites, with all the producers in one site and the consumers in the other.
We're seeing memory fill up on the producing site broker and the "messages
received" falling behind "messages sent", implying that we're not able to
send them over to the receiving side broker quickly enough. These messages
don't need guaranteed delivery, so we're not using transactions or durable
subscriptions.
I set up a test between the two sites using the ActiveMQ Performance Module.
I'm using a stock ActiveMQ 5.2 and the ActiveMQ Performance Module from the
latest SVN. I'm running ActiveMQ on a server in site 1. If I run the
producer and consumer on another server in site 1, I get a system average
throughput of 5184. If I run the same test, except with the client on a
similarly configured server in site 2, I get a throughput of 112.
The transfer rate between the two servers is upwards of 3 MB/s, so I assume
it isn't a bandwidth limitation. The latency between the two sites (measured
using ping) is 90ms. Why would latency have such a large effect on my
message throughput rate? Are there any settings I could change that would
help my performance?
Thanks,
Nick Kleinschmidt
--
View this message in context: http://www.nabble.com/Latency-causing-throughput-problems-in-ActiveMQ-5.2-tp25777286p25777286.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Latency causing throughput problems in ActiveMQ 5.2
Posted by nkleinsch <nk...@lucidmedia.com>.
I tried this as well, but didn't see any improvement. According to the wiki
page about preFetch the default is Short.MAX_VALUE-1 for non-persistent
topics.
rajdavies wrote:
>
> You should try increasing the prefetchSize for the network connection
> - by default its 1000 - setting it to a higher value should help -
> On 7 Oct 2009, at 19:52, nkleinsch wrote:
>
>>
>> Yes, I'm using the ActiveMQ performance module producer. It prints
>> back this
>> line on startup:
>>
>> INFO: Creating producer to: topic://topic.work.complete.main.0 with
>> non-persistent delivery.
>>
>> On our application I've also set non-persistent delivery and are
>> seeing the
>> same performance.
>>
>> - Nick
>>
>>
>> Joe Fernandez wrote:
>>>
>>> Are you setting the delivery mode on the publisher to be non-
>>> persistent?
>>>
>>> Joe
>>> http://www.ttmsolutions.com
>>>
>>>
>>> nkleinsch wrote:
>>>>
>>>> Adding some of the things I've tried in case they'd be of help to
>>>> other
>>>> people in the future.
>>>>
>>>> - Disabled producer flow control
>>>> (http://activemq.apache.org/producer-flow-control.html) - Did
>>>> nothing for
>>>> client throughput
>>>>
>>>> - Switched pendingSubscriberPolicy to a VM cursor
>>>> (http://activemq.apache.org/message-cursors.html) - Did nothing for
>>>> client throughput
>>>>
>>>> - Tried changing the prefetch limit
>>>> (http://activemq.apache.org/what-is-the-prefetch-limit-for.html) -
>>>> Did
>>>> nothing for client throughput. Given that the default was the max
>>>> value
>>>> for nonpersistent topics, I didn't think this would do anything, but
>>>> tried it just in case.
>>>>
>>>> - Set Async Sends on the producer
>>>> (http://activemq.apache.org/async-sends.html) - Did nothing for
>>>> client
>>>> throughput. This was also the default, but checked it to make sure.
>>>>
>>>> Is there anything else I haven't tried here that could help improve
>>>> performance? I've read through all the FAQs and performance
>>>> documents and
>>>> am running out of ideas.
>>>>
>>>> - Nick
>>>>
>>>>
>>>> nkleinsch wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm having throughput issues on ActiveMQ 5.2 and was hoping for
>>>>> some
>>>>> tips to improve performance. We're sending out messages on a
>>>>> topic at an
>>>>> average rate of around 150/sec. The messages are object messages
>>>>> that
>>>>> vary in size, but range up to 5 kilobytes. We're using a network of
>>>>> brokers between two sites, with all the producers in one site and
>>>>> the
>>>>> consumers in the other. We're seeing memory fill up on the
>>>>> producing
>>>>> site broker and the "messages received" falling behind "messages
>>>>> sent",
>>>>> implying that we're not able to send them over to the receiving
>>>>> side
>>>>> broker quickly enough. These messages don't need guaranteed
>>>>> delivery, so
>>>>> we're not using transactions or durable subscriptions.
>>>>>
>>>>> I set up a test between the two sites using the ActiveMQ
>>>>> Performance
>>>>> Module. I'm using a stock ActiveMQ 5.2 and the ActiveMQ Performance
>>>>> Module from the latest SVN. I'm running ActiveMQ on a server in
>>>>> site 1.
>>>>> If I run the producer and consumer on another server in site 1, I
>>>>> get a
>>>>> system average throughput of 5184. If I run the same test, except
>>>>> with
>>>>> the client on a similarly configured server in site 2, I get a
>>>>> throughput of 112.
>>>>>
>>>>> The transfer rate between the two servers is upwards of 3 MB/s,
>>>>> so I
>>>>> assume it isn't a bandwidth limitation. The latency between the two
>>>>> sites (measured using ping) is 90ms. Why would latency have such
>>>>> a large
>>>>> effect on my message throughput rate? Are there any settings I
>>>>> could
>>>>> change that would help my performance?
>>>>>
>>>>> Thanks,
>>>>> Nick Kleinschmidt
>>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Latency-causing-throughput-problems-in-ActiveMQ-5.2-tp25777286p25792204.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Latency-causing-throughput-problems-in-ActiveMQ-5.2-tp25777286p25863099.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Latency causing throughput problems in ActiveMQ 5.2
Posted by Rob Davies <ra...@gmail.com>.
You should try increasing the prefetchSize for the network connection
- by default its 1000 - setting it to a higher value should help -
On 7 Oct 2009, at 19:52, nkleinsch wrote:
>
> Yes, I'm using the ActiveMQ performance module producer. It prints
> back this
> line on startup:
>
> INFO: Creating producer to: topic://topic.work.complete.main.0 with
> non-persistent delivery.
>
> On our application I've also set non-persistent delivery and are
> seeing the
> same performance.
>
> - Nick
>
>
> Joe Fernandez wrote:
>>
>> Are you setting the delivery mode on the publisher to be non-
>> persistent?
>>
>> Joe
>> http://www.ttmsolutions.com
>>
>>
>> nkleinsch wrote:
>>>
>>> Adding some of the things I've tried in case they'd be of help to
>>> other
>>> people in the future.
>>>
>>> - Disabled producer flow control
>>> (http://activemq.apache.org/producer-flow-control.html) - Did
>>> nothing for
>>> client throughput
>>>
>>> - Switched pendingSubscriberPolicy to a VM cursor
>>> (http://activemq.apache.org/message-cursors.html) - Did nothing for
>>> client throughput
>>>
>>> - Tried changing the prefetch limit
>>> (http://activemq.apache.org/what-is-the-prefetch-limit-for.html) -
>>> Did
>>> nothing for client throughput. Given that the default was the max
>>> value
>>> for nonpersistent topics, I didn't think this would do anything, but
>>> tried it just in case.
>>>
>>> - Set Async Sends on the producer
>>> (http://activemq.apache.org/async-sends.html) - Did nothing for
>>> client
>>> throughput. This was also the default, but checked it to make sure.
>>>
>>> Is there anything else I haven't tried here that could help improve
>>> performance? I've read through all the FAQs and performance
>>> documents and
>>> am running out of ideas.
>>>
>>> - Nick
>>>
>>>
>>> nkleinsch wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I'm having throughput issues on ActiveMQ 5.2 and was hoping for
>>>> some
>>>> tips to improve performance. We're sending out messages on a
>>>> topic at an
>>>> average rate of around 150/sec. The messages are object messages
>>>> that
>>>> vary in size, but range up to 5 kilobytes. We're using a network of
>>>> brokers between two sites, with all the producers in one site and
>>>> the
>>>> consumers in the other. We're seeing memory fill up on the
>>>> producing
>>>> site broker and the "messages received" falling behind "messages
>>>> sent",
>>>> implying that we're not able to send them over to the receiving
>>>> side
>>>> broker quickly enough. These messages don't need guaranteed
>>>> delivery, so
>>>> we're not using transactions or durable subscriptions.
>>>>
>>>> I set up a test between the two sites using the ActiveMQ
>>>> Performance
>>>> Module. I'm using a stock ActiveMQ 5.2 and the ActiveMQ Performance
>>>> Module from the latest SVN. I'm running ActiveMQ on a server in
>>>> site 1.
>>>> If I run the producer and consumer on another server in site 1, I
>>>> get a
>>>> system average throughput of 5184. If I run the same test, except
>>>> with
>>>> the client on a similarly configured server in site 2, I get a
>>>> throughput of 112.
>>>>
>>>> The transfer rate between the two servers is upwards of 3 MB/s,
>>>> so I
>>>> assume it isn't a bandwidth limitation. The latency between the two
>>>> sites (measured using ping) is 90ms. Why would latency have such
>>>> a large
>>>> effect on my message throughput rate? Are there any settings I
>>>> could
>>>> change that would help my performance?
>>>>
>>>> Thanks,
>>>> Nick Kleinschmidt
>>>>
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Latency-causing-throughput-problems-in-ActiveMQ-5.2-tp25777286p25792204.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: Latency causing throughput problems in ActiveMQ 5.2
Posted by nkleinsch <nk...@lucidmedia.com>.
Yes, I'm using the ActiveMQ performance module producer. It prints back this
line on startup:
INFO: Creating producer to: topic://topic.work.complete.main.0 with
non-persistent delivery.
On our application I've also set non-persistent delivery and are seeing the
same performance.
- Nick
Joe Fernandez wrote:
>
> Are you setting the delivery mode on the publisher to be non-persistent?
>
> Joe
> http://www.ttmsolutions.com
>
>
> nkleinsch wrote:
>>
>> Adding some of the things I've tried in case they'd be of help to other
>> people in the future.
>>
>> - Disabled producer flow control
>> (http://activemq.apache.org/producer-flow-control.html) - Did nothing for
>> client throughput
>>
>> - Switched pendingSubscriberPolicy to a VM cursor
>> (http://activemq.apache.org/message-cursors.html) - Did nothing for
>> client throughput
>>
>> - Tried changing the prefetch limit
>> (http://activemq.apache.org/what-is-the-prefetch-limit-for.html) - Did
>> nothing for client throughput. Given that the default was the max value
>> for nonpersistent topics, I didn't think this would do anything, but
>> tried it just in case.
>>
>> - Set Async Sends on the producer
>> (http://activemq.apache.org/async-sends.html) - Did nothing for client
>> throughput. This was also the default, but checked it to make sure.
>>
>> Is there anything else I haven't tried here that could help improve
>> performance? I've read through all the FAQs and performance documents and
>> am running out of ideas.
>>
>> - Nick
>>
>>
>> nkleinsch wrote:
>>>
>>> Hi all,
>>>
>>> I'm having throughput issues on ActiveMQ 5.2 and was hoping for some
>>> tips to improve performance. We're sending out messages on a topic at an
>>> average rate of around 150/sec. The messages are object messages that
>>> vary in size, but range up to 5 kilobytes. We're using a network of
>>> brokers between two sites, with all the producers in one site and the
>>> consumers in the other. We're seeing memory fill up on the producing
>>> site broker and the "messages received" falling behind "messages sent",
>>> implying that we're not able to send them over to the receiving side
>>> broker quickly enough. These messages don't need guaranteed delivery, so
>>> we're not using transactions or durable subscriptions.
>>>
>>> I set up a test between the two sites using the ActiveMQ Performance
>>> Module. I'm using a stock ActiveMQ 5.2 and the ActiveMQ Performance
>>> Module from the latest SVN. I'm running ActiveMQ on a server in site 1.
>>> If I run the producer and consumer on another server in site 1, I get a
>>> system average throughput of 5184. If I run the same test, except with
>>> the client on a similarly configured server in site 2, I get a
>>> throughput of 112.
>>>
>>> The transfer rate between the two servers is upwards of 3 MB/s, so I
>>> assume it isn't a bandwidth limitation. The latency between the two
>>> sites (measured using ping) is 90ms. Why would latency have such a large
>>> effect on my message throughput rate? Are there any settings I could
>>> change that would help my performance?
>>>
>>> Thanks,
>>> Nick Kleinschmidt
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Latency-causing-throughput-problems-in-ActiveMQ-5.2-tp25777286p25792204.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Latency causing throughput problems in ActiveMQ 5.2
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
Are you setting the delivery mode on the publisher to be non-persistent?
Joe
http://www.ttmsolutions.com
nkleinsch wrote:
>
> Adding some of the things I've tried in case they'd be of help to other
> people in the future.
>
> - Disabled producer flow control
> (http://activemq.apache.org/producer-flow-control.html) - Did nothing for
> client throughput
>
> - Switched pendingSubscriberPolicy to a VM cursor
> (http://activemq.apache.org/message-cursors.html) - Did nothing for client
> throughput
>
> - Tried changing the prefetch limit
> (http://activemq.apache.org/what-is-the-prefetch-limit-for.html) - Did
> nothing for client throughput. Given that the default was the max value
> for nonpersistent topics, I didn't think this would do anything, but tried
> it just in case.
>
> - Set Async Sends on the producer
> (http://activemq.apache.org/async-sends.html) - Did nothing for client
> throughput. This was also the default, but checked it to make sure.
>
> Is there anything else I haven't tried here that could help improve
> performance? I've read through all the FAQs and performance documents and
> am running out of ideas.
>
> - Nick
>
>
> nkleinsch wrote:
>>
>> Hi all,
>>
>> I'm having throughput issues on ActiveMQ 5.2 and was hoping for some tips
>> to improve performance. We're sending out messages on a topic at an
>> average rate of around 150/sec. The messages are object messages that
>> vary in size, but range up to 5 kilobytes. We're using a network of
>> brokers between two sites, with all the producers in one site and the
>> consumers in the other. We're seeing memory fill up on the producing site
>> broker and the "messages received" falling behind "messages sent",
>> implying that we're not able to send them over to the receiving side
>> broker quickly enough. These messages don't need guaranteed delivery, so
>> we're not using transactions or durable subscriptions.
>>
>> I set up a test between the two sites using the ActiveMQ Performance
>> Module. I'm using a stock ActiveMQ 5.2 and the ActiveMQ Performance
>> Module from the latest SVN. I'm running ActiveMQ on a server in site 1.
>> If I run the producer and consumer on another server in site 1, I get a
>> system average throughput of 5184. If I run the same test, except with
>> the client on a similarly configured server in site 2, I get a throughput
>> of 112.
>>
>> The transfer rate between the two servers is upwards of 3 MB/s, so I
>> assume it isn't a bandwidth limitation. The latency between the two sites
>> (measured using ping) is 90ms. Why would latency have such a large effect
>> on my message throughput rate? Are there any settings I could change that
>> would help my performance?
>>
>> Thanks,
>> Nick Kleinschmidt
>>
>
>
--
View this message in context: http://www.nabble.com/Latency-causing-throughput-problems-in-ActiveMQ-5.2-tp25777286p25791255.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Latency causing throughput problems in ActiveMQ 5.2
Posted by nkleinsch <nk...@lucidmedia.com>.
Adding some of the things I've tried in case they'd be of help to other
people in the future.
- Disabled producer flow control
(http://activemq.apache.org/producer-flow-control.html) - Did nothing for
client throughput
- Switched pendingSubscriberPolicy to a VM cursor
(http://activemq.apache.org/message-cursors.html) - Did nothing for client
throughput
- Tried changing the prefetch limit
(http://activemq.apache.org/what-is-the-prefetch-limit-for.html) - Did
nothing for client throughput. Given that the default was the max value for
nonpersistent topics, I didn't think this would do anything, but tried it
just in case.
- Set Async Sends on the producer
(http://activemq.apache.org/async-sends.html) - Did nothing for client
throughput. This was also the default, but checked it to make sure.
Is there anything else I haven't tried here that could help improve
performance? I've read through all the FAQs and performance documents and am
running out of ideas.
- Nick
nkleinsch wrote:
>
> Hi all,
>
> I'm having throughput issues on ActiveMQ 5.2 and was hoping for some tips
> to improve performance. We're sending out messages on a topic at an
> average rate of around 150/sec. The messages are object messages that vary
> in size, but range up to 5 kilobytes. We're using a network of brokers
> between two sites, with all the producers in one site and the consumers in
> the other. We're seeing memory fill up on the producing site broker and
> the "messages received" falling behind "messages sent", implying that
> we're not able to send them over to the receiving side broker quickly
> enough. These messages don't need guaranteed delivery, so we're not using
> transactions or durable subscriptions.
>
> I set up a test between the two sites using the ActiveMQ Performance
> Module. I'm using a stock ActiveMQ 5.2 and the ActiveMQ Performance Module
> from the latest SVN. I'm running ActiveMQ on a server in site 1. If I run
> the producer and consumer on another server in site 1, I get a system
> average throughput of 5184. If I run the same test, except with the client
> on a similarly configured server in site 2, I get a throughput of 112.
>
> The transfer rate between the two servers is upwards of 3 MB/s, so I
> assume it isn't a bandwidth limitation. The latency between the two sites
> (measured using ping) is 90ms. Why would latency have such a large effect
> on my message throughput rate? Are there any settings I could change that
> would help my performance?
>
> Thanks,
> Nick Kleinschmidt
>
--
View this message in context: http://www.nabble.com/Latency-causing-throughput-problems-in-ActiveMQ-5.2-tp25777286p25791008.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.