You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Rob Lugt <ro...@cheynecapital.com> on 2006/09/22 18:59:04 UTC
Topic consumer appears throttled
Hi
I'm hoping to use AMQ for a stock price publishing system, so high
throughput and low latency are important factors. To verify that the kind
of throughput we are after is achievable I've created a very simple
producer/consumer pair based on the sample code in
http://www.activemq.org/site/nms.html.
I've come across a potential problem where the consumer's receive rate
dramatically falls after the number of enqueued messages exceeds a certain
threshold. I'm pretty sure I must be falling foul of a configuration
problem because the number 1000 seems to be significant.
If the publisher sends 1000 messages, the consumer happilly receives them in
a couple fo seconds. But, if I configure the publisher to send 1500 messages
in quick succession, I see a perculiar pattern: the consumer receives the
first 300 messages quickly, it then receives the next 200 messages very
slowly (approx 5 msgs/s) until there are exactly 1000 messages left in the
queue and then it speeds through the last 1000 in just 2 seconds. If I
increase the number of messages sent, the same pattern remains i.e. the
consumer runs slowly until only 1000 messages remain then speeds up!
Topology:
Client 1 and Client 2 both running on a single workstation, connecting via
OpenWire using NMS
AMQ 4.0.1 Broker running on separate machine (Windows 2003 Server, J2SE
1.5.0_06)
Client 1 opens a session and publishes n (non-persistent) messages to a
single topic.
Client 2 opens a session and registers a listener on the same topic.
I've tried various options to overcome this, none of which seem to have had
any impact.
On connection uri I have: jms.useAsyncSend=true&consumer.dispatchAsync=false
On Topic name I have: jms.useAsyncSend=true&consumer.dispatchAsync=false
Any ideas/solutions gratefully received.
Best regards
Rob Lugt
--
View this message in context: http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6451007
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic consumer appears throttled
Posted by Rob Lugt <ro...@cheynecapital.com>.
> I wonder if you run a Java based consumer at the same time - does the
> Java consumer slow down too or is it just the C# one? If you just run
> the Java consumer, does it still slow down? Am wondering if its a C#
> specific issue
I tried the ConsumerTool from the Java examples. It doesn't suffer the same
fate, which probably explains why you've not seen this before.
> BTW I wonder if you get the same results with a java producer too?
Ah, good question. I tried publishing 1500 messages with the example
PublisherTool. This time the results were worse. The first time I ran it,
the c# program kept up with the flow, but it took a few seconds for the
publisher to publish the full 1500 messages. I reckoned this was nto a good
test because the consumer was probably never more than 1000 messages in
arrears, so I attempted to spped up the producer by redirecting the console
to a file like so:
ant producer > rob.out
This did indeed speed up the producer, but the results were not encouraging.
The Java producer completed okay, but the C# consumer only received 751
messages. I tried running the producer again (with the consumer still
active) but this time the producer hung after sending message 160 and sent
no more. After a few minutes I decided to kill it. When I then attempted to
close the consumer it also hung whilst attempting to disconect from the
broker.
I reran this scenario a few times. The results were random, but repeatable.
Occasionally the consumer received it's full compliment of messages and all
was fine, but usually it received a few hundred and then hung. The producer
would tend to hang some time shortly after the consumer.
Regards
~Rob
--
View this message in context: http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6486859
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic consumer appears throttled
Posted by James Strachan <ja...@gmail.com>.
On 9/23/06, Rob Lugt <ro...@cheynecapital.com> wrote:
>
> I modified the test program to call IMessage.Acknowledge, but unfortunately
> it didn't make any difference.
>
> One thing I've discovered that does make a difference is if the broker is
> moved onto the local machine. When all three components (broker, publisher
> and consumer) are on one machine, the message rate is fast and constant.
> Unfortunately this is no real help because it doesn't match our target
> environment.
I wonder if you run a Java based consumer at the same time - does the
Java consumer slow down too or is it just the C# one? If you just run
the Java consumer, does it still slow down? Am wondering if its a C#
specific issue
> I'm going to take a wild stab in the dark... Is it possible that my
> consumer is being identified as a "slow consumer", and therefore being given
> messages more slowly? I'm completely ignorant to the internal workings of
> the AMQ broker, so have little idea if this is likely - but I did read the
> article [1] which indicates slow consumers are given special attention and
> so the symptoms could possibly be explained by it. If this does turn out to
> be the case, it remains strange for (at least) two reasons:
> 1) the consumer is actually pretty fast, but even though it does nothing
> with the received message it can't consume quite consume at the rate that
> the producer can produce.
> 2) it seems the consumer slows down as soon as there are >1000 messages
> pending. I would expect this slowdown to occur at 32,766 which is, I
> understand, the prefetch limit for a non-durable topic[2].
>
> Anyway, to test this theory I added "?consumer.prefetchSize=2000" to the
> TOPIC name in the consumer. This made no difference :-( Can you confirm
> that all destination options are transported to the broker via NMS/OpenWire?
I'm afraid those options are only supported currently in the Java
client. We've not implemented those in C# yet.
> Another related thought struck me. How can I be sure that I've created a
> non-durable Topic? I use the ISession.GetTopic() method which magically
> creates the topic for me. I presume the result is a non-durable topic?
Things are persistent by default. To use non-persistent topics/queues
you set the delivery mode on a producer - which in NMS is the
Persistent property
http://incubator.apache.org/activemq/nms/ndoc/NMS.IMessageProducerMembers.html
> Thanks for the suggestions. Hopefully I've given a clearer picture of the
> problem.
BTW I wonder if you get the same results with a java producer too?
--
James
-------
http://radio.weblogs.com/0112098/
Re: Topic consumer appears throttled
Posted by Rob Lugt <ro...@cheynecapital.com>.
I modified the test program to call IMessage.Acknowledge, but unfortunately
it didn't make any difference.
One thing I've discovered that does make a difference is if the broker is
moved onto the local machine. When all three components (broker, publisher
and consumer) are on one machine, the message rate is fast and constant.
Unfortunately this is no real help because it doesn't match our target
environment.
I'm going to take a wild stab in the dark... Is it possible that my
consumer is being identified as a "slow consumer", and therefore being given
messages more slowly? I'm completely ignorant to the internal workings of
the AMQ broker, so have little idea if this is likely - but I did read the
article [1] which indicates slow consumers are given special attention and
so the symptoms could possibly be explained by it. If this does turn out to
be the case, it remains strange for (at least) two reasons:
1) the consumer is actually pretty fast, but even though it does nothing
with the received message it can't consume quite consume at the rate that
the producer can produce.
2) it seems the consumer slows down as soon as there are >1000 messages
pending. I would expect this slowdown to occur at 32,766 which is, I
understand, the prefetch limit for a non-durable topic[2].
Anyway, to test this theory I added "?consumer.prefetchSize=2000" to the
TOPIC name in the consumer. This made no difference :-( Can you confirm
that all destination options are transported to the broker via NMS/OpenWire?
Another related thought struck me. How can I be sure that I've created a
non-durable Topic? I use the ISession.GetTopic() method which magically
creates the topic for me. I presume the result is a non-durable topic?
Thanks for the suggestions. Hopefully I've given a clearer picture of the
problem.
Best regards
~Rob Lugt
[1] http://incubator.apache.org/activemq/slow-consumer-handling.html
[2] http://www.activemq.org/site/performance-tuning.html
James.Strachan wrote:
>
> I wonder have you tried calling IMessage.Acknowledge()?
>
> On 9/22/06, Rob Lugt <ro...@cheynecapital.com> wrote:
>>
>> Quick update to my earlier message. I did eventually find the overload
>> on
>> IConnection.CreateSession() to pass an AcknowledgementMode. I tried
>> setting
>> this to either AutoAcknowledge or DupsOkAcknowledge, but neither had an
>> affect on the apparent throttling of the consumer.
>>
>> ~Rob
>>
>>
>> Rob Lugt wrote:
>> >
>> > Hi James, thanks for the quick reply.
>> >
>> > Yes, I'm using NMS and C# for the client. I wasn't specifying any
>> > particular acknowledgement mode, so my previous tests would have been
>> > using the default - but your question prompted me to try a few things
>> out.
>> >
>> > I found this article which describes acknowledgement modes for
>> consumers:
>> >
>> http://www.activemq.org/site/activemq-performance-module-users-manual.html
>> >
>> > I couldn't find a way to directly manipulate the consumer
>> acknowledgement
>> > mode using the NMS classes (if there is a way please tell me), so I
>> used
>> > the URI syntax on the GetTopic() method e.g.
>> >
>> > IDestination destination =
>> >
>> session.GetTopic("PROD.PRICES.LIVE.IBM_LN?consumer.sessAckMode=autoAck");
>> >
>> > Sorry, I don't know if I've got the syntax right, but it didn't make
>> any
>> > difference to the tests. I also tried the value "dupsAck" because I
>> read
>> > somewhere that this could improve performance but it also didn't have
>> any
>> > affect.
>> >
>> > The listener is async (delegate added to consumer.Listener) - and is
>> very
>> > light-weight. I checked the CPU and memory activity on the client and
>> the
>> > server and both were comfortably under utilised.
>> >
>> > Best regards
>> > Rob Lugt
>> >
>> >
>> > James.Strachan wrote:
>> >>
>> >> This sounds like it could be related to acknowledgements - you're
>> >> using NMS and C# as the client right? What acknowledgement mode are
>> >> you using and are you using the async or sync listener?
>> >>
>> >> On 9/22/06, Rob Lugt <ro...@cheynecapital.com> wrote:
>> >>>
>> >>> Hi
>> >>>
>> >>> I'm hoping to use AMQ for a stock price publishing system, so high
>> >>> throughput and low latency are important factors. To verify that the
>> >>> kind
>> >>> of throughput we are after is achievable I've created a very simple
>> >>> producer/consumer pair based on the sample code in
>> >>> http://www.activemq.org/site/nms.html.
>> >>>
>> >>> I've come across a potential problem where the consumer's receive
>> rate
>> >>> dramatically falls after the number of enqueued messages exceeds a
>> >>> certain
>> >>> threshold. I'm pretty sure I must be falling foul of a configuration
>> >>> problem because the number 1000 seems to be significant.
>> >>>
>> >>> If the publisher sends 1000 messages, the consumer happilly receives
>> >>> them in
>> >>> a couple fo seconds. But, if I configure the publisher to send 1500
>> >>> messages
>> >>> in quick succession, I see a perculiar pattern: the consumer receives
>> >>> the
>> >>> first 300 messages quickly, it then receives the next 200 messages
>> very
>> >>> slowly (approx 5 msgs/s) until there are exactly 1000 messages left
>> in
>> >>> the
>> >>> queue and then it speeds through the last 1000 in just 2 seconds. If
>> I
>> >>> increase the number of messages sent, the same pattern remains i.e.
>> the
>> >>> consumer runs slowly until only 1000 messages remain then speeds up!
>> >>>
>> >>> Topology:
>> >>> Client 1 and Client 2 both running on a single workstation,
>> connecting
>> >>> via
>> >>> OpenWire using NMS
>> >>> AMQ 4.0.1 Broker running on separate machine (Windows 2003 Server,
>> >>> J2SE
>> >>> 1.5.0_06)
>> >>>
>> >>> Client 1 opens a session and publishes n (non-persistent) messages to
>> a
>> >>> single topic.
>> >>> Client 2 opens a session and registers a listener on the same topic.
>> >>>
>> >>> I've tried various options to overcome this, none of which seem to
>> have
>> >>> had
>> >>> any impact.
>> >>> On connection uri I have:
>> >>> jms.useAsyncSend=true&consumer.dispatchAsync=false
>> >>> On Topic name I have:
>> jms.useAsyncSend=true&consumer.dispatchAsync=false
>> >>>
>> >>> Any ideas/solutions gratefully received.
>> >>>
>> >>> Best regards
>> >>> Rob Lugt
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6451007
>> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >>
>> >> James
>> >> -------
>> >> http://radio.weblogs.com/0112098/
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6455875
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6467217
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic consumer appears throttled
Posted by James Strachan <ja...@gmail.com>.
I wonder have you tried calling IMessage.Acknowledge()?
On 9/22/06, Rob Lugt <ro...@cheynecapital.com> wrote:
>
> Quick update to my earlier message. I did eventually find the overload on
> IConnection.CreateSession() to pass an AcknowledgementMode. I tried setting
> this to either AutoAcknowledge or DupsOkAcknowledge, but neither had an
> affect on the apparent throttling of the consumer.
>
> ~Rob
>
>
> Rob Lugt wrote:
> >
> > Hi James, thanks for the quick reply.
> >
> > Yes, I'm using NMS and C# for the client. I wasn't specifying any
> > particular acknowledgement mode, so my previous tests would have been
> > using the default - but your question prompted me to try a few things out.
> >
> > I found this article which describes acknowledgement modes for consumers:
> > http://www.activemq.org/site/activemq-performance-module-users-manual.html
> >
> > I couldn't find a way to directly manipulate the consumer acknowledgement
> > mode using the NMS classes (if there is a way please tell me), so I used
> > the URI syntax on the GetTopic() method e.g.
> >
> > IDestination destination =
> > session.GetTopic("PROD.PRICES.LIVE.IBM_LN?consumer.sessAckMode=autoAck");
> >
> > Sorry, I don't know if I've got the syntax right, but it didn't make any
> > difference to the tests. I also tried the value "dupsAck" because I read
> > somewhere that this could improve performance but it also didn't have any
> > affect.
> >
> > The listener is async (delegate added to consumer.Listener) - and is very
> > light-weight. I checked the CPU and memory activity on the client and the
> > server and both were comfortably under utilised.
> >
> > Best regards
> > Rob Lugt
> >
> >
> > James.Strachan wrote:
> >>
> >> This sounds like it could be related to acknowledgements - you're
> >> using NMS and C# as the client right? What acknowledgement mode are
> >> you using and are you using the async or sync listener?
> >>
> >> On 9/22/06, Rob Lugt <ro...@cheynecapital.com> wrote:
> >>>
> >>> Hi
> >>>
> >>> I'm hoping to use AMQ for a stock price publishing system, so high
> >>> throughput and low latency are important factors. To verify that the
> >>> kind
> >>> of throughput we are after is achievable I've created a very simple
> >>> producer/consumer pair based on the sample code in
> >>> http://www.activemq.org/site/nms.html.
> >>>
> >>> I've come across a potential problem where the consumer's receive rate
> >>> dramatically falls after the number of enqueued messages exceeds a
> >>> certain
> >>> threshold. I'm pretty sure I must be falling foul of a configuration
> >>> problem because the number 1000 seems to be significant.
> >>>
> >>> If the publisher sends 1000 messages, the consumer happilly receives
> >>> them in
> >>> a couple fo seconds. But, if I configure the publisher to send 1500
> >>> messages
> >>> in quick succession, I see a perculiar pattern: the consumer receives
> >>> the
> >>> first 300 messages quickly, it then receives the next 200 messages very
> >>> slowly (approx 5 msgs/s) until there are exactly 1000 messages left in
> >>> the
> >>> queue and then it speeds through the last 1000 in just 2 seconds. If I
> >>> increase the number of messages sent, the same pattern remains i.e. the
> >>> consumer runs slowly until only 1000 messages remain then speeds up!
> >>>
> >>> Topology:
> >>> Client 1 and Client 2 both running on a single workstation, connecting
> >>> via
> >>> OpenWire using NMS
> >>> AMQ 4.0.1 Broker running on separate machine (Windows 2003 Server,
> >>> J2SE
> >>> 1.5.0_06)
> >>>
> >>> Client 1 opens a session and publishes n (non-persistent) messages to a
> >>> single topic.
> >>> Client 2 opens a session and registers a listener on the same topic.
> >>>
> >>> I've tried various options to overcome this, none of which seem to have
> >>> had
> >>> any impact.
> >>> On connection uri I have:
> >>> jms.useAsyncSend=true&consumer.dispatchAsync=false
> >>> On Topic name I have: jms.useAsyncSend=true&consumer.dispatchAsync=false
> >>>
> >>> Any ideas/solutions gratefully received.
> >>>
> >>> Best regards
> >>> Rob Lugt
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> >>> http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6451007
> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>>
> >>>
> >>
> >>
> >> --
> >>
> >> James
> >> -------
> >> http://radio.weblogs.com/0112098/
> >>
> >>
> >
> >
>
> --
> View this message in context: http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6455875
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Topic consumer appears throttled
Posted by Rob Lugt <ro...@cheynecapital.com>.
Quick update to my earlier message. I did eventually find the overload on
IConnection.CreateSession() to pass an AcknowledgementMode. I tried setting
this to either AutoAcknowledge or DupsOkAcknowledge, but neither had an
affect on the apparent throttling of the consumer.
~Rob
Rob Lugt wrote:
>
> Hi James, thanks for the quick reply.
>
> Yes, I'm using NMS and C# for the client. I wasn't specifying any
> particular acknowledgement mode, so my previous tests would have been
> using the default - but your question prompted me to try a few things out.
>
> I found this article which describes acknowledgement modes for consumers:
> http://www.activemq.org/site/activemq-performance-module-users-manual.html
>
> I couldn't find a way to directly manipulate the consumer acknowledgement
> mode using the NMS classes (if there is a way please tell me), so I used
> the URI syntax on the GetTopic() method e.g.
>
> IDestination destination =
> session.GetTopic("PROD.PRICES.LIVE.IBM_LN?consumer.sessAckMode=autoAck");
>
> Sorry, I don't know if I've got the syntax right, but it didn't make any
> difference to the tests. I also tried the value "dupsAck" because I read
> somewhere that this could improve performance but it also didn't have any
> affect.
>
> The listener is async (delegate added to consumer.Listener) - and is very
> light-weight. I checked the CPU and memory activity on the client and the
> server and both were comfortably under utilised.
>
> Best regards
> Rob Lugt
>
>
> James.Strachan wrote:
>>
>> This sounds like it could be related to acknowledgements - you're
>> using NMS and C# as the client right? What acknowledgement mode are
>> you using and are you using the async or sync listener?
>>
>> On 9/22/06, Rob Lugt <ro...@cheynecapital.com> wrote:
>>>
>>> Hi
>>>
>>> I'm hoping to use AMQ for a stock price publishing system, so high
>>> throughput and low latency are important factors. To verify that the
>>> kind
>>> of throughput we are after is achievable I've created a very simple
>>> producer/consumer pair based on the sample code in
>>> http://www.activemq.org/site/nms.html.
>>>
>>> I've come across a potential problem where the consumer's receive rate
>>> dramatically falls after the number of enqueued messages exceeds a
>>> certain
>>> threshold. I'm pretty sure I must be falling foul of a configuration
>>> problem because the number 1000 seems to be significant.
>>>
>>> If the publisher sends 1000 messages, the consumer happilly receives
>>> them in
>>> a couple fo seconds. But, if I configure the publisher to send 1500
>>> messages
>>> in quick succession, I see a perculiar pattern: the consumer receives
>>> the
>>> first 300 messages quickly, it then receives the next 200 messages very
>>> slowly (approx 5 msgs/s) until there are exactly 1000 messages left in
>>> the
>>> queue and then it speeds through the last 1000 in just 2 seconds. If I
>>> increase the number of messages sent, the same pattern remains i.e. the
>>> consumer runs slowly until only 1000 messages remain then speeds up!
>>>
>>> Topology:
>>> Client 1 and Client 2 both running on a single workstation, connecting
>>> via
>>> OpenWire using NMS
>>> AMQ 4.0.1 Broker running on separate machine (Windows 2003 Server,
>>> J2SE
>>> 1.5.0_06)
>>>
>>> Client 1 opens a session and publishes n (non-persistent) messages to a
>>> single topic.
>>> Client 2 opens a session and registers a listener on the same topic.
>>>
>>> I've tried various options to overcome this, none of which seem to have
>>> had
>>> any impact.
>>> On connection uri I have:
>>> jms.useAsyncSend=true&consumer.dispatchAsync=false
>>> On Topic name I have: jms.useAsyncSend=true&consumer.dispatchAsync=false
>>>
>>> Any ideas/solutions gratefully received.
>>>
>>> Best regards
>>> Rob Lugt
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6451007
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>>
>> James
>> -------
>> http://radio.weblogs.com/0112098/
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6455875
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic consumer appears throttled
Posted by Rob Lugt <ro...@cheynecapital.com>.
Hi James, thanks for the quick reply.
Yes, I'm using NMS and C# for the client. I wasn't specifying any
particular acknowledgement mode, so my previous tests would have been using
the default - but your question prompted me to try a few things out.
I found this article which describes acknowledgement modes for consumers:
http://www.activemq.org/site/activemq-performance-module-users-manual.html
I couldn't find a way to directly manipulate the consumer acknowledgement
mode using the NMS classes (if there is a way please tell me), so I used the
URI syntax on the GetTopic() method e.g.
IDestination destination =
session.GetTopic("PROD.PRICES.LIVE.IBM_LN?consumer.sessAckMode=autoAck");
Sorry, I don't know if I've got the syntax right, but it didn't make any
difference to the tests. I also tried the value "dupsAck" because I read
somewhere that this could improve performance but it also didn't have any
affect.
The listener is async (delegate added to consumer.Listener) - and is very
light-weight. I checked the CPU and memory activity on the client and the
server and both were comfortably under utilised.
Best regards
Rob Lugt
James.Strachan wrote:
>
> This sounds like it could be related to acknowledgements - you're
> using NMS and C# as the client right? What acknowledgement mode are
> you using and are you using the async or sync listener?
>
> On 9/22/06, Rob Lugt <ro...@cheynecapital.com> wrote:
>>
>> Hi
>>
>> I'm hoping to use AMQ for a stock price publishing system, so high
>> throughput and low latency are important factors. To verify that the
>> kind
>> of throughput we are after is achievable I've created a very simple
>> producer/consumer pair based on the sample code in
>> http://www.activemq.org/site/nms.html.
>>
>> I've come across a potential problem where the consumer's receive rate
>> dramatically falls after the number of enqueued messages exceeds a
>> certain
>> threshold. I'm pretty sure I must be falling foul of a configuration
>> problem because the number 1000 seems to be significant.
>>
>> If the publisher sends 1000 messages, the consumer happilly receives them
>> in
>> a couple fo seconds. But, if I configure the publisher to send 1500
>> messages
>> in quick succession, I see a perculiar pattern: the consumer receives the
>> first 300 messages quickly, it then receives the next 200 messages very
>> slowly (approx 5 msgs/s) until there are exactly 1000 messages left in
>> the
>> queue and then it speeds through the last 1000 in just 2 seconds. If I
>> increase the number of messages sent, the same pattern remains i.e. the
>> consumer runs slowly until only 1000 messages remain then speeds up!
>>
>> Topology:
>> Client 1 and Client 2 both running on a single workstation, connecting
>> via
>> OpenWire using NMS
>> AMQ 4.0.1 Broker running on separate machine (Windows 2003 Server, J2SE
>> 1.5.0_06)
>>
>> Client 1 opens a session and publishes n (non-persistent) messages to a
>> single topic.
>> Client 2 opens a session and registers a listener on the same topic.
>>
>> I've tried various options to overcome this, none of which seem to have
>> had
>> any impact.
>> On connection uri I have:
>> jms.useAsyncSend=true&consumer.dispatchAsync=false
>> On Topic name I have: jms.useAsyncSend=true&consumer.dispatchAsync=false
>>
>> Any ideas/solutions gratefully received.
>>
>> Best regards
>> Rob Lugt
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6451007
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6454526
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic consumer appears throttled
Posted by James Strachan <ja...@gmail.com>.
Many thanks for the patch Rob, I've applied it.
Incidentally if you can think of a cleaner way of implementing the
threading in .Net we'd welcome patches - to be perfectly honest I'm a
bit of a newbie C# developer so have been guessing a little in the
threading code. (It took a long time to figure out how to do the
equivalent of Java's wait() & notifiy() :)
On 9/25/06, Rob Lugt <ro...@cheynecapital.com> wrote:
>
>
> James.Strachan wrote:
> >
> > This sounds like it could be related to acknowledgements - you're
> > using NMS and C# as the client right? What acknowledgement mode are
> > you using and are you using the async or sync listener?
> >
>
> Hi James, your initial thought's on this were correct - the problem is due
> to acknowledgements. After finding that the Java client did not suffer the
> same problems, I took the plunge and debugged the C# NMS code.
>
> I've discovered 2 issues:-
>
> Firstly, the prefetch size for all consumers appears to be fixed at 1000 for
> .Net clients. The statement
>
> private int prefetchSize = 1000;
>
> in ActiveMQ\Session.cs gives the game away on that one. I tried modifying
> this to 2000 on my machine and hey-presto my test case worked. Of course I
> only had to increase the publisher to push out 2500 messaghes for the
> problem to return, so even though this was encouraging it was not the end of
> the story.
>
> Secondly, the real problem appears to be due to resource starvation within
> MessageConsumer.DispatchAsyncMessages(). This method will continue
> processing input messages as long as there are any messages queued, but the
> effect of this is that asynchroous acknowledgements will not be dispatched
> until all inbound messages have been processed. Adding a break statement
> into the function solves the problem. e.g:
>
> public void DispatchAsyncMessages()
> {
> while (listener != null)
> {
> IMessage message = dispatcher.DequeueNoWait();
> if (message != null)
> {
> //here we add the code that if do acknowledge action.
> message = AutoAcknowledge(message);
> listener(message);
> break; // RAL 25/09/2006: Return to allow queued messages to be
> processed
> }
> else
> {
> break;
> }
> }
> }
>
> So, that appears to solve my little problem, although I am still a little
> concerned about the instability this bug managed to cause my environment
> earlier today.
>
> Having taken a quick look through the NMS code I can see that I'm going to
> have other questions e.g.
> Use of ThreadPool - I was expecting all callbacks for a Session to occur on
> a single thread
> Lack of programability of the .Net classes (prefetch size is an example)
> ... but I guess the developer forum is a better location for questions of
> this sort.
>
> Best regards
> Rob Lugt
>
>
> --
> View this message in context: http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6491196
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Topic consumer appears throttled
Posted by Rob Lugt <ro...@cheynecapital.com>.
James.Strachan wrote:
>
> This sounds like it could be related to acknowledgements - you're
> using NMS and C# as the client right? What acknowledgement mode are
> you using and are you using the async or sync listener?
>
Hi James, your initial thought's on this were correct - the problem is due
to acknowledgements. After finding that the Java client did not suffer the
same problems, I took the plunge and debugged the C# NMS code.
I've discovered 2 issues:-
Firstly, the prefetch size for all consumers appears to be fixed at 1000 for
.Net clients. The statement
private int prefetchSize = 1000;
in ActiveMQ\Session.cs gives the game away on that one. I tried modifying
this to 2000 on my machine and hey-presto my test case worked. Of course I
only had to increase the publisher to push out 2500 messaghes for the
problem to return, so even though this was encouraging it was not the end of
the story.
Secondly, the real problem appears to be due to resource starvation within
MessageConsumer.DispatchAsyncMessages(). This method will continue
processing input messages as long as there are any messages queued, but the
effect of this is that asynchroous acknowledgements will not be dispatched
until all inbound messages have been processed. Adding a break statement
into the function solves the problem. e.g:
public void DispatchAsyncMessages()
{
while (listener != null)
{
IMessage message = dispatcher.DequeueNoWait();
if (message != null)
{
//here we add the code that if do acknowledge action.
message = AutoAcknowledge(message);
listener(message);
break; // RAL 25/09/2006: Return to allow queued messages to be
processed
}
else
{
break;
}
}
}
So, that appears to solve my little problem, although I am still a little
concerned about the instability this bug managed to cause my environment
earlier today.
Having taken a quick look through the NMS code I can see that I'm going to
have other questions e.g.
Use of ThreadPool - I was expecting all callbacks for a Session to occur on
a single thread
Lack of programability of the .Net classes (prefetch size is an example)
... but I guess the developer forum is a better location for questions of
this sort.
Best regards
Rob Lugt
--
View this message in context: http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6491196
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Topic consumer appears throttled
Posted by James Strachan <ja...@gmail.com>.
This sounds like it could be related to acknowledgements - you're
using NMS and C# as the client right? What acknowledgement mode are
you using and are you using the async or sync listener?
On 9/22/06, Rob Lugt <ro...@cheynecapital.com> wrote:
>
> Hi
>
> I'm hoping to use AMQ for a stock price publishing system, so high
> throughput and low latency are important factors. To verify that the kind
> of throughput we are after is achievable I've created a very simple
> producer/consumer pair based on the sample code in
> http://www.activemq.org/site/nms.html.
>
> I've come across a potential problem where the consumer's receive rate
> dramatically falls after the number of enqueued messages exceeds a certain
> threshold. I'm pretty sure I must be falling foul of a configuration
> problem because the number 1000 seems to be significant.
>
> If the publisher sends 1000 messages, the consumer happilly receives them in
> a couple fo seconds. But, if I configure the publisher to send 1500 messages
> in quick succession, I see a perculiar pattern: the consumer receives the
> first 300 messages quickly, it then receives the next 200 messages very
> slowly (approx 5 msgs/s) until there are exactly 1000 messages left in the
> queue and then it speeds through the last 1000 in just 2 seconds. If I
> increase the number of messages sent, the same pattern remains i.e. the
> consumer runs slowly until only 1000 messages remain then speeds up!
>
> Topology:
> Client 1 and Client 2 both running on a single workstation, connecting via
> OpenWire using NMS
> AMQ 4.0.1 Broker running on separate machine (Windows 2003 Server, J2SE
> 1.5.0_06)
>
> Client 1 opens a session and publishes n (non-persistent) messages to a
> single topic.
> Client 2 opens a session and registers a listener on the same topic.
>
> I've tried various options to overcome this, none of which seem to have had
> any impact.
> On connection uri I have: jms.useAsyncSend=true&consumer.dispatchAsync=false
> On Topic name I have: jms.useAsyncSend=true&consumer.dispatchAsync=false
>
> Any ideas/solutions gratefully received.
>
> Best regards
> Rob Lugt
>
>
>
> --
> View this message in context: http://www.nabble.com/Topic-consumer-appears-throttled-tf2318926.html#a6451007
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/