You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by benj <ap...@benandi.com> on 2012/12/18 18:48:40 UTC
ActiveMQ slow consumer policy
Hi
I'm developing a finance application, and am evaluating ActiveMQ and various
other middlewares to see what works best for us.
I'm wondering if anybody can advise on the best way to setup ActiveMQ for
the following scenario. It's a pretty standard stock price distribution
system, but needs to cope with a variety of clients.
- I have one server, sending data to many clients.
- Some clients will be on the LAN. Others clients will be over a WAN with
much lower bandwidth and frequent TCP disconnections
- Clients on the LAN should receive updates as frequently as they are
available.
- Clients on the WAN should receive updates for each topic as fast as they
can, but where the bandwidth available is not enough, they should try to
receive only the latest message on each topic, discarding any intermediate
messages which have become obsolete by the time they are ready to receive
them.
- When a client disconnects and reconnects, it should receive the last value
on each topic, regardless of whether any new values are being sent.
At the moment I'm using <lastImageSubscriptionRecoveryPolicy/> but I can't
seem to prevent a large queue of messages building up on the server when a
client is slow to consume.
Is there another policy I can use to ensure the server only retains the last
message on each topic, and discards old messages as soon as a new one is
available?
Thanks in advance,
Ben
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ slow consumer policy
Posted by benj <ap...@benandi.com>.
Thanks - nms.prefetchPolicy.topicPrefetch=1 did the trick.
I'm seeing fairly long latencies when I crank up the producer rate - upto
about 500ms, and the broker is using over 500MB of memory. But it is
managing the queues properly.
gtully wrote
> I had a peek that your code, the problem is on the broker.
> The problem is that the prefetch value specified in broker config via the
> policy entry is not enforced by stomp, so the default topic prefetch value
> (maxInt-1) is in effect.
> You could validate that via jconsole jmx on the broker, looking at the
> topic subscriptions actual prefetch values. we want them to be 1
>
> To workaround this, you need to specify the topicPrefetch via your client.
> Either set the policy on the connection factory or via the brokerUrl nms.
> url params.
>
> A client supplied value will take effect and you should not see the build
> up of messages on slow subscriptions.
>
> I opened https://issues.apache.org/jira/browse/AMQ-4234 to track this.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4661012.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ slow consumer policy
Posted by benj <ap...@benandi.com>.
Thanks - nms.prefetchPolicy.topicPrefetch=1 did the trick.
I'm seeing fairly long latencies - upto about 500ms, and the broker is using
over 500MB of memory. But it is managing the queues properly.
gtully wrote
> I had a peek that your code, the problem is on the broker.
> The problem is that the prefetch value specified in broker config via the
> policy entry is not enforced by stomp, so the default topic prefetch value
> (maxInt-1) is in effect.
> You could validate that via jconsole jmx on the broker, looking at the
> topic subscriptions actual prefetch values. we want them to be 1
>
> To workaround this, you need to specify the topicPrefetch via your client.
> Either set the policy on the connection factory or via the brokerUrl nms.
> url params.
>
> A client supplied value will take effect and you should not see the build
> up of messages on slow subscriptions.
>
> I opened https://issues.apache.org/jira/browse/AMQ-4234 to track this.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4661011.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ slow consumer policy
Posted by Gary Tully <ga...@gmail.com>.
I had a peek that your code, the problem is on the broker.
The problem is that the prefetch value specified in broker config via the
policy entry is not enforced by stomp, so the default topic prefetch value
(maxInt-1) is in effect.
You could validate that via jconsole jmx on the broker, looking at the
topic subscriptions actual prefetch values. we want them to be 1
To workaround this, you need to specify the topicPrefetch via your client.
Either set the policy on the connection factory or via the brokerUrl nms.
url params.
A client supplied value will take effect and you should not see the build
up of messages on slow subscriptions.
I opened https://issues.apache.org/jira/browse/AMQ-4234 to track this.
On 19 December 2012 16:25, benj <ap...@benandi.com> wrote:
> That would be really helpful - thank you!
>
> Here's a zip with the VS2012 solution containing producer/consumer
> projects,
> as well as my activemq.xml: 201212activemqtest.zip
> <
> http://activemq.2283324.n4.nabble.com/file/n4660906/201212activemqtest.zip
> >
>
> I was running this between two machines, so you'll need to change the URI
> that the consumer connects to, unless your machine happens to be called
> benlaptop :-)
>
> Thanks,
>
> Ben
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4660906.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://redhat.com
http://blog.garytully.com
Re: ActiveMQ slow consumer policy
Posted by benj <ap...@benandi.com>.
That would be really helpful - thank you!
Here's a zip with the VS2012 solution containing producer/consumer projects,
as well as my activemq.xml: 201212activemqtest.zip
<http://activemq.2283324.n4.nabble.com/file/n4660906/201212activemqtest.zip>
I was running this between two machines, so you'll need to change the URI
that the consumer connects to, unless your machine happens to be called
benlaptop :-)
Thanks,
Ben
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4660906.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ slow consumer policy
Posted by Gary Tully <ga...@gmail.com>.
can you post your test code? I would like to give it a whirl to investigate.
On 19 December 2012 14:19, benj <ap...@benandi.com> wrote:
> Hi
>
> Sorry - I should have said that I did try preFetch="1", and I also updated
> the constantPendingMessage limit to 0.
>
> Here's my config:
> <policyEntry topic=">" topicPrefetch="1"
> alwaysRetroactive="true">
> <subscriptionRecoveryPolicy>
> <lastImageSubscriptionRecoveryPolicy/>
> </subscriptionRecoveryPolicy>
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="0"/>
> </pendingMessageLimitStrategy>
> <messageEvictionStrategy>
> <oldestMessageEvictionStrategy/>
> </messageEvictionStrategy>
> </policyEntry>
>
> But between the producer and consumer you can see a queue building up, and
> the latency increasing second by second:
>
> Producer, sending approx 5000 msgs/s
> Sending message: 3944, 3944 msg/s
> Sending message: 8302, 4358 msg/s
> Sending message: 13471, 5169 msg/s
> Sending message: 18763, 5292 msg/s
> Sending message: 24212, 5449 msg/s
> Sending message: 29331, 5119 msg/s
> Sending message: 35003, 5672 msg/s
> Sending message: 40666, 5663 msg/s
> Sending message: 46207, 5541 msg/s
> Sending message: 51914, 5707 msg/s
>
> Consumer, receiving only 500-1000 msgs/s, latency increasing, no messages
> being discarded
> Received message: 1525, 1526 msgs/s, latency 21 < 197 < 439
> Received message: 3289, 1764 msgs/s, latency 350 < 624 < 937
> Received message: 4586, 1297 msgs/s, latency 948 < 1240 < 1595
> Received message: 6390, 1804 msgs/s, latency 1591 < 1983 < 2239
> Received message: 7947, 1557 msgs/s, latency 2235 < 2485 < 2784
> Received message: 8958, 1011 msgs/s, latency 3102 < 3345 < 3625
> Received message: 9464, 506 msgs/s, latency 4422 < 4451 < 4503
> Received message: 9970, 506 msgs/s, latency 5014 < 5061 < 5121
> Received message: 10470, 500 msgs/s, latency 5751 < 5812 < 5873
> Received message: 10971, 501 msgs/s, latency 6409 < 6466 < 6527
>
> Each row is a 1 second sampling period
> Messages are simple incrementing ids
> Latency is given as min < average < max, in milliseconds
>
> So, I'm kind of at the end of things to try and get this working... but I
> hope there's something simple I missing
>
> Thanks again
>
> Ben
>
> From: gtully [via ActiveMQ]
> [mailto:ml-node+s2283324n4660890h15@n4.nabble.com]
> Sent: 19 December 2012 13:34
> To: benj
> Subject: Re: ActiveMQ slow consumer policy
>
> > 1000 topics with a prefetch of 100 is 100,000 pending messages in memory.
> >
> > To get real eager discarding try:
> >
> > <policyEntry topic=">" topicPrefetch="1" >
> > <subscriptionRecoveryPolicy>
> > <lastImageSubscriptionRecoveryPolicy/>
> > </subscriptionRecoveryPolicy>
> > <pendingMessageLimitStrategy>
> > <constantPendingMessageLimitStrategy limit="0"/>
> > </pendingMessageLimitStrategy>
> > ...
> >
> > so that it won't keep any messages pending when the consumer is not ready
> > to consume. this will reduce the dispatch work load on the topic and
> > ensure
> > fast consumer get all messages while slow consumers get gaps.
> >
> > The lastImageSubscriptionRecoveryPolicy is only for retroactive
> > returning/new consumers, the get the current last state when the connect.
> >
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4660900.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://redhat.com
http://blog.garytully.com
Re: ActiveMQ slow consumer policy
Posted by benj <ap...@benandi.com>.
Hi
Sorry - I should have said that I did try preFetch="1", and I also updated
the constantPendingMessage limit to 0.
Here's my config:
<policyEntry topic=">" topicPrefetch="1"
alwaysRetroactive="true">
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="0"/>
</pendingMessageLimitStrategy>
<messageEvictionStrategy>
<oldestMessageEvictionStrategy/>
</messageEvictionStrategy>
</policyEntry>
But between the producer and consumer you can see a queue building up, and
the latency increasing second by second:
Producer, sending approx 5000 msgs/s
Sending message: 3944, 3944 msg/s
Sending message: 8302, 4358 msg/s
Sending message: 13471, 5169 msg/s
Sending message: 18763, 5292 msg/s
Sending message: 24212, 5449 msg/s
Sending message: 29331, 5119 msg/s
Sending message: 35003, 5672 msg/s
Sending message: 40666, 5663 msg/s
Sending message: 46207, 5541 msg/s
Sending message: 51914, 5707 msg/s
Consumer, receiving only 500-1000 msgs/s, latency increasing, no messages
being discarded
Received message: 1525, 1526 msgs/s, latency 21 < 197 < 439
Received message: 3289, 1764 msgs/s, latency 350 < 624 < 937
Received message: 4586, 1297 msgs/s, latency 948 < 1240 < 1595
Received message: 6390, 1804 msgs/s, latency 1591 < 1983 < 2239
Received message: 7947, 1557 msgs/s, latency 2235 < 2485 < 2784
Received message: 8958, 1011 msgs/s, latency 3102 < 3345 < 3625
Received message: 9464, 506 msgs/s, latency 4422 < 4451 < 4503
Received message: 9970, 506 msgs/s, latency 5014 < 5061 < 5121
Received message: 10470, 500 msgs/s, latency 5751 < 5812 < 5873
Received message: 10971, 501 msgs/s, latency 6409 < 6466 < 6527
Each row is a 1 second sampling period
Messages are simple incrementing ids
Latency is given as min < average < max, in milliseconds
So, I'm kind of at the end of things to try and get this working... but I
hope there's something simple I missing
Thanks again
Ben
From: gtully [via ActiveMQ]
[mailto:ml-node+s2283324n4660890h15@n4.nabble.com]
Sent: 19 December 2012 13:34
To: benj
Subject: Re: ActiveMQ slow consumer policy
> 1000 topics with a prefetch of 100 is 100,000 pending messages in memory.
>
> To get real eager discarding try:
>
> <policyEntry topic=">" topicPrefetch="1" >
> <subscriptionRecoveryPolicy>
> <lastImageSubscriptionRecoveryPolicy/>
> </subscriptionRecoveryPolicy>
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="0"/>
> </pendingMessageLimitStrategy>
> ...
>
> so that it won't keep any messages pending when the consumer is not ready
> to consume. this will reduce the dispatch work load on the topic and
> ensure
> fast consumer get all messages while slow consumers get gaps.
>
> The lastImageSubscriptionRecoveryPolicy is only for retroactive
> returning/new consumers, the get the current last state when the connect.
>
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4660900.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ slow consumer policy
Posted by Gary Tully <ga...@gmail.com>.
1000 topics with a prefetch of 100 is 100,000 pending messages in memory.
To get real eager discarding try:
<policyEntry topic=">" topicPrefetch="1" >
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="0"/>
</pendingMessageLimitStrategy>
...
so that it won't keep any messages pending when the consumer is not ready
to consume. this will reduce the dispatch work load on the topic and ensure
fast consumer get all messages while slow consumers get gaps.
The lastImageSubscriptionRecoveryPolicy is only for retroactive
returning/new consumers, the get the current last state when the connect.
On 19 December 2012 12:55, benj <ap...@benandi.com> wrote:
> Hi
>
> Thanks for the suggestion of using topicPrefetch.
>
> Unfortunately it doesn't seem to have any effect, I still get a large queue
> building up in the broker.
> I also getting the following message: "Producers will be throttled to the
> rate at which messages are removed from this destination"
>
> Instead of throttling the producer, I want the broker to discard messages
> in
> the queue so that only the last message on each topic is retained.
>
> I'm sure this is just me not understanding how to configure activemq
> properly...
> So any further suggestions much appreciated...!
>
> Thanks again,
>
> Ben
>
>
> From: gtully [via ActiveMQ]
> [mailto:ml-node+s2283324n4660887h22@n4.nabble.com]
> Sent: 19 December 2012 11:53
> To: benj
> Subject: Re: ActiveMQ slow consumer policy
>
> > configure a lower prefetch value for your topic consumers b/c the
> > discarding only takes place once the prefetch limit has been reached and
> > the default value is quite high.
> >
> > so something like <policyEntry topic=">" topicPrefetch=100 ...>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4660888.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://redhat.com
http://blog.garytully.com
Re: ActiveMQ slow consumer policy
Posted by benj <ap...@benandi.com>.
Hi
Thanks for the suggestion of using topicPrefetch.
Unfortunately it doesn't seem to have any effect, I still get a large queue
building up in the broker.
I also getting the following message: "Producers will be throttled to the
rate at which messages are removed from this destination"
Instead of throttling the producer, I want the broker to discard messages in
the queue so that only the last message on each topic is retained.
I'm sure this is just me not understanding how to configure activemq
properly...
So any further suggestions much appreciated...!
Thanks again,
Ben
From: gtully [via ActiveMQ]
[mailto:ml-node+s2283324n4660887h22@n4.nabble.com]
Sent: 19 December 2012 11:53
To: benj
Subject: Re: ActiveMQ slow consumer policy
> configure a lower prefetch value for your topic consumers b/c the
> discarding only takes place once the prefetch limit has been reached and
> the default value is quite high.
>
> so something like <policyEntry topic=">" topicPrefetch=100 ...>
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4660888.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ slow consumer policy
Posted by Gary Tully <ga...@gmail.com>.
configure a lower prefetch value for your topic consumers b/c the
discarding only takes place once the prefetch limit has been reached and
the default value is quite high.
so something like <policyEntry topic=">" topicPrefetch=100 ...>
On 19 December 2012 10:52, benj <ap...@benandi.com> wrote:
> Hi
>
> Thanks for your reply.
>
> I tried using the pendingMessageLimitStrategy, but I think I must be using
> it wrong as it seems to have no effect. I still get a large queue of
> pending
> messages on the server, while the client gets further and further behind.
> What I'm expecting is for the server to discard messages and for the client
> to see gaps in its stream, but never to fall too far behind.
>
> I'm using apache-activemq-5.7.0 with apache.nms.stomp-1.5.3 on Windows
> 7/2008.
>
> Here's my config:
>
> <broker persistent="false"
> xmlns="http://activemq.apache.org/schema/core" brokerName="localhost">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">">
> <subscriptionRecoveryPolicy>
> <lastImageSubscriptionRecoveryPolicy/>
> </subscriptionRecoveryPolicy>
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="1"/>
> </pendingMessageLimitStrategy>
> </policyEntry>
> ...
>
> I'm pumping short messages through at about 2000 messages per second,
> spread
> evenly across 1000 topics.
>
> I create the client first, and then run the server at 2000 msgs/s. I can
> see
> the client receiving only about 500 messages per second (with large
> stutters
> where it receives nothing for a few seconds). It appears to receive every
> message in sequence, so nothing is being discarded. Meanwhile the server
> gets further and further ahead, until the broker hits 1GB of main memory
> and
> then slows the producer down.
>
> Do I need to do something else to get the broker to discard messages more
> aggressively?
>
> Thanks again,
>
> Ben
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4660881.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://redhat.com
http://blog.garytully.com
Re: ActiveMQ slow consumer policy
Posted by benj <ap...@benandi.com>.
Hi
Thanks for your reply.
I tried using the pendingMessageLimitStrategy, but I think I must be using
it wrong as it seems to have no effect. I still get a large queue of pending
messages on the server, while the client gets further and further behind.
What I'm expecting is for the server to discard messages and for the client
to see gaps in its stream, but never to fall too far behind.
I'm using apache-activemq-5.7.0 with apache.nms.stomp-1.5.3 on Windows
7/2008.
Here's my config:
<broker persistent="false"
xmlns="http://activemq.apache.org/schema/core" brokerName="localhost">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">">
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1"/>
</pendingMessageLimitStrategy>
</policyEntry>
...
I'm pumping short messages through at about 2000 messages per second, spread
evenly across 1000 topics.
I create the client first, and then run the server at 2000 msgs/s. I can see
the client receiving only about 500 messages per second (with large stutters
where it receives nothing for a few seconds). It appears to receive every
message in sequence, so nothing is being discarded. Meanwhile the server
gets further and further ahead, until the broker hits 1GB of main memory and
then slows the producer down.
Do I need to do something else to get the broker to discard messages more
aggressively?
Thanks again,
Ben
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859p4660881.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ slow consumer policy
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi Ben,
you should add pending message limit strategy and eviction policy as
explained here
http://activemq.apache.org/slow-consumer-handling.html
Regards
--
Dejan Bosanac
----------------------
Red Hat, Inc.
FuseSource is now part of Red Hat
dbosanac@redhat.com
Twitter: @dejanb
Blog: http://sensatic.net
ActiveMQ in Action: http://www.manning.com/snyder/
On Tue, Dec 18, 2012 at 6:48 PM, benj <ap...@benandi.com> wrote:
> Hi
>
> I'm developing a finance application, and am evaluating ActiveMQ and various
> other middlewares to see what works best for us.
>
> I'm wondering if anybody can advise on the best way to setup ActiveMQ for
> the following scenario. It's a pretty standard stock price distribution
> system, but needs to cope with a variety of clients.
>
> - I have one server, sending data to many clients.
> - Some clients will be on the LAN. Others clients will be over a WAN with
> much lower bandwidth and frequent TCP disconnections
> - Clients on the LAN should receive updates as frequently as they are
> available.
> - Clients on the WAN should receive updates for each topic as fast as they
> can, but where the bandwidth available is not enough, they should try to
> receive only the latest message on each topic, discarding any intermediate
> messages which have become obsolete by the time they are ready to receive
> them.
> - When a client disconnects and reconnects, it should receive the last value
> on each topic, regardless of whether any new values are being sent.
>
> At the moment I'm using <lastImageSubscriptionRecoveryPolicy/> but I can't
> seem to prevent a large queue of messages building up on the server when a
> client is slow to consume.
>
> Is there another policy I can use to ensure the server only retains the last
> message on each topic, and discards old messages as soon as a new one is
> available?
>
> Thanks in advance,
>
> Ben
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slow-consumer-policy-tp4660859.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.