You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Frank Baxter <fr...@baxres.org> on 2006/06/05 06:27:03 UTC
Slow consumer problem
I've read through the slow consumer issues, but I still can't figure out how
to solve this problem.
I'm using AMQ 4.0 with persistent queues. I've got several message
producers that are reading information from an external system and creating
JMS messages. They do this in bursts, sometimes with a high input rate.
The messages end up around 300K in size. I use auto-ack on these producers.
My consumers are slow, and sometimes are not even running. I'm expecting
the JMS system to queue the messages up in the persistent storage. When my
consumers are running they will pull messages off as they can. I use
client-ack on the consumers (they are actually a chain of consumer/producers
with decision points).
It seems that the default behavior for AMQ is to slow down my producers when
the messages queue up. Eventually (seems to be around 2500 messages) they
stop completely. This causes a problem with my initial producers because
they are also receiving inputs from the external (non JMS) system. With AMQ
they are failing because they are timing out talking to that other system.
What I'm looking for is a way to remove that auto-slow down logic in my
producers. I want these producers to write messages as fast as possible so
I don't impact my external system. Does anyone have any ideas on how I can
achieve this with AMQ?
BTW, I have this system implemented and working using JBoss MQ, but I'm
looking to replace it due to performance issues, especially when large
queues exist and the JBoss server restarts.
--
View this message in context: http://www.nabble.com/Slow-consumer-problem-t1733329.html#a4709829
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Slow consumer problem
Posted by Frank Baxter <fr...@baxres.org>.
I verified that the messages are persistent, but I'm still not having much
luck finding the problem. I didn't try building with the patch yet, but as
a short term work around I just turned off journaling in the configuration.
Even with journalling off, I still have the problem.
FYI, here's a sample message from the debug:
21:33:19,099 DEBUG [ActiveMQSession] Sending message: ActiveMQObjectMessage
{commandId = 0, responseRequired = false, messageId =
ID:gorbash-55551-1149730299753-3:1:8:1:1, originalDestination = null,
originalTransactionId = null, producerId =
ID:gorbash-55551-1149730299753-3:1:8:1, destination = queue://checkedinQ,
transactionId = null, expiration = 0, timestamp = 1149730399091, arrival =
0, correlationId = null, replyTo = null, persistent = true, type = null,
priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null,
compressed = false, userID = null, content =
org.apache.activeio.packet.ByteSequence@a5ba31, marshalledProperties = null,
dataStructure = null, redeliveryCounter = 0, size = 0, properties = null,
readOnlyProperties = true, readOnlyBody = true}
I don't know if makes a difference, but I'm using the AMQ RAR in Jboss to
start a VM broker. In the code I then do this:
theFactory = new ActiveMQConnectionFactory("vm://localhost");
to create the factory, then this to create and start my connections:
myConnection = (ActiveMQConnection)
theFactory.createQueueConnection();
myConnection.setClientID(name);
myConnection.start();
Everything seems to work fine as long as I don't have slow consumers. Once
the consumers slow down or stop, the producers grind to a halt.
Is it possible I have 2 vm://localhost brokers? i.e. I can see one created
when the RAR starts, but it almost looks like it's starting another one when
I call new ActiveMQConnectionFactory("vm://localhost"). Is that possible or
will it automatically use the broker created from the RAR? If it is
creating another one, how do I get it to use the broker I configured in the
RAR?
Just messing with the factory configuration I found that when I call
setObjectMessageSerializationDefered(true) on the factory, the producers run
without a problem (I was able to load up 500K messages). However, later,
when I restart the consumers to read the messages, they all come back as
null objects to the consumers. So I guess deferring it isn't good when you
have persistence on? As a side note, is there a place where all those
factory configuration options are documented? There isn't anything in the
Javadocs and I couldn't find the info elsewhere yet.
Any suggestions on what to try next?
--
View this message in context: http://www.nabble.com/Slow-consumer-problem-t1733329.html#a4764635
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Slow consumer problem
Posted by Frank Baxter <fr...@baxres.org>.
In my code, I do this:
QueueSender sender = session.createSender(queue);
sender.setDeliveryMode(DeliveryMode.PERSISTENT);
Is there anything I need to do with the configuration or is this sufficient?
I'll take a look at the latest build to see if it fixes the problem.
Thanks!
Frank
--
View this message in context: http://www.nabble.com/Slow-consumer-problem-t1733329.html#a4724271
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Slow consumer problem
Posted by D Howe <da...@sensis.com.au>.
Thanks James - this explains some of the problems we've been having.
I was looking at the http://activemq.codehaus.org/Download page (the first
page that comes up when you google for activemq), and it only has release
candidate 2. I've since noticed that
http://www.activemq.org/site/download.html has a release version, hence my
confusion over 4.0 not being final.
--
View this message in context: http://www.nabble.com/Slow-consumer-problem-t1733329.html#a4745588
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Slow consumer problem
Posted by James Strachan <ja...@gmail.com>.
On 6/6/06, D Howe <da...@sensis.com.au> wrote:
>
> We are getting similar problems in ActiveMQ 3.2.1. We have a similar
> scenario to Frank, where our consumers may be down (e.g. for maintenance)
> but our producers will continue to produce messages.
Unfortunately 3.x is cannot handle large queues that don't fit in
memory - its only strategy once the RAM buffer is used up is to block
producers until the consumers catch up. So if you want to deal with
fast producers and slow consumers on durable queues well I'd recommend
upgrading to 4.0.
> I was expecting that with the amount of journal capacity and the Oracle
> database that I would be able to get in the millions of entries into the
> queue when there was no consumers, and that they would simply "queue up".
The reason is the design trade off of performance; we've designed
ActiveMQ to be very fast (rather than doing a database query every
time a message is required to be dispatched etc). So 3.x keeps the
pending messages on a queue in RAM. As it happens its amazingly
complex efficiently dispatching messages from a massive queue
efficiently to an arbitrary number of consumers who maybe using
filters so it took us until 4.x to truly figure out how to do it well,
fast and to deal with arbitrarily massive queues. The dispatch model
of massive queues is one of the major reasons for the changes in 4.x
> What actually happens is that around 60,000 messages I start getting lots of
> messages in the log:
>
> 2006-06-02 16:21:22,369 [Thread-0 ] WARN
> ientQueueBoundedMessageManager - memory limit low - forced to remove expired
> messages: publish.request
>
> Around about 65,000 messages I start getting lots of:
>
> 2006-06-02 17:19:05,159 [ocalport=61616]] WARN MemoryBoundedQueue
> - Queue is full, waiting for it to be dequeued.
>
> At this point, my producer seems to lock up and everything stops.
Yes - you've hit the limitation of 3.x - I"d recommend going to 4.x.
We've gone tests with millions of messages on a queue and things still
work fine.
> Is there any way to configure ActiveMQ (3.2.x) to reliably queue up a number
> of messages, only limited by disk/database capacity?
No - you need to move to 4.x.
> The application that
> we are developing could produce average loads of 30 messages/second, so
> 65,000 messages is nothing. Even if we up the maxMem, we will still have a
> limit - and the bigger the maxMem, the worse the consequences are if we
> reach it as we will lose more messages.
>
> Upgrading to 4 is not an option for us at the moment, as we are in final
> stages of testing and don't want to have to re-test with a version that is
> not final release.
4.0 is final?
--
James
-------
http://radio.weblogs.com/0112098/
Re: Slow consumer problem
Posted by D Howe <da...@sensis.com.au>.
We are getting similar problems in ActiveMQ 3.2.1. We have a similar
scenario to Frank, where our consumers may be down (e.g. for maintenance)
but our producers will continue to produce messages. We also want the
message broker to be a buffer to absorb surges in load, until such time that
our consumers can catch up (e.g. during quiet times overnight). We are
sizing our message consuming for the average daily load, not the peak.
In the 3.2.1 release, the maxMem attribute on the broker definition seems to
control how many messages we can get into the queue. I am reasonably sure
that persistence is working, as I can see the journal files being updated
and I can see the messages in the ACTIVEMQ_MSGS table in the database
(Oracle). I have 10 x 40 MB log files and plenty of capacity in Oracle. My
messages are quite small (1 - 2 KB) and are text messages. I have my maxMem
set to the equivalent of 80 MB (in bytes).
I was expecting that with the amount of journal capacity and the Oracle
database that I would be able to get in the millions of entries into the
queue when there was no consumers, and that they would simply "queue up".
What actually happens is that around 60,000 messages I start getting lots of
messages in the log:
2006-06-02 16:21:22,369 [Thread-0 ] WARN
ientQueueBoundedMessageManager - memory limit low - forced to remove expired
messages: publish.request
Around about 65,000 messages I start getting lots of:
2006-06-02 17:19:05,159 [ocalport=61616]] WARN MemoryBoundedQueue
- Queue is full, waiting for it to be dequeued.
At this point, my producer seems to lock up and everything stops.
When I restart ActiveMQ, my messages don't seem to come back. I can see the
65,000 messages in the ACTIVEMQ_MSGS table in Oracle, but my consumer can't
seem to dequeue them. This seems to be covered in
http://issues.apache.org/activemq/browse/AMQ-402 but won't be fixed in the
release 3 codeline. I have tried restarting several times and removing the
journal files, but nothing gets them back.
Is there any way to configure ActiveMQ (3.2.x) to reliably queue up a number
of messages, only limited by disk/database capacity? The application that
we are developing could produce average loads of 30 messages/second, so
65,000 messages is nothing. Even if we up the maxMem, we will still have a
limit - and the bigger the maxMem, the worse the consequences are if we
reach it as we will lose more messages.
Upgrading to 4 is not an option for us at the moment, as we are in final
stages of testing and don't want to have to re-test with a version that is
not final release.
--
View this message in context: http://www.nabble.com/Slow-consumer-problem-t1733329.html#a4724627
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Slow consumer problem
Posted by James Strachan <ja...@gmail.com>.
BTW I wonder if you were hitting this issue which was just fixed in
subversion (thanks for the patch Danielius!)
http://issues.apache.org/activemq/browse/AMQ-736
On 6/5/06, James Strachan <ja...@gmail.com> wrote:
> BTW are you positive you are using persistent queues? i.e. setting hte
> delivery mode on your MessageProducer to be persistent?
>
> On 6/5/06, Frank Baxter <fr...@baxres.org> wrote:
> >
> > I've read through the slow consumer issues, but I still can't figure out how
> > to solve this problem.
> >
> > I'm using AMQ 4.0 with persistent queues. I've got several message
> > producers that are reading information from an external system and creating
> > JMS messages. They do this in bursts, sometimes with a high input rate.
> > The messages end up around 300K in size. I use auto-ack on these producers.
> >
> > My consumers are slow, and sometimes are not even running. I'm expecting
> > the JMS system to queue the messages up in the persistent storage. When my
> > consumers are running they will pull messages off as they can. I use
> > client-ack on the consumers (they are actually a chain of consumer/producers
> > with decision points).
> >
> > It seems that the default behavior for AMQ is to slow down my producers when
> > the messages queue up. Eventually (seems to be around 2500 messages) they
> > stop completely. This causes a problem with my initial producers because
> > they are also receiving inputs from the external (non JMS) system. With AMQ
> > they are failing because they are timing out talking to that other system.
> >
> > What I'm looking for is a way to remove that auto-slow down logic in my
> > producers. I want these producers to write messages as fast as possible so
> > I don't impact my external system. Does anyone have any ideas on how I can
> > achieve this with AMQ?
> >
> > BTW, I have this system implemented and working using JBoss MQ, but I'm
> > looking to replace it due to performance issues, especially when large
> > queues exist and the JBoss server restarts.
> >
> > --
> > View this message in context: http://www.nabble.com/Slow-consumer-problem-t1733329.html#a4709829
> > Sent from the ActiveMQ - User forum at Nabble.com.
> >
> >
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Slow consumer problem
Posted by James Strachan <ja...@gmail.com>.
BTW are you positive you are using persistent queues? i.e. setting hte
delivery mode on your MessageProducer to be persistent?
On 6/5/06, Frank Baxter <fr...@baxres.org> wrote:
>
> I've read through the slow consumer issues, but I still can't figure out how
> to solve this problem.
>
> I'm using AMQ 4.0 with persistent queues. I've got several message
> producers that are reading information from an external system and creating
> JMS messages. They do this in bursts, sometimes with a high input rate.
> The messages end up around 300K in size. I use auto-ack on these producers.
>
> My consumers are slow, and sometimes are not even running. I'm expecting
> the JMS system to queue the messages up in the persistent storage. When my
> consumers are running they will pull messages off as they can. I use
> client-ack on the consumers (they are actually a chain of consumer/producers
> with decision points).
>
> It seems that the default behavior for AMQ is to slow down my producers when
> the messages queue up. Eventually (seems to be around 2500 messages) they
> stop completely. This causes a problem with my initial producers because
> they are also receiving inputs from the external (non JMS) system. With AMQ
> they are failing because they are timing out talking to that other system.
>
> What I'm looking for is a way to remove that auto-slow down logic in my
> producers. I want these producers to write messages as fast as possible so
> I don't impact my external system. Does anyone have any ideas on how I can
> achieve this with AMQ?
>
> BTW, I have this system implemented and working using JBoss MQ, but I'm
> looking to replace it due to performance issues, especially when large
> queues exist and the JBoss server restarts.
>
> --
> View this message in context: http://www.nabble.com/Slow-consumer-problem-t1733329.html#a4709829
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/