Slow consumer configuration

[Muzza <st...@amt-sybex.com>]

Hi,
First off I'm seriously starting to doubt ActiveMQ as a viable production
solution. Let me explain......

I have a J2EE 1.3 application Deployed within WAS 5.1. One producer puts
messages onto an ActiveMQ 4.0.1 queue and then an MDB consumes these
messages, processes them against a DB and forwards them onto a Response
queue.

I have a delay of about 9 seconds on the consumer, so I'm into the classic
slow consumer, fast producer scenario.

Can ActiveMQ 4.0.1 handle this scenario and if so what configuration changes
can I make to ensure it handles the situation ?

I have upgraded from 3.2.2 believing 4.0.1 would have this issue solved.
I've seen many posts enquiring about the Slow Consumer but never seen a
definitive answer to configuring this in ActiveMQ ? (Not hardcoding but
Configuring)

Mention of prefetch policys, prefetchsize and various Destination Policies
have been seen but can you once and for all produce instructions for solving
this situation ?

I believe in providing the above you will head off many more requests like
this as all that I want to be sure of is that I NEVER lose a message, yet
with pausing applications and bizarre memory usage I just don't feel
confident that ActiveMQ 4.X can provide a stable Production Queuing system.

Apologies if this is doom and gloom, but I've spent weeks on this having
upgraded from 3.2.2 to 4.0.1 and still can't get a configuration to suit
many customers.

Thanks,
Muzza.
-- 
View this message in context: http://www.nabble.com/Slow-consumer-configuration-tf2106258.html#a5805734
Sent from the ActiveMQ - User forum at Nabble.com.

RE: Slow consumer configuration

["Pesochinskiy, Vadim (MSCIBARRA)" <Va...@mscibarra.com>]

Sorry do not know much about Generic JMS provider. I hope it is generic
enough to allow you setting the queue name ;). 

-----Original Message-----
From: Muzza [mailto:stephen.murray@amt-sybex.com] 
Sent: Tuesday, August 15, 2006 10:00 AM
To: activemq-users@geronimo.apache.org
Subject: Re: Slow consumer configuration




Vadim Pesochinsky wrote:
> 
> I do not think you can configure it through the configuration file. If

> you want to avoid referencing amq code directly you can use the 
> following name in your call createQueue()
> 
> MY.QUEUE?consumer.prefetchSize=1
> 

I create my queues in WAS 5.1s Generic JMS provider, so I guess this
would be the place to do it then ?
--
View this message in context:
http://www.nabble.com/Slow-consumer-configuration-tf2106258.html#a581785
5
Sent from the ActiveMQ - User forum at Nabble.com.
--------------------------------------------------------

NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error.

Re: Slow consumer configuration

[Muzza <st...@amt-sybex.com>]

Vadim Pesochinsky wrote:
> 
> I do not think you can configure it through the configuration file. If you
> want to avoid referencing amq code directly you can use the following name
> in your call createQueue()
> 
> MY.QUEUE?consumer.prefetchSize=1
> 

I create my queues in WAS 5.1s Generic JMS provider, so I guess this would
be the place to do it then ?
-- 
View this message in context: http://www.nabble.com/Slow-consumer-configuration-tf2106258.html#a5817855
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Slow consumer configuration

[Vadim Pesochinsky <Va...@mscibarra.com>]

I do not think you can configure it through the configuration file. If you
want to avoid referencing amq code directly you can use the following name
in your call createQueue()

MY.QUEUE?consumer.prefetchSize=1
-- 
View this message in context: http://www.nabble.com/Slow-consumer-configuration-tf2106258.html#a5817667
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Slow consumer configuration

[Muzza <st...@amt-sybex.com>]

Hi James, that's fair comment, apologies.

My scenario is below :

Persistant Queues, using out of the box persistance.
Message size approx 110kb.
Max Message Volume on a requestQueue circa 50,000

1 Producer to a requestQueue
1 MDB (with say 10 threads configurable in Websphere AS 5.1) listening on
requestQueue.
MDBs pass message to Stateless Session Bean, which then retrieves data
against a database, it is here that the delay of approx 9 seconds occurs.
Based on the response from the database we can then deliver to an
exceptionQueue, generalExceptionQueue or responseQueue.

My application is J2EE1.3 spec and is deployed within WAS 5.1, as we can
connect to any JMS queue provider our code in the Application is generic and
we work off the QueueConnectionFactory to get to our queues.

My main problem is that I need a configuration of ActiveMQ whereby I can set
the prefetch limit to something really small, so if I have 10 MDBS perhaps
only a prefetch limit of 10 and I guess my frustration lies here, as I'm not
coding the connection to the queue specifically.

Should I set the consumer.preFetchsize=1 when configuring the queues in
Websphere ?

Or should I use the jms.preFetchPolicy.queuePrefetch=1 setting ?

Can I define my queues in ActiveMQ.xml and give them the above settings ?

As you have gleaned already I need a configuration that will simply Accept
messages onto a queue, which will build up to a volume of circa 50000
messages, and then have my consumer process them as and when it can, without
losing messages.

Again, apologies for my rant.
Muzza.



-- 
View this message in context: http://www.nabble.com/Slow-consumer-configuration-tf2106258.html#a5810362
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Slow consumer configuration

[James Strachan <ja...@gmail.com>]

On 8/14/06, Muzza <st...@amt-sybex.com> wrote:
> Hi,
> First off I'm seriously starting to doubt ActiveMQ as a viable production
> solution.

Gee thanks. Comments like that really motivate us volunteers here to
help you out with your problem..


> Let me explain......
>
> I have a J2EE 1.3 application Deployed within WAS 5.1. One producer puts
> messages onto an ActiveMQ 4.0.1 queue and then an MDB consumes these
> messages, processes them against a DB and forwards them onto a Response
> queue.
>
> I have a delay of about 9 seconds on the consumer, so I'm into the classic
> slow consumer, fast producer scenario.

Not really. Slow consumers are only really an issue for topics and
usually non-persistent topics at that.


> Can ActiveMQ 4.0.1 handle this scenario and if so what configuration changes
> can I make to ensure it handles the situation ?

All you are describing is can you load a queue with lots of messages
and load them faster than you can consume them - and yes that's
possible though like all things it depends on exactly what you are
doing, your platform, your configuration, your number of queues, how
you are using them, what messages you are sending, how big they are
etc


> I have upgraded from 3.2.2 believing 4.0.1 would have this issue solved.
> I've seen many posts enquiring about the Slow Consumer but never seen a
> definitive answer to configuring this in ActiveMQ ? (Not hardcoding but
> Configuring)

We're talking queues here I presume though you've not really expanded
too much on that. Persistent queues maybe?


> Mention of prefetch policys, prefetchsize and various Destination Policies
> have been seen but can you once and for all produce instructions for solving
> this situation ?

Messaging is kinda complex - there are so many ways in which it can be
used and so many different use cases and different issues related to
each. Maybe start with describing your actual problem and what your
issue is before assuming that other discussions are immediately
relevant to your situation or casting dispersions on ActiveMQ's
production stability?

-- 

James
-------
http://radio.weblogs.com/0112098/