Variable/dynamic prefetch size

[sdonovan_uk <sd...@yahoo.com>]

We're trying to use ActiveMQ as the queue for a distributed set of
calculation servers (think many machines with 64-cores on-board).

Imagine a single queue with N consumers.  The only efficient way to scale is
to use consumer prefetch.  However.  Each of our messages can take between
3ms and minutes to process (there is a huge range).  When testing with 3ms
per message (with prefetch of 0 or 1) it is not scaling linearly past ~32
consumers.

Is there anyway (either through configuration, programming, etc.) to make
the prefetch a variable/dynamic value?  Ideally, I'd like to add hints to
the message.

Any suggestions would be much appreciated!

Sean



--
View this message in context: http://activemq.2283324.n4.nabble.com/Variable-dynamic-prefetch-size-tp4658760.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Variable/dynamic prefetch size

[Hiram Chirino <hi...@hiramchirino.com>]

You can setup per consumer prefetch sizes.  If you can send to a different
queue based on how long each message will take to process (for example
fastqueue and slowqueue)  Then you can setup consumers on fastqueue with a
large prefetch and consumer on slowqueue with a small prefetch.


On Mon, Nov 5, 2012 at 2:22 AM, sdonovan_uk <sd...@yahoo.com> wrote:

> We're trying to use ActiveMQ as the queue for a distributed set of
> calculation servers (think many machines with 64-cores on-board).
>
> Imagine a single queue with N consumers.  The only efficient way to scale
> is
> to use consumer prefetch.  However.  Each of our messages can take between
> 3ms and minutes to process (there is a huge range).  When testing with 3ms
> per message (with prefetch of 0 or 1) it is not scaling linearly past ~32
> consumers.
>
> Is there anyway (either through configuration, programming, etc.) to make
> the prefetch a variable/dynamic value?  Ideally, I'd like to add hints to
> the message.
>
> Any suggestions would be much appreciated!
>
> Sean
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Variable-dynamic-prefetch-size-tp4658760.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 

**

*Hiram Chirino*

*Engineering | Red Hat, Inc.*

*hchirino@redhat.com <hc...@redhat.com> | fusesource.com | redhat.com*

*skype: hiramchirino | twitter: @hiramchirino<http://twitter.com/hiramchirino>
*

*blog: Hiram Chirino's Bit Mojo <http://hiramchirino.com/blog/>*