Temporarily suspending prefetch on a message consumer.

[Kevin Burton <bu...@spinn3r.com>]

I have a cluster of ActiveMQ boxes and a network of consumers.

The problem I'm having now is that the prefetch on the consumers fetches
too many messages and then other machines in the cluster get starved out of
work.

What I really want is a pattern where I can have say 50 ActiveMQ brokers
and 200 worker servers each with their own set of consumers.  I want each
worker server to only be able read up to nrThreads (usually like 100-200)
to perform the work.  Reading more than nrThreads means its hogging work
which is preventing other workers from consuming these messages.

The only way I can really figure out a way to do this is to have the
prefetch temporarily suspended on some of the consumers.

I really with the ActiveMQ client were fully async via NIO/netty and not
simulated async with threads.  The threading model really really
complicates things and makes things harder than they need to be.

-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>

Re: Temporarily suspending prefetch on a message consumer.

[Tim Bain <tb...@alumni.duke.edu>]

You're using INDIVIDUAL_ACK, right?  Because without that, you should just
use a prefetch of 1 (or 0) and you'd get the behavior you want.

But even with INDIVIDUAL_ACK, can't you simply set prefetch to nrThreads?
Then any message that's dispatched to your consumer is guaranteed to be in
work by a thread.  Or use nrThreads + 1, if you want one additional message
waiting for a consumer for performance reasons.

Tim
On Jun 11, 2016 1:47 PM, "Kevin Burton" <bu...@spinn3r.com> wrote:

I have a cluster of ActiveMQ boxes and a network of consumers.

The problem I'm having now is that the prefetch on the consumers fetches
too many messages and then other machines in the cluster get starved out of
work.

What I really want is a pattern where I can have say 50 ActiveMQ brokers
and 200 worker servers each with their own set of consumers.  I want each
worker server to only be able read up to nrThreads (usually like 100-200)
to perform the work.  Reading more than nrThreads means its hogging work
which is preventing other workers from consuming these messages.

The only way I can really figure out a way to do this is to have the
prefetch temporarily suspended on some of the consumers.

I really with the ActiveMQ client were fully async via NIO/netty and not
simulated async with threads.  The threading model really really
complicates things and makes things harder than they need to be.

--

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>