Producer catch up in large network of brokers

[Kevin McLaughlin <km...@gmail.com>]

We have a network of brokers:

Stomp producers -> 200+ remote brokers -> bridged queue -> 1 central
broker -> ~128 Java consumers

The central broker is 5.8.0 and the remote brokers are a mix of 5.X.

There are latency and throughput differences between the remote
brokers and the central broker.  Some are sub-ms with a large pipe and
some are 100+ms and <2mbps.

Under normal circumstances, the consumers (in total) are processing
roughly 3,000 persistent msgs/second.

The central broker has a small memoryLimit (4m) set on this queue to
force the messages to queue on the remote brokers.  This way if we
have a problem with the central broker (kahadb corruption, host down,
etc.) data loss is minimal.  We are also using message groups on these
messages.  The central broker has its persistent store on a ram disk
so that it can process at this rate.

The max throughput of the central broker is 4-5k messages/second with
this configuration, or, about 1.5 times our sustained message
throughput.

As such, when there is a problem with either the central broker, the
consumers, or the downstream persistent storage, we back up very
quickly.  For example, if the central broker is down for 10 minutes,
it will take roughly 20 minutes to catch up (since we're still
generating 3k msgs/second on the remote brokers).

Give above, I was wondering:

1) During this 'catch up' period, how does ActiveMQ decide which
remote broker to accept messages from?  What we see is some of the
remote brokers seem to starve until we drop below maximum processing
capacity.  For example, remote broker 1 will not have any (or very
few) messages in its backlog processed whereas broker 2 will fully
drain.  Another way to ask this might be how does the central broker
decide which producer to block (flow control)?

2) Is there a better way to configure the central broker to tell it
run without a local disk store even though the messages are sent
persistent?  We want them persistent on the remote brokers, but, not
on the central broker.

Thanks,

--Kevin

Re: Producer catch up in large network of brokers

[Noel OConnor <no...@gmail.com>]

Others would know better but I'd have thought that each remote broker
would be seen as and individual client to the central broker and as
such would have its own prefetch limit and flow control management.
The latency and bandwidth limitations that you mention would also have
an impact.

As for disabling persistence just add <broker persistent="false">
.....  </broker>

There may be other activemq features which would help, take at look
http://www.slideshare.net/dejanb/advanced-messaging-with-apache-activemq
in particular optimizedispatch

On Wed, Aug 13, 2014 at 8:46 AM, Kevin McLaughlin <km...@gmail.com> wrote:
> We have a network of brokers:
>
> Stomp producers -> 200+ remote brokers -> bridged queue -> 1 central
> broker -> ~128 Java consumers
>
> The central broker is 5.8.0 and the remote brokers are a mix of 5.X.
>
> There are latency and throughput differences between the remote
> brokers and the central broker.  Some are sub-ms with a large pipe and
> some are 100+ms and <2mbps.
>
> Under normal circumstances, the consumers (in total) are processing
> roughly 3,000 persistent msgs/second.
>
> The central broker has a small memoryLimit (4m) set on this queue to
> force the messages to queue on the remote brokers.  This way if we
> have a problem with the central broker (kahadb corruption, host down,
> etc.) data loss is minimal.  We are also using message groups on these
> messages.  The central broker has its persistent store on a ram disk
> so that it can process at this rate.
>
> The max throughput of the central broker is 4-5k messages/second with
> this configuration, or, about 1.5 times our sustained message
> throughput.
>
> As such, when there is a problem with either the central broker, the
> consumers, or the downstream persistent storage, we back up very
> quickly.  For example, if the central broker is down for 10 minutes,
> it will take roughly 20 minutes to catch up (since we're still
> generating 3k msgs/second on the remote brokers).
>
> Give above, I was wondering:
>
> 1) During this 'catch up' period, how does ActiveMQ decide which
> remote broker to accept messages from?  What we see is some of the
> remote brokers seem to starve until we drop below maximum processing
> capacity.  For example, remote broker 1 will not have any (or very
> few) messages in its backlog processed whereas broker 2 will fully
> drain.  Another way to ask this might be how does the central broker
> decide which producer to block (flow control)?
>
> 2) Is there a better way to configure the central broker to tell it
> run without a local disk store even though the messages are sent
> persistent?  We want them persistent on the remote brokers, but, not
> on the central broker.
>
> Thanks,
>
> --Kevin