Network of Brokers recommendations

["James A. Robinson" <ji...@gmail.com>]

Hi folks,

I joined this list because we're starting to look into setting up
ActiveMQ for our developers.  My preference at this time is to set
up a highly reliable system that doesn't have single points of
failure.

Due to the concern about single points of failure I didn't want to
introduce a shared persistence layer.  That's why I've been looking
at replicated LevelDB.

After spending some time reading up on ActiveMQ my first inclination
has been to propose setting up 3-node replicated LevelDB clusters,
and to build a network of brokers out of sets of those 3-node
clusters.

E.g., start with two clusters, and then keep adding on as we see
the need in terms of capacity.

[amq-1a, amq-1b, amq-1c]
[amq-2a, amq-2b, amq-2c]
...
[amq-9a, amq-9b, amq-9c]
...

I'm not sure what the best option is w/re to the network of broker
topology.  I could see mesh working, but my first instinct would
be to set up a complete graph, and have each broker cluster configured
with a uni-directional network connector to every other cluster
using the masterslave scheme.

I am wondering if this is a naive approach, and if so why it wouldn't
be a good idea.

If it is reasonable, I'm still trying to fully understand how one
deals with forwarding messages from brokers w/o consumers to those
brokers w/ consumers.  The page

http://activemq.apache.org/networks-of-brokers.html

discusses this topic and mentions rebalanceClusterClients, and it
isn't clear to me how that option will help if the clusters and
clients are fairly static:

  rebalanceClusterClients:
    if true, connected clients will be asked to rebalance across a
    cluster of brokers when a new broker joins the network of brokers
    (note: priorityBackup=true can override)

Reading the list of options available for transport connectors, it
seems reasonable to configure the transport connector with:

updateClusterClients="true"
rebalanceClusterClients="true"
updateClusterClientsOnRemove="true"

But if one has mostly static clients and we aren't frequently
bringing broker clusters up and down, I would assume there would
still be a problem of having messages on a broker that doesn't have
consumers attached.  If I'm wrong I'd like to understand what I'm
missing.

The network-of-brokers page, after mentioning rebalanceClusterClients
as one solution, continues the discussion and states that:

  Another, is to allow replay of messages back to the origin as
  there is no local consumer on that broker.

  There is a destination policy that allows this behavior for queues
  by configuring a conditionalNetworkBridgeFilterFactory with
  replayWhenNoConsumers=true. The conditionalNetworkBridgeFilterFactory
  provides an optional replayDelay based on the broker-in time.

  N.B.: When using replayWhenNoConsumers=true for versions < 5.9,
  it is necessary to also disable the cursors duplicate detection
  using enableAudit=false as the cursor could mark the replayed
  messages as duplicates (depending on the time window between
  playing and replaying these messages over the network bridge).
  The problem is fully explained in this blog post.

And I see in that ticket AMQ-4607 says this about the network
connector options:

  networkTTL sets both consumerTTL and messageTTL a value of -1
denotes infinite hops.
  In a mesh, use consumerTTL=1, messageTTL=-1 to allow consumers to
bounce around the mesh and messages to follow demand.
  In a linear topology use networkTTL == length of network (as before)

So would it be reasonable to set up a complete graph, with each
3-node cluster configured with a uni-directional network connector
to every other 3-node cluster using masterslave, and to set

  consumerTTL="1"
  messageTTL="-1"
  decreaseNetworkConsumerPriority="true"

and a networkBridgeFilterFactory/conditionalNetworkBridgeFilterFactory
with

  replayWhenNoConsumers="true"

I'd appreciate any input from the folks here, including any pointers
on any more material I should read.

Jim

Re: Network of Brokers recommendations

["James A. Robinson" <ji...@highwire.org>]

Hi,

Thanks for the reply.  My earlier distinction was sloppy, I ought to have
written something like 'partially connected mesh' vs. a complete graph.   I
was trying to distinguish between a topology where nodes are only connected
to immediate neighbors, and therefore may always be required to travel
across multiple brokers to get to a particular destination.

In this particular setup all the brokers will have the same capabilities
and will be connected to the same network, so I think I'll go ahead with
these scheme.  I appreciate the feedback!

Jim

On Sun, Apr 26, 2015 at 9:10 AM Tim Bain <tb...@alumni.duke.edu> wrote:

> Your understanding of this is good and you've got the right concepts here.
> The only two reasons I can think of that you shouldn't use a mesh (which I
> think is the same as a complete graph, though you drew a distinction
> between the two in your email so maybe there's a difference I'm not
> understanding) is if you have non-homogeneous network links where the most
> efficient routing between two brokers isn't necessarily the path "directly"
> (from a network-of-brokers standpoint, not from a network standpoint)
> between the two, or if you don't want to have to specify all of the other
> clusters so you'd use a hub-and-spoke topology so you only have to specify
> the hub node in your config.
>
> On Fri, Mar 13, 2015 at 12:21 AM, James A. Robinson <
> jim.robinson@gmail.com>
> wrote:
>
> > Hi folks,
> >
> > I joined this list because we're starting to look into setting up
> > ActiveMQ for our developers.  My preference at this time is to set
> > up a highly reliable system that doesn't have single points of
> > failure.
> >
> > Due to the concern about single points of failure I didn't want to
> > introduce a shared persistence layer.  That's why I've been looking
> > at replicated LevelDB.
> >
> > After spending some time reading up on ActiveMQ my first inclination
> > has been to propose setting up 3-node replicated LevelDB clusters,
> > and to build a network of brokers out of sets of those 3-node
> > clusters.
> >
> > E.g., start with two clusters, and then keep adding on as we see
> > the need in terms of capacity.
> >
> > [amq-1a, amq-1b, amq-1c]
> > [amq-2a, amq-2b, amq-2c]
> > ...
> > [amq-9a, amq-9b, amq-9c]
> > ...
> >
> > I'm not sure what the best option is w/re to the network of broker
> > topology.  I could see mesh working, but my first instinct would
> > be to set up a complete graph, and have each broker cluster configured
> > with a uni-directional network connector to every other cluster
> > using the masterslave scheme.
> >
> > I am wondering if this is a naive approach, and if so why it wouldn't
> > be a good idea.
> >
> > If it is reasonable, I'm still trying to fully understand how one
> > deals with forwarding messages from brokers w/o consumers to those
> > brokers w/ consumers.  The page
> >
> > http://activemq.apache.org/networks-of-brokers.html
> >
> > discusses this topic and mentions rebalanceClusterClients, and it
> > isn't clear to me how that option will help if the clusters and
> > clients are fairly static:
> >
> >   rebalanceClusterClients:
> >     if true, connected clients will be asked to rebalance across a
> >     cluster of brokers when a new broker joins the network of brokers
> >     (note: priorityBackup=true can override)
> >
> > Reading the list of options available for transport connectors, it
> > seems reasonable to configure the transport connector with:
> >
> > updateClusterClients="true"
> > rebalanceClusterClients="true"
> > updateClusterClientsOnRemove="true"
> >
> > But if one has mostly static clients and we aren't frequently
> > bringing broker clusters up and down, I would assume there would
> > still be a problem of having messages on a broker that doesn't have
> > consumers attached.  If I'm wrong I'd like to understand what I'm
> > missing.
> >
> > The network-of-brokers page, after mentioning rebalanceClusterClients
> > as one solution, continues the discussion and states that:
> >
> >   Another, is to allow replay of messages back to the origin as
> >   there is no local consumer on that broker.
> >
> >   There is a destination policy that allows this behavior for queues
> >   by configuring a conditionalNetworkBridgeFilterFactory with
> >   replayWhenNoConsumers=true. The conditionalNetworkBridgeFilterFactory
> >   provides an optional replayDelay based on the broker-in time.
> >
> >   N.B.: When using replayWhenNoConsumers=true for versions < 5.9,
> >   it is necessary to also disable the cursors duplicate detection
> >   using enableAudit=false as the cursor could mark the replayed
> >   messages as duplicates (depending on the time window between
> >   playing and replaying these messages over the network bridge).
> >   The problem is fully explained in this blog post.
> >
> > And I see in that ticket AMQ-4607 says this about the network
> > connector options:
> >
> >   networkTTL sets both consumerTTL and messageTTL a value of -1
> > denotes infinite hops.
> >   In a mesh, use consumerTTL=1, messageTTL=-1 to allow consumers to
> > bounce around the mesh and messages to follow demand.
> >   In a linear topology use networkTTL == length of network (as before)
> >
> > So would it be reasonable to set up a complete graph, with each
> > 3-node cluster configured with a uni-directional network connector
> > to every other 3-node cluster using masterslave, and to set
> >
> >   consumerTTL="1"
> >   messageTTL="-1"
> >   decreaseNetworkConsumerPriority="true"
> >
> > and a networkBridgeFilterFactory/conditionalNetworkBridgeFilterFactory
> > with
> >
> >   replayWhenNoConsumers="true"
> >
> > I'd appreciate any input from the folks here, including any pointers
> > on any more material I should read.
> >
> > Jim
> >
>

Re: Network of Brokers recommendations

[Tim Bain <tb...@alumni.duke.edu>]

Your understanding of this is good and you've got the right concepts here.
The only two reasons I can think of that you shouldn't use a mesh (which I
think is the same as a complete graph, though you drew a distinction
between the two in your email so maybe there's a difference I'm not
understanding) is if you have non-homogeneous network links where the most
efficient routing between two brokers isn't necessarily the path "directly"
(from a network-of-brokers standpoint, not from a network standpoint)
between the two, or if you don't want to have to specify all of the other
clusters so you'd use a hub-and-spoke topology so you only have to specify
the hub node in your config.

On Fri, Mar 13, 2015 at 12:21 AM, James A. Robinson <ji...@gmail.com>
wrote:

> Hi folks,
>
> I joined this list because we're starting to look into setting up
> ActiveMQ for our developers.  My preference at this time is to set
> up a highly reliable system that doesn't have single points of
> failure.
>
> Due to the concern about single points of failure I didn't want to
> introduce a shared persistence layer.  That's why I've been looking
> at replicated LevelDB.
>
> After spending some time reading up on ActiveMQ my first inclination
> has been to propose setting up 3-node replicated LevelDB clusters,
> and to build a network of brokers out of sets of those 3-node
> clusters.
>
> E.g., start with two clusters, and then keep adding on as we see
> the need in terms of capacity.
>
> [amq-1a, amq-1b, amq-1c]
> [amq-2a, amq-2b, amq-2c]
> ...
> [amq-9a, amq-9b, amq-9c]
> ...
>
> I'm not sure what the best option is w/re to the network of broker
> topology.  I could see mesh working, but my first instinct would
> be to set up a complete graph, and have each broker cluster configured
> with a uni-directional network connector to every other cluster
> using the masterslave scheme.
>
> I am wondering if this is a naive approach, and if so why it wouldn't
> be a good idea.
>
> If it is reasonable, I'm still trying to fully understand how one
> deals with forwarding messages from brokers w/o consumers to those
> brokers w/ consumers.  The page
>
> http://activemq.apache.org/networks-of-brokers.html
>
> discusses this topic and mentions rebalanceClusterClients, and it
> isn't clear to me how that option will help if the clusters and
> clients are fairly static:
>
>   rebalanceClusterClients:
>     if true, connected clients will be asked to rebalance across a
>     cluster of brokers when a new broker joins the network of brokers
>     (note: priorityBackup=true can override)
>
> Reading the list of options available for transport connectors, it
> seems reasonable to configure the transport connector with:
>
> updateClusterClients="true"
> rebalanceClusterClients="true"
> updateClusterClientsOnRemove="true"
>
> But if one has mostly static clients and we aren't frequently
> bringing broker clusters up and down, I would assume there would
> still be a problem of having messages on a broker that doesn't have
> consumers attached.  If I'm wrong I'd like to understand what I'm
> missing.
>
> The network-of-brokers page, after mentioning rebalanceClusterClients
> as one solution, continues the discussion and states that:
>
>   Another, is to allow replay of messages back to the origin as
>   there is no local consumer on that broker.
>
>   There is a destination policy that allows this behavior for queues
>   by configuring a conditionalNetworkBridgeFilterFactory with
>   replayWhenNoConsumers=true. The conditionalNetworkBridgeFilterFactory
>   provides an optional replayDelay based on the broker-in time.
>
>   N.B.: When using replayWhenNoConsumers=true for versions < 5.9,
>   it is necessary to also disable the cursors duplicate detection
>   using enableAudit=false as the cursor could mark the replayed
>   messages as duplicates (depending on the time window between
>   playing and replaying these messages over the network bridge).
>   The problem is fully explained in this blog post.
>
> And I see in that ticket AMQ-4607 says this about the network
> connector options:
>
>   networkTTL sets both consumerTTL and messageTTL a value of -1
> denotes infinite hops.
>   In a mesh, use consumerTTL=1, messageTTL=-1 to allow consumers to
> bounce around the mesh and messages to follow demand.
>   In a linear topology use networkTTL == length of network (as before)
>
> So would it be reasonable to set up a complete graph, with each
> 3-node cluster configured with a uni-directional network connector
> to every other 3-node cluster using masterslave, and to set
>
>   consumerTTL="1"
>   messageTTL="-1"
>   decreaseNetworkConsumerPriority="true"
>
> and a networkBridgeFilterFactory/conditionalNetworkBridgeFilterFactory
> with
>
>   replayWhenNoConsumers="true"
>
> I'd appreciate any input from the folks here, including any pointers
> on any more material I should read.
>
> Jim
>