You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Nicky Sandhu <ni...@hotmail.com> on 2007/06/20 20:17:12 UTC
How to have a highly available and scalable setup using network of
brokers?
I am in the process of trying to setup a network of brokers for HA behind a
load balancer. The aim is to allow clients to see a virtual broker (which is
really load balanced and with fail over) with a single IP/port. The LBR is
round robin spraying connections to all the masters and the slaves are there
known only to other masters for draining messages if the master fails.
Here's an image of the conceptual architecture.
http://www.nabble.com/file/p11219295/ActiveMQCluster.jpg
Comments/suggestions/alternatives? If this is feasible what does this mean
for durable subscribes as I forsee producers/consumers connecting to any
broker.
Has anyone tried such an architecture? Any alternative ways to achieve
HA/scalability which shields the client from configuration changes as we
scale up?
I am attaching a preliminary config files for this setup (network only, no
master/slave yet) http://www.nabble.com/file/p11219295/activemq.xml
server1.xml , http://www.nabble.com/file/p11219295/activemq.xml server2.xml
and http://www.nabble.com/file/p11219295/activemq.xml server3.xml
--
View this message in context: http://www.nabble.com/How-to-have-a-highly-available-and-scalable-setup-using-network-of-brokers--tf3954224s2354.html#a11219295
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How to have a highly available and scalable setup using network
of brokers?
Posted by Nicky Sandhu <ni...@hotmail.com>.
James.Strachan wrote:
>
> It'd be interesting to see what kinds of throughput, RAM & CPU usage
> you get in your environment; you might find a single broker can easily
> handle that load on a decent 2-4 way blade etc
>
>
Wanted to make sure I have options before painting myself into a corner.
Simple is what we will start with :)
James.Strachan wrote:
>
>
> This is slightly different from the 'pick the broker with the least
> traffic for all my messaging load' as you can instead do the load
> balancing at a more finely grained level; at each destination rather
> than connection.
>
OK, so as long as we are dreaming, I'll push the envelope even further.
Does this open the door to using some kind of broker load monitoring service
and using the load information to dynamically change which broker a "jedi"
client would produce its next batch of messages to or consume from ?
So in other words instead of making the decision of which broker to choose
for which destination in configuration it would be a self adjusting system
which chooses the broker for the next batch of messages to produce/consume
based on some load metrics collected from the brokers.
Ofcourse, I don't have a clue to what this would do when mixed in with the
fact that there is an overhead of moving the message between brokers when
clients hop around dynamically.
--
View this message in context: http://www.nabble.com/How-to-have-a-highly-available-and-scalable-setup-using-network-of-brokers--tf3954224s2354.html#a11235683
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How to have a highly available and scalable setup using network of brokers?
Posted by James Strachan <ja...@gmail.com>.
On 6/20/07, Nicky Sandhu <ni...@hotmail.com> wrote:
> James.Strachan wrote:
> >
> > On 6/20/07, Nicky Sandhu <ni...@hotmail.com> wrote:
> >>
> >> I am in the process of trying to setup a network of brokers for HA
> >
> > If you want HA then you want a Master/Slave arrangement as you
> > probably want to replicate messages to multiple physical brokers (or
> > share state across multiple brokers via JDBC or a SAN)..
> >
> > http://activemq.apache.org/masterslave.html
> >
> I have that (see the diagram) so I get what you are saying... but
>
> James.Strachan wrote:
> >
> >
> >> behind a
> >> load balancer. The aim is to allow clients to see a virtual broker (which
> >> is
> >> really load balanced and with fail over) with a single IP/port.
> >
> > FWIW do you really need to load balance across brokers? What kinds of
> > message throughputs are we talking about?
> >
> > i.e. you only need to move from a master/slave to a network of
> > masters/slaves when you have pretty large throughput.
> >
> ... we need to scale upto 3-4 K msgs/sec with durable subscribes on topics,
> with persistent messages and upwards of a couple hundred publishers and
> couple thousand consumers. Msg sizes range from 1K - 10 M. I am worried
> about memory requirements on a single master broker being able to handle
> that.
It'd be interesting to see what kinds of throughput, RAM & CPU usage
you get in your environment; you might find a single broker can easily
handle that load on a decent 2-4 way blade etc
I guess what I was trying to say was; start out doing the easiest
thing that might work; then only get more complex when you really need
it.
> James.Strachan wrote:
> >
> >> Any alternative ways to achieve
> >> HA/scalability which shields the client from configuration changes as we
> >> scale up?
> >
> > On the client side, its purely about the discovery of brokers -
> > whatever the topology & number of brokers - so you could use one of
> > the existing discovery mechanisms (e.g. zeroconf or multicast) or
> > write your own if you want a non-multicast based one.
> >
> > http://activemq.apache.org/discovery.html
> >
> so let me understand what you are saying. I could potentially write a load
> balancing discovery mechanism that could connect clients with the least
> stressed out broker at the moment of connection request ? cool !!!
Sure. We've tried to make every aspect of ActiveMQ easily pluggable,
so you can tinker to your hearts content to get the exact messaging
bus of your dreams :). Its pretty simple to plug in any kind of
discovery mechanism you wanna use to find the available brokers (an
LDAP option might be interesting, or a HTTP based one where brokers
POST themselves to a web site & clients GET the list of active brokers
that have heart-beated within a time period etc).
Incidentally, something even cooler - the JEDI transport...
https://issues.apache.org/activemq/browse/AMQ-816
Its not been implemented yet - I think John Heitmann had a version
somewhere but he never got around to contribute it :(. But its
basically a derivation of the FanOutTransport.
Basically each client connects to all brokers (or all Masters if
you're using Master/Slaves).
Then as commands need to be sent to a broker (sending messages,
performing transactions, acking etc), the client can, at runtime,
choose which broker to use for each *destination*. i.e. the
JediTransport is a kinda CompositeTransport; choosing the actual
transport implementations to send each Command down; for publishing on
a topic, it might be all of them; for publishing on a queue, it'd pick
one broker etc.
So for example; you could then introduce a partiioning mechanism;
where all consumers of a particular destination use a particular
broker. This is kinda like Message Groups, but on the client side
rather than the broker side. A client can basically choose to load
balance its use of destinations across whichever brokers it wishes.
Its also an alternative to store & forward brokers; as the clients can
connect & consume from every broker (so it saves an extra hop).
This is slightly different from the 'pick the broker with the least
traffic for all my messaging load' as you can instead do the load
balancing at a more finely grained level; at each destination rather
than connection.
The main downside of the Jedi transport is the possible increase in
the numbers of connections open on a broker; if you were talking
50,000 connections, then a store/forward network of brokers might be
better; but if its only in the 1-5,000 range, then a single broker can
handle that (providing you configure enough file descriptors on unix).
--
James
-------
http://macstrac.blogspot.com/
Re: How to have a highly available and scalable setup using network
of brokers?
Posted by Nicky Sandhu <ni...@hotmail.com>.
James.Strachan wrote:
>
> On 6/20/07, Nicky Sandhu <ni...@hotmail.com> wrote:
>>
>> I am in the process of trying to setup a network of brokers for HA
>
> If you want HA then you want a Master/Slave arrangement as you
> probably want to replicate messages to multiple physical brokers (or
> share state across multiple brokers via JDBC or a SAN)..
>
> http://activemq.apache.org/masterslave.html
>
I have that (see the diagram) so I get what you are saying... but
James.Strachan wrote:
>
>
>> behind a
>> load balancer. The aim is to allow clients to see a virtual broker (which
>> is
>> really load balanced and with fail over) with a single IP/port.
>
> FWIW do you really need to load balance across brokers? What kinds of
> message throughputs are we talking about?
>
> i.e. you only need to move from a master/slave to a network of
> masters/slaves when you have pretty large throughput.
>
... we need to scale upto 3-4 K msgs/sec with durable subscribes on topics,
with persistent messages and upwards of a couple hundred publishers and
couple thousand consumers. Msg sizes range from 1K - 10 M. I am worried
about memory requirements on a single master broker being able to handle
that.
James.Strachan wrote:
>
>> Any alternative ways to achieve
>> HA/scalability which shields the client from configuration changes as we
>> scale up?
>
> On the client side, its purely about the discovery of brokers -
> whatever the topology & number of brokers - so you could use one of
> the existing discovery mechanisms (e.g. zeroconf or multicast) or
> write your own if you want a non-multicast based one.
>
> http://activemq.apache.org/discovery.html
>
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
>
so let me understand what you are saying. I could potentially write a load
balancing discovery mechanism that could connect clients with the least
stressed out broker at the moment of connection request ? cool !!!
--
View this message in context: http://www.nabble.com/How-to-have-a-highly-available-and-scalable-setup-using-network-of-brokers--tf3954224s2354.html#a11221053
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How to have a highly available and scalable setup using network of brokers?
Posted by James Strachan <ja...@gmail.com>.
On 6/20/07, Nicky Sandhu <ni...@hotmail.com> wrote:
>
> I am in the process of trying to setup a network of brokers for HA
If you want HA then you want a Master/Slave arrangement as you
probably want to replicate messages to multiple physical brokers (or
share state across multiple brokers via JDBC or a SAN)..
http://activemq.apache.org/masterslave.html
> behind a
> load balancer. The aim is to allow clients to see a virtual broker (which is
> really load balanced and with fail over) with a single IP/port.
FWIW do you really need to load balance across brokers? What kinds of
message throughputs are we talking about?
i.e. you only need to move from a master/slave to a network of
masters/slaves when you have pretty large throughput.
> Any alternative ways to achieve
> HA/scalability which shields the client from configuration changes as we
> scale up?
On the client side, its purely about the discovery of brokers -
whatever the topology & number of brokers - so you could use one of
the existing discovery mechanisms (e.g. zeroconf or multicast) or
write your own if you want a non-multicast based one.
http://activemq.apache.org/discovery.html
--
James
-------
http://macstrac.blogspot.com/
Re: How to have a highly available and scalable setup using network
of brokers?
Posted by Joel Schaubert <fo...@gmail.com>.
Nicky Sandhu wrote:
>
>
> Joel Schaubert wrote:
>>
>> What I have found during my testing of the network of brokers is that you
>> do get this behaviour. If I write 100 message to broker A, then I kill
>> it, then I subscribe to broker B, I can read all of the messages. (and
>> various versions of this scenario tested in our QA environment).
>>
> Interesting. That means the broker A is pushing messages to broker B
> before sending publisher the ack. Thats quite an overhead. Or is broker A
> somehow sharing the message store with broker B? I thought the forwarding
> was on a demand based policy. Did you configure it differently?
>
You should test it yourself because it's so easy to do so. You can run 2
(or more) brokers on one box, just move the ports for the second broker to a
different set. You'll need to switch what subdir it uses for its persistent
storage, and I think the jmx and that's about it.
I did no special config on the brokers (except moving ports as above) and
just added static route to the other broker (uses the openwire connector so
you want that connectors port number if you use static).
I should add that all of my messages were perstistent, with client-ack set
(not auto-ack which is default).
I think you'll be surprised how easy this is to setup and test, and how well
it works.
I am having a problem with the STOMP connector tossing away prefetch
messages, but if your access is java then that's not an issue.
[ https://issues.apache.org/activemq/browse/AMQ-1287 Stomp Losing Prefetch
Messages]
Joel Schaubert
--
View this message in context: http://www.nabble.com/How-to-have-a-highly-available-and-scalable-setup-using-network-of-brokers--tf3954224s2354.html#a11238730
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How to have a highly available and scalable setup using network
of brokers?
Posted by Nicky Sandhu <ni...@hotmail.com>.
Joel Schaubert wrote:
>
> What I have found during my testing of the network of brokers is that you
> do get this behaviour. If I write 100 message to broker A, then I kill
> it, then I subscribe to broker B, I can read all of the messages. (and
> various versions of this scenario tested in our QA environment).
>
Interesting. That means the broker A is pushing messages to broker B before
sending publisher the ack. Thats quite an overhead. Or is broker A somehow
sharing the message store with broker B? I thought the forwarding was on a
demand based policy. Did you configure it differently?
Joel Schaubert wrote:
>
> What's also nice about this is unlike master/slave, when broker A comes
> back up it will be noticed by the load balancer and can take any new
> client connections then. Read the section on how to do recovery on a
> failed master in master slave, it doesn't sound anywhere near as easy as a
> network of brokers behind a load balancer.
>
I agree. If what you are saying works then I have no need for M/S
configuration and the headaches of recovery.
Joel Schaubert wrote:
>
> One thing to note if you try network of brokers is that the auto discovery
> is very complete. So if you had 2 different networks of brokers I don't
> think auto discovery is useful because they would all join up into one big
> network. For example if you had one network in production, one for QA, I
> think the autodiscovery would have every broker find every other broker.
> We are testing the method where you statically list each of the other
> brokers you want to participate with.
>
I believe James is saying that you can write your own discovery policy. So
a discovery agent based on some service that tells it what brokers are
willing to join in that network should work in the above scenario.
Also I am no expert in multicast, but I think you can specify a different
host /port combination in the multicast discovery configuration between prod
and QA brokers and that would partition the brokers as such.
--
View this message in context: http://www.nabble.com/How-to-have-a-highly-available-and-scalable-setup-using-network-of-brokers--tf3954224s2354.html#a11235844
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How to have a highly available and scalable setup using network
of brokers?
Posted by Joel Schaubert <fo...@gmail.com>.
Nicky Sandhu wrote:
>
>
> Joel Schaubert wrote:
>>
>> What are the extra benefits of going to the more complicated setup of
>> pairs of master/slave's behind the load balancer instead of a flat space
>> of a network of brokers?
>>
> If a master were to fail then the slave would connect as a master to the
> network to deliver any messages that were in the master's queue/topic for
> consumers on other masters in the network. At least my idea was to not
> have any messages stuck in the failed master's queue and have them
> delivered by the slave. However the slave behind the load balancer would
> not get any load (fresh client connections) which would mean that the load
> after failure would be shared by remaining masters in the network.
>
What I have found during my testing of the network of brokers is that you do
get this behaviour. If I write 100 message to broker A, then I kill it,
then I subscribe to broker B, I can read all of the messages. (and various
versions of this scenario tested in our QA environment). What's also nice
about this is unlike master/slave, when broker A comes back up it will be
noticed by the load balancer and can take any new client connections then.
Read the section on how to do recovery on a failed master in master slave,
it doesn't sound anywhere near as easy as a network of brokers behind a load
balancer.
One thing to note if you try network of brokers is that the auto discovery
is very complete. So if you had 2 different networks of brokers I don't
think auto discovery is useful because they would all join up into one big
network. For example if you had one network in production, one for QA, I
think the autodiscovery would have every broker find every other broker. We
are testing the method where you statically list each of the other brokers
you want to participate with.
Joel Schaubert
--
View this message in context: http://www.nabble.com/How-to-have-a-highly-available-and-scalable-setup-using-network-of-brokers--tf3954224s2354.html#a11235164
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How to have a highly available and scalable setup using network
of brokers?
Posted by Nicky Sandhu <ni...@hotmail.com>.
Joel Schaubert wrote:
>
>
> Our strategy is to use the "network of brokers" hosted behind a load
> balancer.
> http://activemq.apache.org/networks-of-brokers.html
>
> What are the extra benefits of going to the more complicated setup of
> pairs of master/slave's behind the load balancer instead of a flat space
> of a network of brokers?
>
If a master were to fail then the slave would connect as a master to the
network to deliver any messages that were in the master's queue/topic for
consumers on other masters in the network. At least my idea was to not have
any messages stuck in the failed master's queue and have them delivered by
the slave. However the slave behind the load balancer would not get any load
(fresh client connections) which would mean that the load after failure
would be shared by remaining masters in the network.
--
View this message in context: http://www.nabble.com/How-to-have-a-highly-available-and-scalable-setup-using-network-of-brokers--tf3954224s2354.html#a11226753
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: How to have a highly available and scalable setup using network
of brokers?
Posted by Joel Schaubert <fo...@gmail.com>.
Nicky Sandhu wrote:
>
> I am in the process of trying to setup a network of brokers for HA behind
> a load balancer. The aim is to allow clients to see a virtual broker
> (which is really load balanced and with fail over) with a single IP/port.
>
We've been testing a simpler failover strategy with the same goal of hosting
brokers behind a load balancer. Similar requirements of persistence on all
messages and easy failover but with the additional requirement that we use
STOMP clients. These clients do not have the java connector logic in them
to do failover so using the load balancer in front works great for that.
Our strategy is to use the "network of brokers" hosted behind a load
balancer.
http://activemq.apache.org/networks-of-brokers.html
What are the extra benefits of going to the more complicated setup of pairs
of master/slave's behind the load balancer instead of a flat space of a
network of brokers?
Joel Schaubert
--
View this message in context: http://www.nabble.com/How-to-have-a-highly-available-and-scalable-setup-using-network-of-brokers--tf3954224s2354.html#a11222027
Sent from the ActiveMQ - User mailing list archive at Nabble.com.