You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by gfrank <gr...@netfrank.net> on 2007/11/30 00:08:32 UTC
looking for best practices for High Availability and guaranteed
message delivery
After reading as much as possible on this subject both on the ActiveMQ wiki
and here in the forum there are a few questions remaining. Hopefully
someone here can provide advice and/or cautions.
Background: I'm working on a message processing system that must have
guaranteed order of delivery and no message loss. This system has a human
safety impact so conservative choices are best. Throughput and load
balancing for performance purposes are the lowest priorities compared to
solid reliability and resilience.
Questions:
First, i see that recovery from a failed Master in a Master/Slave
configuration requires both some manual intervention in moving the message
store from the slave to the master as well as some incurred downtime due to
a restart of all the brokers. Is there a road-map item to address this
situation and simplify the recovery process ... hopefully making it
automatic? Has anyone here experienced this in production as it is
currently coded now and if so, how well does it work?
How about alternatives to the Master/Slave choice. Is there any way to
provide message replication for safety purposes within a network of brokers?
For example, is there any option using a network of brokers and a single
shared JDBC message store across multiple brokers? I know that this just
makes the JDBC store a new single point of failure but there are
off-the-shelf database cluster solutions which can help address that issue.
Finally, does anyone have experience with how Master/Slave and Network of
Brokers holdup under the circumstance of network failure within the cluster
causing multiple Masters, etc? I'm concerned about situations that can lead
to duplicate message processing and/or out-of-order messages.
Thanks for any advice,
Greg
--
View this message in context: http://www.nabble.com/looking-for-best-practices-for-High-Availability-and-guaranteed-message-delivery-tf4900564s2354.html#a14037207
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: looking for best practices for High Availability and guaranteed message delivery
Posted by James Strachan <ja...@gmail.com>.
I'd just like to echo what Viswanath just said - it sounds like
relying on a JDBC based master/slave is your best bet.
Note that Master/Slave is used for message replication and
master/slave failover - networks of brokers is purely about store and
forward of messages - not replication.
You may want to look at a replicated database such as Oracle RAC or
something if you want to avoid message loss in a severe hardware
failure type situation.
On 30/11/2007, Viswanath Durbha <Vi...@gmail.com> wrote:
>
> Hi,
>
> I can answer one of the many questions you had regarding master/slave setup.
>
> I use ActiveMQ 5.0 snapshot version and I have successfully setup a
> master/slave configuration using MySQL database. I followed the instructions
> at http://activemq.apache.org/jdbc-master-slave.html
> http://activemq.apache.org/jdbc-master-slave.html .
>
> The master first locks the database and the slave is in waiting mode. If the
> master goes down for any reason, the slave automatically comes up without
> any manual intervention. So this slave becomes the new master. There can be
> as many instances of ActiveMQ as needed all waiting for the database lock.
> This would take care of the High Availability (HA).
>
> The same can also be achieved using a shared network drive (SAN). The
> details can be found at
> http://activemq.apache.org/shared-file-system-master-slave.html
> http://activemq.apache.org/shared-file-system-master-slave.html
>
> Hope this helps.
>
> Thanks,
> Viswanath
>
>
> gfrank wrote:
> >
> > After reading as much as possible on this subject both on the ActiveMQ
> > wiki and here in the forum there are a few questions remaining. Hopefully
> > someone here can provide advice and/or cautions.
> >
> > Background: I'm working on a message processing system that must have
> > guaranteed order of delivery and no message loss. This system has a human
> > safety impact so conservative choices are best. Throughput and load
> > balancing for performance purposes are the lowest priorities compared to
> > solid reliability and resilience.
> >
> > Questions:
> >
> > First, i see that recovery from a failed Master in a Master/Slave
> > configuration requires both some manual intervention in moving the message
> > store from the slave to the master as well as some incurred downtime due
> > to a restart of all the brokers. Is there a road-map item to address this
> > situation and simplify the recovery process ... hopefully making it
> > automatic? Has anyone here experienced this in production as it is
> > currently coded now and if so, how well does it work?
> >
> > How about alternatives to the Master/Slave choice. Is there any way to
> > provide message replication for safety purposes within a network of
> > brokers? For example, is there any option using a network of brokers and
> > a single shared JDBC message store across multiple brokers? I know that
> > this just makes the JDBC store a new single point of failure but there are
> > off-the-shelf database cluster solutions which can help address that
> > issue.
> >
> > Finally, does anyone have experience with how Master/Slave and Network of
> > Brokers holdup under the circumstance of network failure within the
> > cluster causing multiple Masters, etc? I'm concerned about situations
> > that can lead to duplicate message processing and/or out-of-order
> > messages.
> >
> > Thanks for any advice,
> > Greg
> >
>
> --
> View this message in context: http://www.nabble.com/looking-for-best-practices-for-High-Availability-and-guaranteed-message-delivery-tf4900564s2354.html#a14042472
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://macstrac.blogspot.com/
Open Source Integration
http://open.iona.com
Re: looking for best practices for High Availability and guaranteed
message delivery
Posted by Viswanath Durbha <Vi...@gmail.com>.
Hi,
I can answer one of the many questions you had regarding master/slave setup.
I use ActiveMQ 5.0 snapshot version and I have successfully setup a
master/slave configuration using MySQL database. I followed the instructions
at http://activemq.apache.org/jdbc-master-slave.html
http://activemq.apache.org/jdbc-master-slave.html .
The master first locks the database and the slave is in waiting mode. If the
master goes down for any reason, the slave automatically comes up without
any manual intervention. So this slave becomes the new master. There can be
as many instances of ActiveMQ as needed all waiting for the database lock.
This would take care of the High Availability (HA).
The same can also be achieved using a shared network drive (SAN). The
details can be found at
http://activemq.apache.org/shared-file-system-master-slave.html
http://activemq.apache.org/shared-file-system-master-slave.html
Hope this helps.
Thanks,
Viswanath
gfrank wrote:
>
> After reading as much as possible on this subject both on the ActiveMQ
> wiki and here in the forum there are a few questions remaining. Hopefully
> someone here can provide advice and/or cautions.
>
> Background: I'm working on a message processing system that must have
> guaranteed order of delivery and no message loss. This system has a human
> safety impact so conservative choices are best. Throughput and load
> balancing for performance purposes are the lowest priorities compared to
> solid reliability and resilience.
>
> Questions:
>
> First, i see that recovery from a failed Master in a Master/Slave
> configuration requires both some manual intervention in moving the message
> store from the slave to the master as well as some incurred downtime due
> to a restart of all the brokers. Is there a road-map item to address this
> situation and simplify the recovery process ... hopefully making it
> automatic? Has anyone here experienced this in production as it is
> currently coded now and if so, how well does it work?
>
> How about alternatives to the Master/Slave choice. Is there any way to
> provide message replication for safety purposes within a network of
> brokers? For example, is there any option using a network of brokers and
> a single shared JDBC message store across multiple brokers? I know that
> this just makes the JDBC store a new single point of failure but there are
> off-the-shelf database cluster solutions which can help address that
> issue.
>
> Finally, does anyone have experience with how Master/Slave and Network of
> Brokers holdup under the circumstance of network failure within the
> cluster causing multiple Masters, etc? I'm concerned about situations
> that can lead to duplicate message processing and/or out-of-order
> messages.
>
> Thanks for any advice,
> Greg
>
--
View this message in context: http://www.nabble.com/looking-for-best-practices-for-High-Availability-and-guaranteed-message-delivery-tf4900564s2354.html#a14042472
Sent from the ActiveMQ - User mailing list archive at Nabble.com.