You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Abdul Alhazred <th...@tradedesksoftware.com> on 2006/08/03 15:38:04 UTC
Question about clustering and reliable delivery
OK, here's the scenario:
I have a number of producers, which produce a rather large number of
messages, thus I need to be able to direct these messages to a set of
clustered ActiveMQ brokers, and likewise there are numbers of consumers on
the other side of the cluster receiving these messages.
So far this is all fine, but these messages MUST be guaranteed delivered and
processed in order. Exclusive Consumer is fine for this. Now what happens if
the database server backing one of the brokers in the cluster fails? Both
the consumers and producers can fail over to another broker, but now
whatever messages are stored in that particular database are GONE (IE any
which haven't yet been consumed).
What would be a recommended solution? It looks like ActiveMQ 4.1's JDBC
master/slave would be a good answer, but I'd really like to have a solution
I can deploy in the near term. Pure database replication doesn't appear to
be a real answer because master/slave replication in RDBMS are asynchronous
(thus the slave database server might not yet have received all updates at
the point where the crash occured). Besides in that case it would seem a new
broker would have to join the cluster and start up using the replica
database, which brings up various issues with what would happen with
ordering of message delivery in the mean time.
--
View this message in context: http://www.nabble.com/Question-about-clustering-and-reliable-delivery-tf2045679.html#a5632611
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Question about clustering and reliable delivery
Posted by Abdul Alhazred <th...@tradedesksoftware.com>.
James.Strachan wrote:
>
>
> OK, I guess I'll have to experiment with JDBC replication. The section on
> that says something about only being fully supported on 4.1, but if it
> works on 4.0.2 that will be fine. I can't really deploy non-production
> software releases in our production systems...
>
> I'm also curious how master/slave interacts with clustering. Can cluster
> nodes be replicated? If so how would THAT affect message delivery? I can
> see how it would be POSSIBLE for that to work, I guess the question is
> does it? I would expect a naive implementation to fail to deliver messages
> in order in that case (IE master fails, other brokers pick up its
> connections via client failover, some new messages are delivered,
> meanwhile slave joins the cluster and now has older messages available).
> Maybe I'm just overthinking the whole thing, but we have anticipated
> requirements to deliver in excess of 500k messages per hour under peak
> loads with an ABSOLUTE zero loss rate. ;)
>
> On 8/3/06, Abdul Alhazred <th...@tradedesksoftware.com> wrote:
>>
>> OK, here's the scenario:
>>
>> I have a number of producers, which produce a rather large number of
>> messages, thus I need to be able to direct these messages to a set of
>> clustered ActiveMQ brokers, and likewise there are numbers of consumers
>> on
>> the other side of the cluster receiving these messages.
>>
>> So far this is all fine, but these messages MUST be guaranteed delivered
>> and
>> processed in order. Exclusive Consumer is fine for this. Now what happens
>> if
>> the database server backing one of the brokers in the cluster fails? Both
>> the consumers and producers can fail over to another broker, but now
>> whatever messages are stored in that particular database are GONE (IE any
>> which haven't yet been consumed).
>
> The assumption is that the file or database is reliable. Failing that
> you can use Master/Slave to replicate to different disks.
>
> http://incubator.apache.org/activemq/masterslave.html
>
>
>> What would be a recommended solution? It looks like ActiveMQ 4.1's JDBC
>> master/slave would be a good answer, but I'd really like to have a
>> solution
>> I can deploy in the near term.
>
> JDBC master/slave works great today - assuming you have reilable
> databases. Though its fairly slow since it requires you to disable the
> high performance journal.
>
> Using a shared file system Master/Slave or using straight Master/Slave
> is probably your best option if you have high performance requirements
>
>
>> Pure database replication doesn't appear to
>> be a real answer because master/slave replication in RDBMS are
>> asynchronous
>> (thus the slave database server might not yet have received all updates
>> at
>> the point where the crash occured). Besides in that case it would seem a
>> new
>> broker would have to join the cluster and start up using the replica
>> database, which brings up various issues with what would happen with
>> ordering of message delivery in the mean time.
>
> Ordering is guarrenteed with Shared-FileSystem and JDBC based
> Master/Slaves.
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Question-about-clustering-and-reliable-delivery-tf2045679.html#a5649121
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Question about clustering and reliable delivery
Posted by James Strachan <ja...@gmail.com>.
On 8/3/06, Abdul Alhazred <th...@tradedesksoftware.com> wrote:
>
> OK, here's the scenario:
>
> I have a number of producers, which produce a rather large number of
> messages, thus I need to be able to direct these messages to a set of
> clustered ActiveMQ brokers, and likewise there are numbers of consumers on
> the other side of the cluster receiving these messages.
>
> So far this is all fine, but these messages MUST be guaranteed delivered and
> processed in order. Exclusive Consumer is fine for this. Now what happens if
> the database server backing one of the brokers in the cluster fails? Both
> the consumers and producers can fail over to another broker, but now
> whatever messages are stored in that particular database are GONE (IE any
> which haven't yet been consumed).
The assumption is that the file or database is reliable. Failing that
you can use Master/Slave to replicate to different disks.
http://incubator.apache.org/activemq/masterslave.html
> What would be a recommended solution? It looks like ActiveMQ 4.1's JDBC
> master/slave would be a good answer, but I'd really like to have a solution
> I can deploy in the near term.
JDBC master/slave works great today - assuming you have reilable
databases. Though its fairly slow since it requires you to disable the
high performance journal.
Using a shared file system Master/Slave or using straight Master/Slave
is probably your best option if you have high performance requirements
> Pure database replication doesn't appear to
> be a real answer because master/slave replication in RDBMS are asynchronous
> (thus the slave database server might not yet have received all updates at
> the point where the crash occured). Besides in that case it would seem a new
> broker would have to join the cluster and start up using the replica
> database, which brings up various issues with what would happen with
> ordering of message delivery in the mean time.
Ordering is guarrenteed with Shared-FileSystem and JDBC based Master/Slaves.
--
James
-------
http://radio.weblogs.com/0112098/