Network of Master Slave brokers question

[pollotek <cl...@gmail.com>]

I am setting up a network of master slave brokers. I used
https://github.com/FuseByExample/getting_started_with_activemq as template
for brokers configuration. I ran several tests and I have a few questions. I
hope you guys don't find them unreasonable and unanswerable

On a Network of Master-Slave Brokers with persistence enabled, when one of
the master brokers dies the traffic gets redirected to one of the slaves.
The messages stored in the affected server will be processed only when the
failing server comes back on-line. Correct? or can I expect all messages to
be replicated to the Slave just like it happens with Master Slave share
nothing configuration?

If the messages were not redundantly stored, in the unfortunate case the
file system gets corrupted or damaged in the server that just died, the
messages stored there forever will be lost since they're only stored in this
one server at a time (Store and Forward). Correct?

If the previous scenario is correct, is there an actual way to have a
Network of Master-Slave Brokers with redundant storage? So if there is a
catastrophic file system failure on an active broker messages without
messages.

Finally if the Network of Master Slave brokers does redundantly persist
messages in both, master and slave. When the master fails and the slave
takes over, would I have to copy the slave's storage back to the master
before restarting the master? 


--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-Master-Slave-brokers-question-tp4604449.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Network of Master Slave brokers question

[Jakub Korab <ja...@gmail.com>]

Hi pollotek,

Responses inline:


pollotek wrote
> 
> I am setting up a network of master slave brokers. I used
> https://github.com/FuseByExample/getting_started_with_activemq as template
> for brokers configuration. I ran several tests and I have a few questions.
> I hope you guys don't find them unreasonable and unanswerable
> 
> On a Network of Master-Slave Brokers with persistence enabled, when one of
> the master brokers dies the traffic gets redirected to one of the slaves.
> The messages stored in the affected server will be processed only when the
> failing server comes back on-line. Correct? or can I expect all messages
> to be replicated to the Slave just like it happens with Master Slave share
> nothing configuration?
> 

What you're seeing in that config are two master-slave pairs, connected to
each other via shared storage
(http://activemq.apache.org/shared-file-system-master-slave.html). Thake a
look at the broker@dataDirectory location to confirm that each pair points
to the same location in the file system. 

In either pair, if the master dies the slave starts up using the same
message store. That means that whatever the state was of the master when it
fell over, that state is used my the slave - it sends any unsent messages
that were persisted in the store.


pollotek wrote
> 
> If the messages were not redundantly stored [they are], in the unfortunate
> case the file system gets corrupted or damaged in the server that just
> died, the messages stored there forever will be lost since they're only
> stored in this one server at a time (Store and Forward). Correct?
> 

Typically you would place the shared storage outside of the server the
broker is running on, and expose access to it via a shared file system (that
supports distributed locks - if you don't have one you can use a DB
instead).


pollotek wrote
> 
> If the previous scenario is correct, is there an actual way to have a
> Network of Master-Slave Brokers with redundant storage? So if there is a
> catastrophic file system failure on an active broker messages without
> messages.
> 

If you are using shared storage via a file system (typically on a SAN) or a
DB, the mechanism of making that redundant is specific to your vendor (it's
outside of ActiveMQ's control). If you don't want to do that then you have
to set up some sort of eventually consistent network of brokers, the sort
that you were implying earlier with no master-slave setups but rather
relying on connected peers and store-and-forward. Like everything in
messaging, it's a tradeoff. 


pollotek wrote
> 
> Finally if the Network of Master Slave brokers does redundantly persist
> messages in both, master and slave. When the master fails and the slave
> takes over, would I have to copy the slave's storage back to the master
> before restarting the master?
> 

Not in the config in those samples - what you are thinking of only applies
to a shared-nothing master-slave configuration
(http://activemq.apache.org/pure-master-slave.html), and is one of its key
drawbacks - it's a "high-touch" solution. Shared-nothing is in reality used
only when you can't use a shared file system (don't have the appropriate
hardware) or DB.

Jakub


--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-Master-Slave-brokers-question-tp4604449p4605608.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.