You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by JohnBeeBee <jo...@ipfin.co.uk> on 2017/11/29 11:45:56 UTC
Artemis Disaster Recovery options
I am looking at options for handling local failover and disaster recovery.
Our existing primary and secondary data centre hosted services run mainly
active-passive and have SAN storage but do not support SAN replication.
There is a dedicated network between the two DC's so we have fast, reliable
connectivity.
For local single node failure in the primary we would failover to a local
backup server configured to use shared storage, however I'm not sure what
the best options are for handling a complete primary DC failure so that we
can failover to the secondary DC. As I said we don't have SAN replication.
Is there anything similar to SQL log shipping that can be performed? We
currently use BizTalk which uses SQL log shipping and so we already accept a
certain amount of message loss in the case of a DR. Even better is there an
option which would remove all data loss from Artemis in the case of a full
DR scenario?
Thanks
John
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Artemis Disaster Recovery options
Posted by Gary Tully <ga...@gmail.com>.
>
> > What if any file/directory management would need to be done on the
> secondary DC file structure if doing a straight file copy. I wouldn't want
> to copy the whole directory every 15 minutes for example, but rather just
> reflect the changes.
>
> There is a challenge with just copying changes because the journal files
> can and will be re-used. This is where a replicated block device would be
> nice.
>
> rsync will do a good job of just copying changes, even when reused b/c it
does a full scan of the file to determine what (blocks) have changed.
This full scan; ie: full read to determine checksums, can be expensive on
large journal files so one mitigation is to increase the number of journals
in the pool and decrease the journal max file size.
An ideal rsync would be aware of the current journal file append pointer
though some sort of journal emitter and hence know exactly what regions can
have changed. That would require a modified/customised rsync and a modified
journal or some sort of write hook.
If rsync on small journals does not work this could be investigated further.
Re: Artemis Disaster Recovery options
Posted by Justin Bertram <jb...@redhat.com>.
> What config, if any, would the node in the secondary DC need to have in
common with the nodes in the primary DC?
It would need to have the same essential configuration (e.g. same
addresses, queues, etc.), but it wouldn't necessarily need to be clustered
or have its own HA config, etc.
> I assume my secondary DC node would have to have the same node id so that
it can treat the journals as its own?
That warning is really for clusters where all the nodes will be active
concurrently. You can't have multiple nodes running concurrently with the
same node ID. In the case of backups it is expected that they will have
the same node ID. Therefore, there's no worry with copying the full
journal.
> I assume the node in the secondary DC would not need to be/must not be in
a cluster the primary DC's nodes?
Correct. The node in the secondary DC wouldn't even be active/started.
Starting the node would happen manually after the "disaster" happened using
the latest data backup from the primary DC.
> I assume there would need to be a node per data directory being copied,
e.g. if I have two active nodes in the primary DC then I will need to
standby nodes in the secondary DC.
Correct. Each node in the cluster owns messages independently of the other
cluster nodes so each node in the cluster should be backed up in order not
to lose data.
> What if any file/directory management would need to be done on the
secondary DC file structure if doing a straight file copy. I wouldn't want
to copy the whole directory every 15 minutes for example, but rather just
reflect the changes.
There is a challenge with just copying changes because the journal files
can and will be re-used. This is where a replicated block device would be
nice.
> I assume the secondary DC nodes should be cold until required and then be
manually activated?
Correct.
> What would be ideal I think would be...
There are lots of possible, graceful solutions here. Almost anything would
be better than what we have right now (i.e. nothing). Contributions are
always welcome.
Justin
On Wed, Nov 29, 2017 at 8:59 AM, John Bundred <jo...@ipfin.co.uk>
wrote:
> Thanks Justin. Do you know what sort of considerations I would need to
> make
> if doing shipping, e.g.
>
> 1) What config, if any, would the node in the secondary DC need to have in
> common with the nodes in the primary DC? e.g. see the warning box
> regarding
> copying data directories and unique node id here
> <http://activemq.apache.org/artemis/docs/latest/clusters.html> . I
> assume
> my secondary DC node would have to have the same node id so that it can
> treat the journals as its own?
> 2) I assume the node in the secondary DC would not need to be/must not be
> in
> a cluster the primary DC's nodes?
> 3) I assume there would need to be a node per data directory being copied,
> e.g. if I have two active nodes in the primary DC then I will need to
> standby nodes in the secondary DC.
> 4) What if any file/directory management would need to be done on the
> secondary DC file structure if doing a straight file copy. I wouldn't want
> to copy the whole directory every 15 minutes for example, but rather just
> reflect the changes.
> 5) I assume the secondary DC nodes should be cold until required and then
> be
> manually activated?
>
> If anyone else has experience of doing this (DR requirements aren't rare
> :))
> then I'd be very keen to understand your experiences.
>
> Another option I totally forgot about would be to use a core bridge to push
> messages across to a cluster in the secondary DC. The problem with this I
> guess is that as messages are removed from the primary DC queues they will
> remain in the secondary DC queues. Not sure if there is a TTL that can be
> configured to provide some pruning of messages from the secondary DC
> queues.
> However if this TTL is set too large then a large amount of duplicate
> messages would be delivered, or if it is set too low and the failover
> doesn't occur in time then there will be message loss.
>
> What would be ideal I think would be the option to use shared storage for
> local HA, but be able to combine this with the replication option for
> remote
> DR to a cold backup node. So rather than live-backup being a one to one
> relationship, a live node could have 1 shared storage backup and 1
> replication backup, but only one of which can be auto failed to.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: Artemis Disaster Recovery options
Posted by John Bundred <jo...@ipfin.co.uk>.
Thanks Justin. Do you know what sort of considerations I would need to make
if doing shipping, e.g.
1) What config, if any, would the node in the secondary DC need to have in
common with the nodes in the primary DC? e.g. see the warning box regarding
copying data directories and unique node id here
<http://activemq.apache.org/artemis/docs/latest/clusters.html> . I assume
my secondary DC node would have to have the same node id so that it can
treat the journals as its own?
2) I assume the node in the secondary DC would not need to be/must not be in
a cluster the primary DC's nodes?
3) I assume there would need to be a node per data directory being copied,
e.g. if I have two active nodes in the primary DC then I will need to
standby nodes in the secondary DC.
4) What if any file/directory management would need to be done on the
secondary DC file structure if doing a straight file copy. I wouldn't want
to copy the whole directory every 15 minutes for example, but rather just
reflect the changes.
5) I assume the secondary DC nodes should be cold until required and then be
manually activated?
If anyone else has experience of doing this (DR requirements aren't rare :))
then I'd be very keen to understand your experiences.
Another option I totally forgot about would be to use a core bridge to push
messages across to a cluster in the secondary DC. The problem with this I
guess is that as messages are removed from the primary DC queues they will
remain in the secondary DC queues. Not sure if there is a TTL that can be
configured to provide some pruning of messages from the secondary DC queues.
However if this TTL is set too large then a large amount of duplicate
messages would be delivered, or if it is set too low and the failover
doesn't occur in time then there will be message loss.
What would be ideal I think would be the option to use shared storage for
local HA, but be able to combine this with the replication option for remote
DR to a cold backup node. So rather than live-backup being a one to one
relationship, a live node could have 1 shared storage backup and 1
replication backup, but only one of which can be auto failed to.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Artemis Disaster Recovery options
Posted by Justin Bertram <jb...@apache.org>.
There's nothing built in to Artemis at this point specifically for the DR
use case. However, I believe the "data" directory (where persistent data
is stored by default) can be replicated (e.g. via a block-level storage
replication solution) or "shipped" via an external process (e.g. rsync) to
a DR backup.
Justin
On Wed, Nov 29, 2017 at 5:45 AM, JohnBeeBee <jo...@ipfin.co.uk>
wrote:
> I am looking at options for handling local failover and disaster recovery.
> Our existing primary and secondary data centre hosted services run mainly
> active-passive and have SAN storage but do not support SAN replication.
> There is a dedicated network between the two DC's so we have fast, reliable
> connectivity.
>
> For local single node failure in the primary we would failover to a local
> backup server configured to use shared storage, however I'm not sure what
> the best options are for handling a complete primary DC failure so that we
> can failover to the secondary DC. As I said we don't have SAN replication.
> Is there anything similar to SQL log shipping that can be performed? We
> currently use BizTalk which uses SQL log shipping and so we already accept
> a
> certain amount of message loss in the case of a DR. Even better is there
> an
> option which would remove all data loss from Artemis in the case of a full
> DR scenario?
>
> Thanks
>
> John
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: Artemis Disaster Recovery options
Posted by Justin Bertram <jb...@apache.org>.
> So when the network is down between master and slave (eg slaves network
card fails)... the master will keep ACK-ing messages it receieves in this
case?
In general, yes. Like I said before, the master can be configured to
initiate a quorum vote and will shut itself down if it's isolated.
> If so -- is there some timeout setting that tells the master how long it
should wait before considering the slave as failed?
As soon as the network between the master and slave fails then the master
considers the slave as failed. This is not configurable.
> Aren't [messages, acks, etc] stored as transactions within the broker
journal
Only records which *need* to be stored as part of a transaction are stored
as such. Lots of records stand on their own, not as a part of a
transaction. All that said, I'm not sure it's really relevant.
> So if there is an incomplete/corrupt transaction in the journal wouldn't
the broker just roll it back and ignore it?
Essentially, yes. At least that's what I understand from reading the code
[1].
Justin
[1]
https://github.com/apache/activemq-artemis/blob/master/artemis-journal/src/main/java/org/apache/activemq/artemis/core/journal/impl/JournalImpl.java#L488
On Sun, Jun 23, 2019 at 7:24 PM warm-sun <wa...@tutanota.com> wrote:
> >>> Technically speaking, replication is asynchronous. However, the broker
> will not send a response to the client until it has received a reply from
> the slave that the data has been received.
> ...
> >>> If the network between the master and slave goes down then by default
> >>> the master continues like nothing happened.
>
> So when the network is down between master and slave (eg slaves network
> card
> fails)... the master will keep ACK-ing messages it receieves in this case?
> If so -- is there some timeout setting that tells the master how long it
> should wait before considering the slave as failed?
>
> -----
> >>> For what it's worth, I've found that data integrity and high
> performance
> >>> and generally at odds with each other.
> Agreed. A balance has to be found. We will be replicating with the built-in
> replication on the LAN within the same data centers... and replicating via
> asynchronous DRBD across data centers (best effort -- but not guaranteed)
>
> -----
> >>> I'm not sure the kind of data loss you're describing has ever been
> >>> tested. As far as I know, Artemis expects the data it writes to the
> >>> journal to still be in the journal when it is re-loaded. In general, I
> >>> would expect that any solution designed to ensure data integrity would
> >>> consider message loss or imperfect replication an unacceptable failure.
>
> A small amount of message loss may be acceptable across data center
> replication -- but corrupt journals that a broker cannot use may not be. A
> Couple of questions on this very important point:
> 1)
> Aren't [messages, acks, etc] stored as transactions within the broker
> journal ie:
> begin
> message
> end
>
> So if there is an incomplete/corrupt transaction in the journal wouldn't
> the
> broker just roll it back and ignore it?
> This would lead to message loss but not journal corruption. Is this how the
> journal / broker work?
>
> 2)
> What if only journal blocks that were full were replicated to the other
> data
> center... would this ensure an uncorrupted journal? ie does the broker /
> journal only store complete messages at the end of journal blocks (as the
> journal block is getting full)? Another way to say this: can the journal
> store 1/2 a message at the end of 1 journal block and the other 1/2 at the
> next journal block?
>
> Case 1) would seem to be the preferable solution (imho)... is a feature
> request for this likely to be implemented?
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Artemis Disaster Recovery options
Posted by Clebert Suconic <cl...@gmail.com>.
Ohhh.. you're talking about rsync... perhaps if we disabled reclaim..
it would always be a new file... as right now we reuse a file
if we disabled reclaim for your case.. it wouldn't need to reuse
files... and it wouldn't have a case of corruption.
On Mon, Jun 24, 2019 at 2:40 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> I don't understand what you're talking about with small corruptions of
> the journal?
> We write on the backup.. and wait for confirmation.. so clients are
> blocked until the backup has a copy of the data.
>
>
> On Sun, Jun 23, 2019 at 8:24 PM warm-sun <wa...@tutanota.com> wrote:
> >
> > >>> Technically speaking, replication is asynchronous. However, the broker
> > will not send a response to the client until it has received a reply from
> > the slave that the data has been received.
> > ...
> > >>> If the network between the master and slave goes down then by default
> > >>> the master continues like nothing happened.
> >
> > So when the network is down between master and slave (eg slaves network card
> > fails)... the master will keep ACK-ing messages it receieves in this case?
> > If so -- is there some timeout setting that tells the master how long it
> > should wait before considering the slave as failed?
> >
> > -----
> > >>> For what it's worth, I've found that data integrity and high performance
> > >>> and generally at odds with each other.
> > Agreed. A balance has to be found. We will be replicating with the built-in
> > replication on the LAN within the same data centers... and replicating via
> > asynchronous DRBD across data centers (best effort -- but not guaranteed)
> >
> > -----
> > >>> I'm not sure the kind of data loss you're describing has ever been
> > >>> tested. As far as I know, Artemis expects the data it writes to the
> > >>> journal to still be in the journal when it is re-loaded. In general, I
> > >>> would expect that any solution designed to ensure data integrity would
> > >>> consider message loss or imperfect replication an unacceptable failure.
> >
> > A small amount of message loss may be acceptable across data center
> > replication -- but corrupt journals that a broker cannot use may not be. A
> > Couple of questions on this very important point:
> > 1)
> > Aren't [messages, acks, etc] stored as transactions within the broker
> > journal ie:
> > begin
> > message
> > end
> >
> > So if there is an incomplete/corrupt transaction in the journal wouldn't the
> > broker just roll it back and ignore it?
> > This would lead to message loss but not journal corruption. Is this how the
> > journal / broker work?
> >
> > 2)
> > What if only journal blocks that were full were replicated to the other data
> > center... would this ensure an uncorrupted journal? ie does the broker /
> > journal only store complete messages at the end of journal blocks (as the
> > journal block is getting full)? Another way to say this: can the journal
> > store 1/2 a message at the end of 1 journal block and the other 1/2 at the
> > next journal block?
> >
> > Case 1) would seem to be the preferable solution (imho)... is a feature
> > request for this likely to be implemented?
> >
> >
> >
> > --
> > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
>
>
> --
> Clebert Suconic
--
Clebert Suconic
Re: Artemis Disaster Recovery options
Posted by Clebert Suconic <cl...@gmail.com>.
I don't understand what you're talking about with small corruptions of
the journal?
We write on the backup.. and wait for confirmation.. so clients are
blocked until the backup has a copy of the data.
On Sun, Jun 23, 2019 at 8:24 PM warm-sun <wa...@tutanota.com> wrote:
>
> >>> Technically speaking, replication is asynchronous. However, the broker
> will not send a response to the client until it has received a reply from
> the slave that the data has been received.
> ...
> >>> If the network between the master and slave goes down then by default
> >>> the master continues like nothing happened.
>
> So when the network is down between master and slave (eg slaves network card
> fails)... the master will keep ACK-ing messages it receieves in this case?
> If so -- is there some timeout setting that tells the master how long it
> should wait before considering the slave as failed?
>
> -----
> >>> For what it's worth, I've found that data integrity and high performance
> >>> and generally at odds with each other.
> Agreed. A balance has to be found. We will be replicating with the built-in
> replication on the LAN within the same data centers... and replicating via
> asynchronous DRBD across data centers (best effort -- but not guaranteed)
>
> -----
> >>> I'm not sure the kind of data loss you're describing has ever been
> >>> tested. As far as I know, Artemis expects the data it writes to the
> >>> journal to still be in the journal when it is re-loaded. In general, I
> >>> would expect that any solution designed to ensure data integrity would
> >>> consider message loss or imperfect replication an unacceptable failure.
>
> A small amount of message loss may be acceptable across data center
> replication -- but corrupt journals that a broker cannot use may not be. A
> Couple of questions on this very important point:
> 1)
> Aren't [messages, acks, etc] stored as transactions within the broker
> journal ie:
> begin
> message
> end
>
> So if there is an incomplete/corrupt transaction in the journal wouldn't the
> broker just roll it back and ignore it?
> This would lead to message loss but not journal corruption. Is this how the
> journal / broker work?
>
> 2)
> What if only journal blocks that were full were replicated to the other data
> center... would this ensure an uncorrupted journal? ie does the broker /
> journal only store complete messages at the end of journal blocks (as the
> journal block is getting full)? Another way to say this: can the journal
> store 1/2 a message at the end of 1 journal block and the other 1/2 at the
> next journal block?
>
> Case 1) would seem to be the preferable solution (imho)... is a feature
> request for this likely to be implemented?
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
--
Clebert Suconic
Re: Artemis Disaster Recovery options
Posted by warm-sun <wa...@tutanota.com>.
>>> Technically speaking, replication is asynchronous. However, the broker
will not send a response to the client until it has received a reply from
the slave that the data has been received.
...
>>> If the network between the master and slave goes down then by default
>>> the master continues like nothing happened.
So when the network is down between master and slave (eg slaves network card
fails)... the master will keep ACK-ing messages it receieves in this case?
If so -- is there some timeout setting that tells the master how long it
should wait before considering the slave as failed?
-----
>>> For what it's worth, I've found that data integrity and high performance
>>> and generally at odds with each other.
Agreed. A balance has to be found. We will be replicating with the built-in
replication on the LAN within the same data centers... and replicating via
asynchronous DRBD across data centers (best effort -- but not guaranteed)
-----
>>> I'm not sure the kind of data loss you're describing has ever been
>>> tested. As far as I know, Artemis expects the data it writes to the
>>> journal to still be in the journal when it is re-loaded. In general, I
>>> would expect that any solution designed to ensure data integrity would
>>> consider message loss or imperfect replication an unacceptable failure.
A small amount of message loss may be acceptable across data center
replication -- but corrupt journals that a broker cannot use may not be. A
Couple of questions on this very important point:
1)
Aren't [messages, acks, etc] stored as transactions within the broker
journal ie:
begin
message
end
So if there is an incomplete/corrupt transaction in the journal wouldn't the
broker just roll it back and ignore it?
This would lead to message loss but not journal corruption. Is this how the
journal / broker work?
2)
What if only journal blocks that were full were replicated to the other data
center... would this ensure an uncorrupted journal? ie does the broker /
journal only store complete messages at the end of journal blocks (as the
journal block is getting full)? Another way to say this: can the journal
store 1/2 a message at the end of 1 journal block and the other 1/2 at the
next journal block?
Case 1) would seem to be the preferable solution (imho)... is a feature
request for this likely to be implemented?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Artemis Disaster Recovery options
Posted by Justin Bertram <jb...@apache.org>.
> RedHat AMQ 7 (which is using Artemis under the hood) in their
"configuring broker" documentation recommend NOT using [HA replication]
across data centers.What is the Artemis position (not on AMQ, but if using
Artemis)?
Replication was designed to be used across a low-latency, high-performance
network connection which generally is not the kind of connection you have
between data centers. It doesn't matter whether you're using Artemis
standalone or embedded into something else.
> Is this HA replication always: synchronous/blocking or is there an
asynchronous version too?
Technically speaking, replication is asynchronous. However, the broker will
not send a response to the client until it has received a reply from the
slave that the data has been received. Therefore, once the client receives
the response from the broker it is assured that both the master and the
slave have the data.
> If the network goes down between master and slave: what happens to the
service the prod master brokers provide (does it block clients)?
If the network between the master and slave goes down then by default the
master continues like nothing happened. The master can be configured to
initiate a quorum vote in this situation to determine whether or not it
should remain alive or shut itself down (e.g. in the case that it's
isolated).
> For high performance scenarios: is it still the recommendation to use
asynchronous DRBD (
https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device) across
data centers?
If you're referring to what I said in the original response on this thread
then I think "recommendation" is too strong of a word. I was simply
offering an idea of what I believed might work.
For what it's worth, I've found that data integrity and high performance
and generally at odds with each other.
> Using asynchronous replication can lead to small message loss and
imperfect replication of the journal. How resilient is Artemis to these
small corruptions of the journal? Can it start the broker and ignore the
"corrupt"/incomplete replica blocks?
I'm not sure the kind of data loss you're describing has ever been tested.
As far as I know, Artemis expects the data it writes to the journal to
still be in the journal when it is re-loaded.
In general, I would expect that any solution designed to ensure data
integrity would consider message loss or imperfect replication an
unacceptable failure.
> Is there any existing documentation on this?
I'm not aware of any.
Justin
On Wed, Jun 19, 2019 at 5:05 PM warm-sun <wa...@tutanota.com> wrote:
> I have a very similar scenario to the original post. (Multi data center
> replication is required)
> I have read all the documentation -- but am unclear about a couple of
> points:
>
> 1) RedHat AMQ 7 (which is using Artemis under the hood) in their
> "configuring broker" documentation recommend NOT using [HA replication]
> across data centers.
> What is the Artemis position (not on AMQ, but if using Artemis)?
> Is this HA replication always: synchronous/blocking or is there an
> asynchronous version too?
> If the network goes down between master and slave: what happens to the
> service the prod master brokers provide (does it block clients)?
>
> 2) For high performance scenarios: is it still the recommendation to use
> asynchronous DRBD
> (https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device) across
> data centers?
>
> 3) Using asynchronous replication can lead to small message loss and
> imperfect replication of the journal. How resilient is Artemis to these
> small corruptions of the journal? Can it start the broker and ignore the
> "corrupt"/incomplete replica blocks?
>
> 4) Is there any existing documentation on this? This is what I found:
>
> https://www.linbit.com/downloads/tech-guides/DRBD8_ActiveMQ_HA_and_DR_on_RHEL7.pdf
> (ActiveMQ but not Artemis)
> https://www.rabbitmq.com/pacemaker.html (DRBD as well)
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Artemis Disaster Recovery options
Posted by warm-sun <wa...@tutanota.com>.
I have a very similar scenario to the original post. (Multi data center
replication is required)
I have read all the documentation -- but am unclear about a couple of
points:
1) RedHat AMQ 7 (which is using Artemis under the hood) in their
"configuring broker" documentation recommend NOT using [HA replication]
across data centers.
What is the Artemis position (not on AMQ, but if using Artemis)?
Is this HA replication always: synchronous/blocking or is there an
asynchronous version too?
If the network goes down between master and slave: what happens to the
service the prod master brokers provide (does it block clients)?
2) For high performance scenarios: is it still the recommendation to use
asynchronous DRBD
(https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device) across
data centers?
3) Using asynchronous replication can lead to small message loss and
imperfect replication of the journal. How resilient is Artemis to these
small corruptions of the journal? Can it start the broker and ignore the
"corrupt"/incomplete replica blocks?
4) Is there any existing documentation on this? This is what I found:
https://www.linbit.com/downloads/tech-guides/DRBD8_ActiveMQ_HA_and_DR_on_RHEL7.pdf
(ActiveMQ but not Artemis)
https://www.rabbitmq.com/pacemaker.html (DRBD as well)
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html