DR: Queue backup / replay architecture (Artemis)

[wa...@tutanota.com]

Scenario:
You have simple/normal: request/response queues for clients.
For DR (Disaster Recovery) purposes it is required to backup each queue's messages for 1 month.
(And this is required to be done on the broker itself in backup-request/backup-response queues.)
This way if clients lose their message store -- it can be replayed to them using the backup queues.

With Artemis this could be implemented using:
Diverts to backup-queues (non-exclusive: so they are copied) and then using a transformer plugin which would set the expiry message-property to 1 month (in the diverted backup-queue messages). No expiry-queue would be set, so messages would be dropped when they expire.

1) Have I missed anything?
2) Is there a better way to do this?
3) Will Artemis store (in journal) 1 copy of the message body and use pointers to refer to that message body in diverted/copied messages?
4) Artemis divert will change the message-ID. It would be nice to keep the existing message-ID. If the transformer plugin sets this message-ID back to the original -- will this cause problems?

Re: DR: Queue backup / replay architecture (Artemis)

[Justin Bertram <jb...@apache.org>]

> Have I missed anything?

Perhaps I have misunderstood your requirements, but what you're describing
(i.e. making a copy of the message to keep on the *same* broker) doesn't
conform with what I understand to be "disaster recovery." If a "disaster"
strikes that broker then the backup-request and backup-response data will
be lost. It seems like you'd want to have a bridge to forward that backup
data to a different broker in a geographically isolated area.

> Artemis divert will change the message-ID. It would be nice to keep the
existing message-ID. If the transformer plugin sets this message-ID back to
the original -- will this cause problems?

Yes, I expect this would cause problems. For things like messages sent to
the dead-letter address the broker will set properties on the message to
indicate the original address, queue, and message ID. Your solution could
do the same.


Justin

On Sat, Jul 6, 2019 at 1:26 AM <wa...@tutanota.com> wrote:

> Scenario:
> You have simple/normal: request/response queues for clients.
> For DR (Disaster Recovery) purposes it is required to backup each queue's
> messages for 1 month.
> (And this is required to be done on the broker itself in
> backup-request/backup-response queues.)
> This way if clients lose their message store -- it can be replayed to them
> using the backup queues.
>
> With Artemis this could be implemented using:
> Diverts to backup-queues (non-exclusive: so they are copied) and then
> using a transformer plugin which would set the expiry message-property to 1
> month (in the diverted backup-queue messages). No expiry-queue would be
> set, so messages would be dropped when they expire.
>
> 1) Have I missed anything?
> 2) Is there a better way to do this?
> 3) Will Artemis store (in journal) 1 copy of the message body and use
> pointers to refer to that message body in diverted/copied messages?
> 4) Artemis divert will change the message-ID. It would be nice to keep the
> existing message-ID. If the transformer plugin sets this message-ID back to
> the original -- will this cause problems?
>
>