You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@artemis.apache.org by Mikhail Lukyanov <lu...@gmail.com> on 2026/05/27 12:44:32 UTC
Artemis Cluster with HA and with Mesh Mirror
Hello everyone.
I would like to ask for advice and recommendations regarding a
high-availability cluster with mirroring to another data centers.
Currently we have the following setup: in each data center there are 3
machines/JVMs, each running 2 Artemis instances — a primary of one group
and a backup of another — along with ZooKeeper. In total there are 6
Artemis instances, forming 3 primary/backup pairs managed by CuratorManager
within one data center. Each Artemis instance mirrors data to the other
data center, so that if one data center goes down, we can bring up the full
cluster and continue processing in the other data center. Mirroring is
configured per pair failover: if the primary fails, we send data to its
backup. In this scheme, all broker connections on the standby data center
are disabled. The primary and backup configuration xml in attachment to the
letter.
[image: image.png]
Now there is a requirement to mirror data to a third, additional data
center. This means each active Artemis instance in the first data center
must mirror data to 2 standby data centers. However, this introduces
significant new complications. For example, Artemis1 in standby DC2 has
broker connections both to Artemis1 in active DC1 and to Artemis1 in
standby DC3. As a result, events received via mirroring from Artemis1.DC1
are then attempted to be re-mirrored from Artemis1.DC2 to Artemis1.DC3, and
they accumulate in `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`
— even though Artemis1.DC1 is already forwarding those events directly to
Artemis1.DC3. Mirroring from Artemis1.DC2 to Artemis1.DC3 is disabled, yet
messages continue to accumulate in
`$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
My understanding is that the solution to this problem is Mesh Mirror, as
described in https://issues.apache.org/jira/browse/ARTEMIS-5925.
This raises the question: are Cluster HA and Mesh Mirror compatible with
each other, or are they mutually exclusive mechanisms?
It also follows that ZooKeeper would need to be deployed separately for HA
and for Mesh Mirror — how can this be configured and managed?
By what mechanisms can I actually achieve the desired behavior?
--
*With best regards, Lukyanov Mikhail*
*Tel: **+7-909-69-71-547*
Re: Artemis Cluster with HA and with Mesh Mirror
Posted by M Lukyanov <lu...@gmail.com>.
Got it, I apologize, I'll send it to users.
чт, 28 мая 2026 г. в 13:32, Robbie Gemmell <ro...@gmail.com>:
> That advice was incorrect (I replied to that thread noting so for any
> future readers of it). You should send such messages to the users@
> list as you originally did.
>
> You should re-send your reply below on the existing users@ thread
> (https://lists.apache.org/thread/71nfjlcmkqm573rs2o2qpy0ggfn7lc9j for
> anyone later reading this) given that it has the messages you are
> actually replying about, allowing a complete discussion someone can
> follow later. Let this dev@ thread end.
>
> On Thu, 28 May 2026 at 09:59, M Lukyanov <lu...@gmail.com> wrote:
> >
> > Yes, that all makes sense — thank you for the information. I was advised
> > yesterday to also post my question to the dev@artemis.apache.org mailing
> > list, so I'm sending it here as well.
> >
> > I'd like to clarify a few points regarding your answer.
> >
> > *1. Replacing classic HA with mirroring*
> >
> > If I drop classic active/passive HA, am I correct that the replacement
> > would be synchronous mirroring between two active nodes? That is, I could
> > keep the cluster for load and client distribution, and replace the
> Artemis
> > instance that used to be the backup of a primary with a node that does
> > synchronous mirroring but has client connections blocked via the lock
> > manager — with the former backup becoming an additional mirroring target.
> >
> > This scheme raises several questions:
> >
> > 1.1 Can this active node (the former backup) participate in the same
> > cluster as the primary? Are there any pitfalls here?
> >
> > 1.2 Does this scheme require additional settings analogous to classic HA
> so
> > that specifically the primary in DataCenter1 acquires the lock
> > preferentially and reclaims it after recovery — something like `
> > *allow-failback*` and `*failover-on-shutdown*`?
> >
> > 1.3 Mirroring does not forward messages that already existed in the
> broker
> > at the time mirroring was configured ("Previously existing messages will
> > not be forwarded"). In that case, how do I correctly synchronize the
> > already-accumulated messages when migrating to this scheme?
> >
> > *2. Combining classic HA and Mesh Mirror*
> >
> > If I combine classic HA and Mesh Mirror, is my understanding of the
> > ZooKeeper layout correct? For classic HA, this is an ensemble of 3
> > ZooKeeper nodes inside DataCenter1 (DC1). Additionally, for Mesh Mirror
> of
> > the artemis1 group, I would need a separate ensemble of 3 ZooKeeper nodes
> > spread across data centers: one each on jvm1 in DC1, DC2, and DC3 — 3
> nodes
> > to avoid split-brain.
> >
> > 2.1 Is this understanding correct? Can the number of ZooKeeper nodes be
> > reduced while still protecting against split-brain, or is it needed to
> > combine all ZooKeeper nodes into a single shared ensemble?
> >
> > 2.2 Same question as 1.2: how can I make artemis1 in DC1 the preferred
> node
> > so that it acquires the lock first while it is running? Or is the only
> > option to manually shut down the other nodes so that only one remains?
> >
> > чт, 28 мая 2026 г. в 07:06, Clebert Suconic <cl...@gmail.com>:
> >
> > > I answered you on users. Didn’t I ?
> > >
> > > Clebert Suconic
> > >
> > >
> > > On Wed, May 27, 2026 at 8:45 AM Mikhail Lukyanov <
> lukyanovmiv@gmail.com>
> > > wrote:
> > >
> > >> Hello everyone.
> > >>
> > >> I would like to ask for advice and recommendations regarding a
> > >> high-availability cluster with mirroring to another data centers.
> > >>
> > >> Currently we have the following setup: in each data center there are 3
> > >> machines/JVMs, each running 2 Artemis instances — a primary of one
> group
> > >> and a backup of another — along with ZooKeeper. In total there are 6
> > >> Artemis instances, forming 3 primary/backup pairs managed by
> CuratorManager
> > >> within one data center. Each Artemis instance mirrors data to the
> other
> > >> data center, so that if one data center goes down, we can bring up
> the full
> > >> cluster and continue processing in the other data center. Mirroring is
> > >> configured per pair failover: if the primary fails, we send data to
> its
> > >> backup. In this scheme, all broker connections on the standby data
> center
> > >> are disabled. The primary and backup configuration xml in attachment
> to the
> > >> letter.
> > >>
> > >> [image: image.png]
> > >> Now there is a requirement to mirror data to a third, additional data
> > >> center. This means each active Artemis instance in the first data
> center
> > >> must mirror data to 2 standby data centers. However, this introduces
> > >> significant new complications. For example, Artemis1 in standby DC2
> has
> > >> broker connections both to Artemis1 in active DC1 and to Artemis1 in
> > >> standby DC3. As a result, events received via mirroring from
> Artemis1.DC1
> > >> are then attempted to be re-mirrored from Artemis1.DC2 to
> Artemis1.DC3, and
> > >> they accumulate in
> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`
> > >> — even though Artemis1.DC1 is already forwarding those events
> directly to
> > >> Artemis1.DC3. Mirroring from Artemis1.DC2 to Artemis1.DC3 is
> disabled, yet
> > >> messages continue to accumulate in
> > >> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
> > >>
> > >> My understanding is that the solution to this problem is Mesh Mirror,
> as
> > >> described in https://issues.apache.org/jira/browse/ARTEMIS-5925.
> > >>
> > >> This raises the question: are Cluster HA and Mesh Mirror compatible
> with
> > >> each other, or are they mutually exclusive mechanisms?
> > >>
> > >> It also follows that ZooKeeper would need to be deployed separately
> for
> > >> HA and for Mesh Mirror — how can this be configured and managed?
> > >>
> > >> By what mechanisms can I actually achieve the desired behavior?
> > >>
> > >> --
> > >> *With best regards, Lukyanov Mikhail*
> > >> *Tel: **+7-909-69-71-547*
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@artemis.apache.org
> > >> For additional commands, e-mail: dev-help@artemis.apache.org
> > >
> > >
> >
> > --
> > *With best regards, Lukyanov Mikhail*
> > *Tel: **+7-909-69-71-547*
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@artemis.apache.org
> For additional commands, e-mail: dev-help@artemis.apache.org
>
>
--
*With best regards, Lukyanov Mikhail*
*Tel: **+7-909-69-71-547*
Re: Artemis Cluster with HA and with Mesh Mirror
Posted by Robbie Gemmell <ro...@gmail.com>.
That advice was incorrect (I replied to that thread noting so for any
future readers of it). You should send such messages to the users@
list as you originally did.
You should re-send your reply below on the existing users@ thread
(https://lists.apache.org/thread/71nfjlcmkqm573rs2o2qpy0ggfn7lc9j for
anyone later reading this) given that it has the messages you are
actually replying about, allowing a complete discussion someone can
follow later. Let this dev@ thread end.
On Thu, 28 May 2026 at 09:59, M Lukyanov <lu...@gmail.com> wrote:
>
> Yes, that all makes sense — thank you for the information. I was advised
> yesterday to also post my question to the dev@artemis.apache.org mailing
> list, so I'm sending it here as well.
>
> I'd like to clarify a few points regarding your answer.
>
> *1. Replacing classic HA with mirroring*
>
> If I drop classic active/passive HA, am I correct that the replacement
> would be synchronous mirroring between two active nodes? That is, I could
> keep the cluster for load and client distribution, and replace the Artemis
> instance that used to be the backup of a primary with a node that does
> synchronous mirroring but has client connections blocked via the lock
> manager — with the former backup becoming an additional mirroring target.
>
> This scheme raises several questions:
>
> 1.1 Can this active node (the former backup) participate in the same
> cluster as the primary? Are there any pitfalls here?
>
> 1.2 Does this scheme require additional settings analogous to classic HA so
> that specifically the primary in DataCenter1 acquires the lock
> preferentially and reclaims it after recovery — something like `
> *allow-failback*` and `*failover-on-shutdown*`?
>
> 1.3 Mirroring does not forward messages that already existed in the broker
> at the time mirroring was configured ("Previously existing messages will
> not be forwarded"). In that case, how do I correctly synchronize the
> already-accumulated messages when migrating to this scheme?
>
> *2. Combining classic HA and Mesh Mirror*
>
> If I combine classic HA and Mesh Mirror, is my understanding of the
> ZooKeeper layout correct? For classic HA, this is an ensemble of 3
> ZooKeeper nodes inside DataCenter1 (DC1). Additionally, for Mesh Mirror of
> the artemis1 group, I would need a separate ensemble of 3 ZooKeeper nodes
> spread across data centers: one each on jvm1 in DC1, DC2, and DC3 — 3 nodes
> to avoid split-brain.
>
> 2.1 Is this understanding correct? Can the number of ZooKeeper nodes be
> reduced while still protecting against split-brain, or is it needed to
> combine all ZooKeeper nodes into a single shared ensemble?
>
> 2.2 Same question as 1.2: how can I make artemis1 in DC1 the preferred node
> so that it acquires the lock first while it is running? Or is the only
> option to manually shut down the other nodes so that only one remains?
>
> чт, 28 мая 2026 г. в 07:06, Clebert Suconic <cl...@gmail.com>:
>
> > I answered you on users. Didn’t I ?
> >
> > Clebert Suconic
> >
> >
> > On Wed, May 27, 2026 at 8:45 AM Mikhail Lukyanov <lu...@gmail.com>
> > wrote:
> >
> >> Hello everyone.
> >>
> >> I would like to ask for advice and recommendations regarding a
> >> high-availability cluster with mirroring to another data centers.
> >>
> >> Currently we have the following setup: in each data center there are 3
> >> machines/JVMs, each running 2 Artemis instances — a primary of one group
> >> and a backup of another — along with ZooKeeper. In total there are 6
> >> Artemis instances, forming 3 primary/backup pairs managed by CuratorManager
> >> within one data center. Each Artemis instance mirrors data to the other
> >> data center, so that if one data center goes down, we can bring up the full
> >> cluster and continue processing in the other data center. Mirroring is
> >> configured per pair failover: if the primary fails, we send data to its
> >> backup. In this scheme, all broker connections on the standby data center
> >> are disabled. The primary and backup configuration xml in attachment to the
> >> letter.
> >>
> >> [image: image.png]
> >> Now there is a requirement to mirror data to a third, additional data
> >> center. This means each active Artemis instance in the first data center
> >> must mirror data to 2 standby data centers. However, this introduces
> >> significant new complications. For example, Artemis1 in standby DC2 has
> >> broker connections both to Artemis1 in active DC1 and to Artemis1 in
> >> standby DC3. As a result, events received via mirroring from Artemis1.DC1
> >> are then attempted to be re-mirrored from Artemis1.DC2 to Artemis1.DC3, and
> >> they accumulate in `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`
> >> — even though Artemis1.DC1 is already forwarding those events directly to
> >> Artemis1.DC3. Mirroring from Artemis1.DC2 to Artemis1.DC3 is disabled, yet
> >> messages continue to accumulate in
> >> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
> >>
> >> My understanding is that the solution to this problem is Mesh Mirror, as
> >> described in https://issues.apache.org/jira/browse/ARTEMIS-5925.
> >>
> >> This raises the question: are Cluster HA and Mesh Mirror compatible with
> >> each other, or are they mutually exclusive mechanisms?
> >>
> >> It also follows that ZooKeeper would need to be deployed separately for
> >> HA and for Mesh Mirror — how can this be configured and managed?
> >>
> >> By what mechanisms can I actually achieve the desired behavior?
> >>
> >> --
> >> *With best regards, Lukyanov Mikhail*
> >> *Tel: **+7-909-69-71-547*
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@artemis.apache.org
> >> For additional commands, e-mail: dev-help@artemis.apache.org
> >
> >
>
> --
> *With best regards, Lukyanov Mikhail*
> *Tel: **+7-909-69-71-547*
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@artemis.apache.org
For additional commands, e-mail: dev-help@artemis.apache.org
Re: Artemis Cluster with HA and with Mesh Mirror
Posted by M Lukyanov <lu...@gmail.com>.
Yes, that all makes sense — thank you for the information. I was advised
yesterday to also post my question to the dev@artemis.apache.org mailing
list, so I'm sending it here as well.
I'd like to clarify a few points regarding your answer.
*1. Replacing classic HA with mirroring*
If I drop classic active/passive HA, am I correct that the replacement
would be synchronous mirroring between two active nodes? That is, I could
keep the cluster for load and client distribution, and replace the Artemis
instance that used to be the backup of a primary with a node that does
synchronous mirroring but has client connections blocked via the lock
manager — with the former backup becoming an additional mirroring target.
This scheme raises several questions:
1.1 Can this active node (the former backup) participate in the same
cluster as the primary? Are there any pitfalls here?
1.2 Does this scheme require additional settings analogous to classic HA so
that specifically the primary in DataCenter1 acquires the lock
preferentially and reclaims it after recovery — something like `
*allow-failback*` and `*failover-on-shutdown*`?
1.3 Mirroring does not forward messages that already existed in the broker
at the time mirroring was configured ("Previously existing messages will
not be forwarded"). In that case, how do I correctly synchronize the
already-accumulated messages when migrating to this scheme?
*2. Combining classic HA and Mesh Mirror*
If I combine classic HA and Mesh Mirror, is my understanding of the
ZooKeeper layout correct? For classic HA, this is an ensemble of 3
ZooKeeper nodes inside DataCenter1 (DC1). Additionally, for Mesh Mirror of
the artemis1 group, I would need a separate ensemble of 3 ZooKeeper nodes
spread across data centers: one each on jvm1 in DC1, DC2, and DC3 — 3 nodes
to avoid split-brain.
2.1 Is this understanding correct? Can the number of ZooKeeper nodes be
reduced while still protecting against split-brain, or is it needed to
combine all ZooKeeper nodes into a single shared ensemble?
2.2 Same question as 1.2: how can I make artemis1 in DC1 the preferred node
so that it acquires the lock first while it is running? Or is the only
option to manually shut down the other nodes so that only one remains?
чт, 28 мая 2026 г. в 07:06, Clebert Suconic <cl...@gmail.com>:
> I answered you on users. Didn’t I ?
>
> Clebert Suconic
>
>
> On Wed, May 27, 2026 at 8:45 AM Mikhail Lukyanov <lu...@gmail.com>
> wrote:
>
>> Hello everyone.
>>
>> I would like to ask for advice and recommendations regarding a
>> high-availability cluster with mirroring to another data centers.
>>
>> Currently we have the following setup: in each data center there are 3
>> machines/JVMs, each running 2 Artemis instances — a primary of one group
>> and a backup of another — along with ZooKeeper. In total there are 6
>> Artemis instances, forming 3 primary/backup pairs managed by CuratorManager
>> within one data center. Each Artemis instance mirrors data to the other
>> data center, so that if one data center goes down, we can bring up the full
>> cluster and continue processing in the other data center. Mirroring is
>> configured per pair failover: if the primary fails, we send data to its
>> backup. In this scheme, all broker connections on the standby data center
>> are disabled. The primary and backup configuration xml in attachment to the
>> letter.
>>
>> [image: image.png]
>> Now there is a requirement to mirror data to a third, additional data
>> center. This means each active Artemis instance in the first data center
>> must mirror data to 2 standby data centers. However, this introduces
>> significant new complications. For example, Artemis1 in standby DC2 has
>> broker connections both to Artemis1 in active DC1 and to Artemis1 in
>> standby DC3. As a result, events received via mirroring from Artemis1.DC1
>> are then attempted to be re-mirrored from Artemis1.DC2 to Artemis1.DC3, and
>> they accumulate in `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`
>> — even though Artemis1.DC1 is already forwarding those events directly to
>> Artemis1.DC3. Mirroring from Artemis1.DC2 to Artemis1.DC3 is disabled, yet
>> messages continue to accumulate in
>> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
>>
>> My understanding is that the solution to this problem is Mesh Mirror, as
>> described in https://issues.apache.org/jira/browse/ARTEMIS-5925.
>>
>> This raises the question: are Cluster HA and Mesh Mirror compatible with
>> each other, or are they mutually exclusive mechanisms?
>>
>> It also follows that ZooKeeper would need to be deployed separately for
>> HA and for Mesh Mirror — how can this be configured and managed?
>>
>> By what mechanisms can I actually achieve the desired behavior?
>>
>> --
>> *With best regards, Lukyanov Mikhail*
>> *Tel: **+7-909-69-71-547*
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@artemis.apache.org
>> For additional commands, e-mail: dev-help@artemis.apache.org
>
>
--
*With best regards, Lukyanov Mikhail*
*Tel: **+7-909-69-71-547*
Re: Artemis Cluster with HA and with Mesh Mirror
Posted by Clebert Suconic <cl...@gmail.com>.
I answered you on users. Didn’t I ?
Clebert Suconic
On Wed, May 27, 2026 at 8:45 AM Mikhail Lukyanov <lu...@gmail.com>
wrote:
> Hello everyone.
>
> I would like to ask for advice and recommendations regarding a
> high-availability cluster with mirroring to another data centers.
>
> Currently we have the following setup: in each data center there are 3
> machines/JVMs, each running 2 Artemis instances — a primary of one group
> and a backup of another — along with ZooKeeper. In total there are 6
> Artemis instances, forming 3 primary/backup pairs managed by CuratorManager
> within one data center. Each Artemis instance mirrors data to the other
> data center, so that if one data center goes down, we can bring up the full
> cluster and continue processing in the other data center. Mirroring is
> configured per pair failover: if the primary fails, we send data to its
> backup. In this scheme, all broker connections on the standby data center
> are disabled. The primary and backup configuration xml in attachment to the
> letter.
>
> [image: image.png]
> Now there is a requirement to mirror data to a third, additional data
> center. This means each active Artemis instance in the first data center
> must mirror data to 2 standby data centers. However, this introduces
> significant new complications. For example, Artemis1 in standby DC2 has
> broker connections both to Artemis1 in active DC1 and to Artemis1 in
> standby DC3. As a result, events received via mirroring from Artemis1.DC1
> are then attempted to be re-mirrored from Artemis1.DC2 to Artemis1.DC3, and
> they accumulate in `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`
> — even though Artemis1.DC1 is already forwarding those events directly to
> Artemis1.DC3. Mirroring from Artemis1.DC2 to Artemis1.DC3 is disabled, yet
> messages continue to accumulate in
> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
>
> My understanding is that the solution to this problem is Mesh Mirror, as
> described in https://issues.apache.org/jira/browse/ARTEMIS-5925.
>
> This raises the question: are Cluster HA and Mesh Mirror compatible with
> each other, or are they mutually exclusive mechanisms?
>
> It also follows that ZooKeeper would need to be deployed separately for HA
> and for Mesh Mirror — how can this be configured and managed?
>
> By what mechanisms can I actually achieve the desired behavior?
>
> --
> *With best regards, Lukyanov Mikhail*
> *Tel: **+7-909-69-71-547*
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@artemis.apache.org
> For additional commands, e-mail: dev-help@artemis.apache.org