You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@artemis.apache.org by Michael <lu...@gmail.com> on 2026/07/21 06:45:36 UTC
Re: Artemis Cluster with HA and with Mesh Mirror
Hello Clebert,
Following up on your last message: you were right that the replica must
not start the
lock coordinator until it is activated, and I went ahead and wrote the
tests for it.
Two things turned out to be broken in our setup (replication ha-policy
inside each DC,
mesh mirror between DCs, acceptors and broker connections bound to a lock
coordinator).
First, ActiveMQServerImpl.internalStart() starts every configured lock
coordinator right
away, whatever the HA role is, so a passive backup contends for the lock
along with the
active primary. If the backup wins it, it holds the lock while none of
its acceptors or
broker connections exist yet, so the elements bound to that coordinator
are down on the
whole group.
Second, even once that is fixed, all the active brokers still race for
the lock: after a
primary fails, its backup needs a few seconds to activate while the
brokers in the other
DCs keep polling every check-period, so the lock usually lands on a
remote broker rather
than on the backup taking over.
I have a patch for both. The coordinators are still created in
internalStart(), since
acceptors and broker connections look them up by name, but polling now
starts in
completeActivation(). On top of that, the auto-start=false you were
planning, as an
attribute on lock-coordinator, plus
startLockCoordinator/stopLockCoordinator management
operations, so a standby DC only takes the lock over when somebody
decides so.
Both are covered by tests that fail without the fix: a replicated pair
sharing one
coordinator, where the passive backup must stay out of the lock and take
it over once the
primary is gone, and the full start/stop cycle through management on an
acceptor bound to
an auto-start=false coordinator.
ARTEMIS-6162: https://issues.apache.org/jira/browse/ARTEMIS-6162
ARTEMIS-6163: https://issues.apache.org/jira/browse/ARTEMIS-6163
PR (one commit per issue, the first one stands on its own):
https://github.com/apache/artemis/pull/6590
Happy to rework anything you don't like, and to move the discussion to
the dev list if
you prefer.
--
With best regards, Lukyanov Michael
пн, 29 июн. 2026 г. в 06:02, Clebert Suconic <cl...@gmail.com>:
> My intention was to replace replication with mirro. For that reason I
> didn’t look at any requirements in The replicated journal.
>
>
> I think I have to make sure the replica won’t start the lock coordinator
> until activated.
>
>
> I don’t know if it will work or not. But since I didn’t add a test I
> don’t know.
>
> Clebert Suconic
>
>
> On Thu, Jun 25, 2026 at 5:44 AM Michael <lu...@gmail.com> wrote:
>
>> Yes, <ha-policy> <replication>.
>>
>> вт, 23 июн. 2026 г., 17:57 Clebert Suconic <cl...@gmail.com>:
>>
>>> in high available setup you mean journal replication?
>>>
>>>
>>> On Fri, Jun 19, 2026 at 7:18 AM Michael <lu...@gmail.com> wrote:
>>>
>>>> Good day.
>>>>
>>>> We managed to set up a *mesh mirror* with synchronous replication
>>>> ha-policy with different groups of Zookeepers. The broker-connections and
>>>> acceptors stay active while the *lock-coordinator* lock is held.
>>>>
>>>> An additional question has come up: when the primary broker fails in a
>>>> high-availability setup, how can we ensure that the *lock is acquired*
>>>> specifically *by its backup* broker that has become active? Is there a
>>>> plan to add such functionality, or could this be achieved some other way?
>>>>
>>>> чт, 28 мая 2026 г. в 19:57, M Lukyanov <lu...@gmail.com>:
>>>>
>>>>> Thank you for the clarification.
>>>>>
>>>>> Of course, I'll follow the design and implementation of these tasks
>>>>> through the dev-list. I'm ready to help if there's anything I can do.
>>>>>
>>>>> For now, I will test various configurations, including the ZK setup,
>>>>> on the latest version of Artemis.
>>>>>
>>>>> чт, 28 мая 2026 г. в 18:58, Clebert Suconic <clebert.suconic@gmail.com
>>>>> >:
>>>>>
>>>>>> 1. if you require synchronous.. there are a few improvements that I
>>>>>> will make on it in the next few months. Right there's no way to make a
>>>>>> mirror to go async if the backup fails. I also want to have a N Copy of the
>>>>>> messages and some transactional control... I'm still on design for this..
>>>>>> but perhaps we could keep in touch for when I'm designing this? I would do
>>>>>> it through the dev-list
>>>>>>
>>>>>>
>>>>>> 2. you can combine classicHA and Mesh Mirror.. shoulddn't be an issue.
>>>>>>
>>>>>> 2.1. I'm not sure about the setup on Zookeeper. perhaps 2.2 would fix
>>>>>> the issue for you.. you just have a set of ZK on each datacenter and use a
>>>>>> manual procedure to activate a Datacenter.
>>>>>>
>>>>>>
>>>>>> 2.2. There's one small change i'm going to make soon. have a way to
>>>>>> manually start a LockManager. Usually DCs are disconnected... and require
>>>>>> human intervention.
>>>>>> I plan to make the lockmanager auto-start=false
>>>>>>
>>>>>> On Thu, May 28, 2026 at 6:52 AM M Lukyanov <lu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you for the information.
>>>>>>>
>>>>>>> I'd like to clarify a few points regarding your answer.
>>>>>>>
>>>>>>> *1. Replacing classic HA with mirroring*
>>>>>>>
>>>>>>> If I drop classic active/passive HA, am I correct that the
>>>>>>> replacement would be synchronous mirroring between two active nodes? That
>>>>>>> is, I could keep the cluster for load and client distribution, and replace
>>>>>>> the Artemis instance that used to be the backup of a primary with a node
>>>>>>> that does synchronous mirroring but has client connections blocked via the
>>>>>>> lock manager — with the former backup becoming an additional mirroring
>>>>>>> target.
>>>>>>>
>>>>>>> This scheme raises several questions:
>>>>>>>
>>>>>>> 1.1 Can this active node (the former backup) participate in the same
>>>>>>> cluster as the primary? Are there any pitfalls here?
>>>>>>>
>>>>>>> 1.2 Does this scheme require additional settings analogous to
>>>>>>> classic HA so that specifically the primary in DataCenter1 acquires the
>>>>>>> lock preferentially and reclaims it after recovery — something like `
>>>>>>> *allow-failback*` and `*failover-on-shutdown*`?
>>>>>>>
>>>>>>> 1.3 Mirroring does not forward messages that already existed in the
>>>>>>> broker at the time mirroring was configured ("Previously existing messages
>>>>>>> will not be forwarded"). In that case, how do I correctly synchronize the
>>>>>>> already-accumulated messages when migrating to this scheme?
>>>>>>>
>>>>>>> *2. Combining classic HA and Mesh Mirror*
>>>>>>>
>>>>>>> If I combine classic HA and Mesh Mirror, is my understanding of the
>>>>>>> ZooKeeper layout correct? For classic HA, this is an ensemble of 3
>>>>>>> ZooKeeper nodes inside DataCenter1 (DC1). Additionally, for Mesh Mirror of
>>>>>>> the artemis1 group, I would need a separate ensemble of 3 ZooKeeper nodes
>>>>>>> spread across data centers: one each on jvm1 in DC1, DC2, and DC3 — 3 nodes
>>>>>>> to avoid split-brain.
>>>>>>>
>>>>>>> 2.1 Is this understanding correct? Can the number of ZooKeeper nodes
>>>>>>> be reduced while still protecting against split-brain, or is it needed to
>>>>>>> combine all ZooKeeper nodes into a single shared ensemble?
>>>>>>>
>>>>>>> 2.2 Same question as 1.2: how can I make artemis1 in DC1 the
>>>>>>> preferred node so that it acquires the lock first while it is running? Or
>>>>>>> is the only option to manually shut down the other nodes so that only one
>>>>>>> remains?
>>>>>>>
>>>>>>> ср, 27 мая 2026 г. в 19:36, Clebert Suconic <
>>>>>>> clebert.suconic@gmail.com>:
>>>>>>>
>>>>>>>> You probably seen this, there's an example here for mesh mirror:
>>>>>>>> (it's good to point it out if someone else finds this thread anyway)
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/apache/artemis-examples/tree/main/examples/features/broker-connection/ha-with-mesh-mirror
>>>>>>>>
>>>>>>>> On Wed, May 27, 2026 at 12:34 PM Clebert Suconic <
>>>>>>>> clebert.suconic@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> When I wrote the MeshMirror example, I was trying to get rid of
>>>>>>>>> replication and clustering... but it should be compatible...
>>>>>>>>>
>>>>>>>>> But yeah.. the lock manager is the solution for this case.
>>>>>>>>>
>>>>>>>>> You don't need a separate Zookeeper instance for any of this... It
>>>>>>>>> just needs a ZK entry point where it can connect to.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I had in the past fixed issues with dual mirroring and
>>>>>>>>> replication, and it should work fine with meshMirror. But it's always best
>>>>>>>>> to test it, right?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If you could only use Mirror on your solution, it would be simpler
>>>>>>>>> though.
>>>>>>>>>
>>>>>>>>> On Wed, May 27, 2026 at 5:48 AM Mikhail Lukyanov <
>>>>>>>>> lukyanovmiv@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello everyone.
>>>>>>>>>>
>>>>>>>>>> I would like to ask for advice and recommendations regarding a
>>>>>>>>>> high-availability cluster with mirroring to another data centers.
>>>>>>>>>>
>>>>>>>>>> Currently we have the following setup: in each data center there
>>>>>>>>>> are 3 machines/JVMs, each running 2 Artemis instances — a primary of one
>>>>>>>>>> group and a backup of another — along with ZooKeeper. In total there are 6
>>>>>>>>>> Artemis instances, forming 3 primary/backup pairs managed by CuratorManager
>>>>>>>>>> within one data center. Each Artemis instance mirrors data to the other
>>>>>>>>>> data center, so that if one data center goes down, we can bring up the full
>>>>>>>>>> cluster and continue processing in the other data center. Mirroring is
>>>>>>>>>> configured per pair failover: if the primary fails, we send data to its
>>>>>>>>>> backup. In this scheme, all broker connections on the standby data center
>>>>>>>>>> are disabled. The primary and backup configuration xml in attachment to the
>>>>>>>>>> letter.
>>>>>>>>>>
>>>>>>>>>> [image: image.png]
>>>>>>>>>> Now there is a requirement to mirror data to a third, additional
>>>>>>>>>> data center. This means each active Artemis instance in the first data
>>>>>>>>>> center must mirror data to 2 standby data centers. However, this introduces
>>>>>>>>>> significant new complications. For example, Artemis1 in standby DC2 has
>>>>>>>>>> broker connections both to Artemis1 in active DC1 and to Artemis1 in
>>>>>>>>>> standby DC3. As a result, events received via mirroring from Artemis1.DC1
>>>>>>>>>> are then attempted to be re-mirrored from Artemis1.DC2 to Artemis1.DC3, and
>>>>>>>>>> they accumulate in `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`
>>>>>>>>>> — even though Artemis1.DC1 is already forwarding those events directly to
>>>>>>>>>> Artemis1.DC3. Mirroring from Artemis1.DC2 to Artemis1.DC3 is disabled, yet
>>>>>>>>>> messages continue to accumulate in
>>>>>>>>>> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
>>>>>>>>>>
>>>>>>>>>> My understanding is that the solution to this problem is Mesh
>>>>>>>>>> Mirror, as described in
>>>>>>>>>> https://issues.apache.org/jira/browse/ARTEMIS-5925.
>>>>>>>>>>
>>>>>>>>>> This raises the question: are Cluster HA and Mesh Mirror
>>>>>>>>>> compatible with each other, or are they mutually exclusive mechanisms?
>>>>>>>>>>
>>>>>>>>>> It also follows that ZooKeeper would need to be deployed
>>>>>>>>>> separately for HA and for Mesh Mirror — how can this be configured and
>>>>>>>>>> managed?
>>>>>>>>>>
>>>>>>>>>> By what mechanisms can I actually achieve the desired behavior?
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
>>>>>>>>>> For additional commands, e-mail: users-help@artemis.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Clebert Suconic
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Clebert Suconic
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Clebert Suconic
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *With best regards, Lukyanov Mikhail*
>>>>> *Tel: **+7-909-69-71-547*
>>>>>
>>>>
>>>>
>>>> --
>>>> *With best regards, Lukyanov Mikhail*
>>>> *Tel: **+7-909-69-71-547*
>>>>
>>>
>>>
>>> --
>>> Clebert Suconic
>>>
>>
--
*With best regards, Lukyanov Mikhail*
*Tel: **+7-909-69-71-547*
Re: Artemis Cluster with HA and with Mesh Mirror
Posted by Michael <lu...@gmail.com>.
Got it, thanks for the merge - and for the heads-up.
I have one more PR open - #6586 (ARTEMIS-6159, bounding the lock manager
start so an unreachable ZooKeeper doesn't hang the broker).
I've added Assisted-by: Claude to its commit message there as well, and
rebased it onto the current main, so it now sits on top of the
merged 6162/6163 work.
пн, 27 июл. 2026 г. в 22:18, Clebert Suconic <cl...@gmail.com>:
> I have pushed -f directly into your branch .. (it's a github feature)..
> please make sure you rebase against your branch if you do anything else on
> it.
>
>
> Also: I added ./artemis lock stop <name> and ./artemis lock start <name>
>
>
> I also sent a Pull Request to the web console:
> https://github.com/apache/artemis-console/pull/217
>
> (I don't know if that's the right way to do it on the web console)
>
> On Thu, Jul 23, 2026 at 5:05 AM Michael <lu...@gmail.com> wrote:
>
>> Thanks, and enjoy your vacation.
>>
>> I'll be around if anything needs reworking.
>>
>> --
>> With best regards, Lukyanov Michael
>>
>> ср, 22 июл. 2026 г. в 16:29, Clebert Suconic <cl...@gmail.com>:
>>
>>> I had a brief look. Thank you so much for these.
>>>
>>> I’m in vacation this week. I will have a proper look on Monday. And I
>>> will merge it.
>>>
>>> Clebert Suconic
>>>
>>>
>>> On Tue, Jul 21, 2026 at 2:46 AM Michael <lu...@gmail.com> wrote:
>>>
>>>> Hello Clebert,
>>>>
>>>> Following up on your last message: you were right that the replica
>>>> must not start the
>>>> lock coordinator until it is activated, and I went ahead and wrote
>>>> the tests for it.
>>>>
>>>> Two things turned out to be broken in our setup (replication
>>>> ha-policy inside each DC,
>>>> mesh mirror between DCs, acceptors and broker connections bound to a
>>>> lock coordinator).
>>>>
>>>> First, ActiveMQServerImpl.internalStart() starts every configured
>>>> lock coordinator right
>>>> away, whatever the HA role is, so a passive backup contends for the
>>>> lock along with the
>>>> active primary. If the backup wins it, it holds the lock while none
>>>> of its acceptors or
>>>> broker connections exist yet, so the elements bound to that
>>>> coordinator are down on the
>>>> whole group.
>>>>
>>>> Second, even once that is fixed, all the active brokers still race
>>>> for the lock: after a
>>>> primary fails, its backup needs a few seconds to activate while the
>>>> brokers in the other
>>>> DCs keep polling every check-period, so the lock usually lands on a
>>>> remote broker rather
>>>> than on the backup taking over.
>>>>
>>>> I have a patch for both. The coordinators are still created in
>>>> internalStart(), since
>>>> acceptors and broker connections look them up by name, but polling
>>>> now starts in
>>>> completeActivation(). On top of that, the auto-start=false you were
>>>> planning, as an
>>>> attribute on lock-coordinator, plus
>>>> startLockCoordinator/stopLockCoordinator management
>>>> operations, so a standby DC only takes the lock over when somebody
>>>> decides so.
>>>>
>>>> Both are covered by tests that fail without the fix: a replicated
>>>> pair sharing one
>>>> coordinator, where the passive backup must stay out of the lock and
>>>> take it over once the
>>>> primary is gone, and the full start/stop cycle through management on
>>>> an acceptor bound to
>>>> an auto-start=false coordinator.
>>>>
>>>> ARTEMIS-6162: https://issues.apache.org/jira/browse/ARTEMIS-6162
>>>> ARTEMIS-6163: https://issues.apache.org/jira/browse/ARTEMIS-6163
>>>> PR (one commit per issue, the first one stands on its own):
>>>> https://github.com/apache/artemis/pull/6590
>>>>
>>>> Happy to rework anything you don't like, and to move the discussion
>>>> to the dev list if
>>>> you prefer.
>>>>
>>>> --
>>>> With best regards, Lukyanov Michael
>>>>
>>>> пн, 29 июн. 2026 г. в 06:02, Clebert Suconic <clebert.suconic@gmail.com
>>>> >:
>>>>
>>>>> My intention was to replace replication with mirro. For that reason I
>>>>> didn’t look at any requirements in The replicated journal.
>>>>>
>>>>>
>>>>> I think I have to make sure the replica won’t start the lock
>>>>> coordinator until activated.
>>>>>
>>>>>
>>>>> I don’t know if it will work or not. But since I didn’t add a test I
>>>>> don’t know.
>>>>>
>>>>> Clebert Suconic
>>>>>
>>>>>
>>>>> On Thu, Jun 25, 2026 at 5:44 AM Michael <lu...@gmail.com> wrote:
>>>>>
>>>>>> Yes, <ha-policy> <replication>.
>>>>>>
>>>>>> вт, 23 июн. 2026 г., 17:57 Clebert Suconic <clebert.suconic@gmail.com
>>>>>> >:
>>>>>>
>>>>>>> in high available setup you mean journal replication?
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 19, 2026 at 7:18 AM Michael <lu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Good day.
>>>>>>>>
>>>>>>>> We managed to set up a *mesh mirror* with synchronous replication
>>>>>>>> ha-policy with different groups of Zookeepers. The broker-connections and
>>>>>>>> acceptors stay active while the *lock-coordinator* lock is held.
>>>>>>>>
>>>>>>>> An additional question has come up: when the primary broker fails
>>>>>>>> in a high-availability setup, how can we ensure that the *lock is
>>>>>>>> acquired* specifically *by its backup* broker that has become
>>>>>>>> active? Is there a plan to add such functionality, or could this be
>>>>>>>> achieved some other way?
>>>>>>>>
>>>>>>>> чт, 28 мая 2026 г. в 19:57, M Lukyanov <lu...@gmail.com>:
>>>>>>>>
>>>>>>>>> Thank you for the clarification.
>>>>>>>>>
>>>>>>>>> Of course, I'll follow the design and implementation of these
>>>>>>>>> tasks through the dev-list. I'm ready to help if there's anything I can do.
>>>>>>>>>
>>>>>>>>> For now, I will test various configurations, including the ZK
>>>>>>>>> setup, on the latest version of Artemis.
>>>>>>>>>
>>>>>>>>> чт, 28 мая 2026 г. в 18:58, Clebert Suconic <
>>>>>>>>> clebert.suconic@gmail.com>:
>>>>>>>>>
>>>>>>>>>> 1. if you require synchronous.. there are a few improvements that
>>>>>>>>>> I will make on it in the next few months. Right there's no way to make a
>>>>>>>>>> mirror to go async if the backup fails. I also want to have a N Copy of the
>>>>>>>>>> messages and some transactional control... I'm still on design for this..
>>>>>>>>>> but perhaps we could keep in touch for when I'm designing this? I would do
>>>>>>>>>> it through the dev-list
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2. you can combine classicHA and Mesh Mirror.. shoulddn't be an
>>>>>>>>>> issue.
>>>>>>>>>>
>>>>>>>>>> 2.1. I'm not sure about the setup on Zookeeper. perhaps 2.2 would
>>>>>>>>>> fix the issue for you.. you just have a set of ZK on each datacenter and
>>>>>>>>>> use a manual procedure to activate a Datacenter.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2.2. There's one small change i'm going to make soon. have a way
>>>>>>>>>> to manually start a LockManager. Usually DCs are disconnected... and
>>>>>>>>>> require human intervention.
>>>>>>>>>> I plan to make the lockmanager auto-start=false
>>>>>>>>>>
>>>>>>>>>> On Thu, May 28, 2026 at 6:52 AM M Lukyanov <lu...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thank you for the information.
>>>>>>>>>>>
>>>>>>>>>>> I'd like to clarify a few points regarding your answer.
>>>>>>>>>>>
>>>>>>>>>>> *1. Replacing classic HA with mirroring*
>>>>>>>>>>>
>>>>>>>>>>> If I drop classic active/passive HA, am I correct that the
>>>>>>>>>>> replacement would be synchronous mirroring between two active nodes? That
>>>>>>>>>>> is, I could keep the cluster for load and client distribution, and replace
>>>>>>>>>>> the Artemis instance that used to be the backup of a primary with a node
>>>>>>>>>>> that does synchronous mirroring but has client connections blocked via the
>>>>>>>>>>> lock manager — with the former backup becoming an additional mirroring
>>>>>>>>>>> target.
>>>>>>>>>>>
>>>>>>>>>>> This scheme raises several questions:
>>>>>>>>>>>
>>>>>>>>>>> 1.1 Can this active node (the former backup) participate in the
>>>>>>>>>>> same cluster as the primary? Are there any pitfalls here?
>>>>>>>>>>>
>>>>>>>>>>> 1.2 Does this scheme require additional settings analogous to
>>>>>>>>>>> classic HA so that specifically the primary in DataCenter1 acquires the
>>>>>>>>>>> lock preferentially and reclaims it after recovery — something like `
>>>>>>>>>>> *allow-failback*` and `*failover-on-shutdown*`?
>>>>>>>>>>>
>>>>>>>>>>> 1.3 Mirroring does not forward messages that already existed in
>>>>>>>>>>> the broker at the time mirroring was configured ("Previously existing
>>>>>>>>>>> messages will not be forwarded"). In that case, how do I correctly
>>>>>>>>>>> synchronize the already-accumulated messages when migrating to this scheme?
>>>>>>>>>>>
>>>>>>>>>>> *2. Combining classic HA and Mesh Mirror*
>>>>>>>>>>>
>>>>>>>>>>> If I combine classic HA and Mesh Mirror, is my understanding of
>>>>>>>>>>> the ZooKeeper layout correct? For classic HA, this is an ensemble of 3
>>>>>>>>>>> ZooKeeper nodes inside DataCenter1 (DC1). Additionally, for Mesh Mirror of
>>>>>>>>>>> the artemis1 group, I would need a separate ensemble of 3 ZooKeeper nodes
>>>>>>>>>>> spread across data centers: one each on jvm1 in DC1, DC2, and DC3 — 3 nodes
>>>>>>>>>>> to avoid split-brain.
>>>>>>>>>>>
>>>>>>>>>>> 2.1 Is this understanding correct? Can the number of ZooKeeper
>>>>>>>>>>> nodes be reduced while still protecting against split-brain, or is it
>>>>>>>>>>> needed to combine all ZooKeeper nodes into a single shared ensemble?
>>>>>>>>>>>
>>>>>>>>>>> 2.2 Same question as 1.2: how can I make artemis1 in DC1 the
>>>>>>>>>>> preferred node so that it acquires the lock first while it is running? Or
>>>>>>>>>>> is the only option to manually shut down the other nodes so that only one
>>>>>>>>>>> remains?
>>>>>>>>>>>
>>>>>>>>>>> ср, 27 мая 2026 г. в 19:36, Clebert Suconic <
>>>>>>>>>>> clebert.suconic@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> You probably seen this, there's an example here for mesh
>>>>>>>>>>>> mirror: (it's good to point it out if someone else finds this thread anyway)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/apache/artemis-examples/tree/main/examples/features/broker-connection/ha-with-mesh-mirror
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 27, 2026 at 12:34 PM Clebert Suconic <
>>>>>>>>>>>> clebert.suconic@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> When I wrote the MeshMirror example, I was trying to get rid
>>>>>>>>>>>>> of replication and clustering... but it should be compatible...
>>>>>>>>>>>>>
>>>>>>>>>>>>> But yeah.. the lock manager is the solution for this case.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You don't need a separate Zookeeper instance for any of
>>>>>>>>>>>>> this... It just needs a ZK entry point where it can connect to.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I had in the past fixed issues with dual mirroring and
>>>>>>>>>>>>> replication, and it should work fine with meshMirror. But it's always best
>>>>>>>>>>>>> to test it, right?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you could only use Mirror on your solution, it would be
>>>>>>>>>>>>> simpler though.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, May 27, 2026 at 5:48 AM Mikhail Lukyanov <
>>>>>>>>>>>>> lukyanovmiv@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello everyone.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would like to ask for advice and recommendations regarding
>>>>>>>>>>>>>> a high-availability cluster with mirroring to another data centers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Currently we have the following setup: in each data center
>>>>>>>>>>>>>> there are 3 machines/JVMs, each running 2 Artemis instances — a primary of
>>>>>>>>>>>>>> one group and a backup of another — along with ZooKeeper. In total there
>>>>>>>>>>>>>> are 6 Artemis instances, forming 3 primary/backup pairs managed by
>>>>>>>>>>>>>> CuratorManager within one data center. Each Artemis instance mirrors data
>>>>>>>>>>>>>> to the other data center, so that if one data center goes down, we can
>>>>>>>>>>>>>> bring up the full cluster and continue processing in the other data center.
>>>>>>>>>>>>>> Mirroring is configured per pair failover: if the primary fails, we send
>>>>>>>>>>>>>> data to its backup. In this scheme, all broker connections on the standby
>>>>>>>>>>>>>> data center are disabled. The primary and backup configuration xml in
>>>>>>>>>>>>>> attachment to the letter.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>>>> Now there is a requirement to mirror data to a third,
>>>>>>>>>>>>>> additional data center. This means each active Artemis instance in the
>>>>>>>>>>>>>> first data center must mirror data to 2 standby data centers. However, this
>>>>>>>>>>>>>> introduces significant new complications. For example, Artemis1 in standby
>>>>>>>>>>>>>> DC2 has broker connections both to Artemis1 in active DC1 and to Artemis1
>>>>>>>>>>>>>> in standby DC3. As a result, events received via mirroring from
>>>>>>>>>>>>>> Artemis1.DC1 are then attempted to be re-mirrored from Artemis1.DC2 to
>>>>>>>>>>>>>> Artemis1.DC3, and they accumulate in
>>>>>>>>>>>>>> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3` — even though
>>>>>>>>>>>>>> Artemis1.DC1 is already forwarding those events directly to Artemis1.DC3.
>>>>>>>>>>>>>> Mirroring from Artemis1.DC2 to Artemis1.DC3 is disabled, yet messages
>>>>>>>>>>>>>> continue to accumulate in
>>>>>>>>>>>>>> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My understanding is that the solution to this problem is Mesh
>>>>>>>>>>>>>> Mirror, as described in
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/ARTEMIS-5925.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This raises the question: are Cluster HA and Mesh Mirror
>>>>>>>>>>>>>> compatible with each other, or are they mutually exclusive mechanisms?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It also follows that ZooKeeper would need to be deployed
>>>>>>>>>>>>>> separately for HA and for Mesh Mirror — how can this be configured and
>>>>>>>>>>>>>> managed?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> By what mechanisms can I actually achieve the desired
>>>>>>>>>>>>>> behavior?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> users-help@artemis.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Clebert Suconic
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Clebert Suconic
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Clebert Suconic
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Clebert Suconic
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> *With best regards, Lukyanov Mikhail*
>>>> *Tel: **+7-909-69-71-547*
>>>>
>>>
>>
>> --
>> *With best regards, Lukyanov Mikhail*
>> *Tel: **+7-909-69-71-547*
>>
>
>
> --
> Clebert Suconic
>
--
*With best regards, Lukyanov Mikhail*
*Tel: **+7-909-69-71-547*
Re: Artemis Cluster with HA and with Mesh Mirror
Posted by Clebert Suconic <cl...@gmail.com>.
I have pushed -f directly into your branch .. (it's a github feature)..
please make sure you rebase against your branch if you do anything else on
it.
Also: I added ./artemis lock stop <name> and ./artemis lock start <name>
I also sent a Pull Request to the web console:
https://github.com/apache/artemis-console/pull/217
(I don't know if that's the right way to do it on the web console)
On Thu, Jul 23, 2026 at 5:05 AM Michael <lu...@gmail.com> wrote:
> Thanks, and enjoy your vacation.
>
> I'll be around if anything needs reworking.
>
> --
> With best regards, Lukyanov Michael
>
> ср, 22 июл. 2026 г. в 16:29, Clebert Suconic <cl...@gmail.com>:
>
>> I had a brief look. Thank you so much for these.
>>
>> I’m in vacation this week. I will have a proper look on Monday. And I
>> will merge it.
>>
>> Clebert Suconic
>>
>>
>> On Tue, Jul 21, 2026 at 2:46 AM Michael <lu...@gmail.com> wrote:
>>
>>> Hello Clebert,
>>>
>>> Following up on your last message: you were right that the replica
>>> must not start the
>>> lock coordinator until it is activated, and I went ahead and wrote the
>>> tests for it.
>>>
>>> Two things turned out to be broken in our setup (replication ha-policy
>>> inside each DC,
>>> mesh mirror between DCs, acceptors and broker connections bound to a
>>> lock coordinator).
>>>
>>> First, ActiveMQServerImpl.internalStart() starts every configured lock
>>> coordinator right
>>> away, whatever the HA role is, so a passive backup contends for the
>>> lock along with the
>>> active primary. If the backup wins it, it holds the lock while none of
>>> its acceptors or
>>> broker connections exist yet, so the elements bound to that
>>> coordinator are down on the
>>> whole group.
>>>
>>> Second, even once that is fixed, all the active brokers still race for
>>> the lock: after a
>>> primary fails, its backup needs a few seconds to activate while the
>>> brokers in the other
>>> DCs keep polling every check-period, so the lock usually lands on a
>>> remote broker rather
>>> than on the backup taking over.
>>>
>>> I have a patch for both. The coordinators are still created in
>>> internalStart(), since
>>> acceptors and broker connections look them up by name, but polling now
>>> starts in
>>> completeActivation(). On top of that, the auto-start=false you were
>>> planning, as an
>>> attribute on lock-coordinator, plus
>>> startLockCoordinator/stopLockCoordinator management
>>> operations, so a standby DC only takes the lock over when somebody
>>> decides so.
>>>
>>> Both are covered by tests that fail without the fix: a replicated pair
>>> sharing one
>>> coordinator, where the passive backup must stay out of the lock and
>>> take it over once the
>>> primary is gone, and the full start/stop cycle through management on
>>> an acceptor bound to
>>> an auto-start=false coordinator.
>>>
>>> ARTEMIS-6162: https://issues.apache.org/jira/browse/ARTEMIS-6162
>>> ARTEMIS-6163: https://issues.apache.org/jira/browse/ARTEMIS-6163
>>> PR (one commit per issue, the first one stands on its own):
>>> https://github.com/apache/artemis/pull/6590
>>>
>>> Happy to rework anything you don't like, and to move the discussion to
>>> the dev list if
>>> you prefer.
>>>
>>> --
>>> With best regards, Lukyanov Michael
>>>
>>> пн, 29 июн. 2026 г. в 06:02, Clebert Suconic <clebert.suconic@gmail.com
>>> >:
>>>
>>>> My intention was to replace replication with mirro. For that reason I
>>>> didn’t look at any requirements in The replicated journal.
>>>>
>>>>
>>>> I think I have to make sure the replica won’t start the lock
>>>> coordinator until activated.
>>>>
>>>>
>>>> I don’t know if it will work or not. But since I didn’t add a test I
>>>> don’t know.
>>>>
>>>> Clebert Suconic
>>>>
>>>>
>>>> On Thu, Jun 25, 2026 at 5:44 AM Michael <lu...@gmail.com> wrote:
>>>>
>>>>> Yes, <ha-policy> <replication>.
>>>>>
>>>>> вт, 23 июн. 2026 г., 17:57 Clebert Suconic <clebert.suconic@gmail.com
>>>>> >:
>>>>>
>>>>>> in high available setup you mean journal replication?
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 19, 2026 at 7:18 AM Michael <lu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Good day.
>>>>>>>
>>>>>>> We managed to set up a *mesh mirror* with synchronous replication
>>>>>>> ha-policy with different groups of Zookeepers. The broker-connections and
>>>>>>> acceptors stay active while the *lock-coordinator* lock is held.
>>>>>>>
>>>>>>> An additional question has come up: when the primary broker fails in
>>>>>>> a high-availability setup, how can we ensure that the *lock is
>>>>>>> acquired* specifically *by its backup* broker that has become
>>>>>>> active? Is there a plan to add such functionality, or could this be
>>>>>>> achieved some other way?
>>>>>>>
>>>>>>> чт, 28 мая 2026 г. в 19:57, M Lukyanov <lu...@gmail.com>:
>>>>>>>
>>>>>>>> Thank you for the clarification.
>>>>>>>>
>>>>>>>> Of course, I'll follow the design and implementation of these tasks
>>>>>>>> through the dev-list. I'm ready to help if there's anything I can do.
>>>>>>>>
>>>>>>>> For now, I will test various configurations, including the ZK
>>>>>>>> setup, on the latest version of Artemis.
>>>>>>>>
>>>>>>>> чт, 28 мая 2026 г. в 18:58, Clebert Suconic <
>>>>>>>> clebert.suconic@gmail.com>:
>>>>>>>>
>>>>>>>>> 1. if you require synchronous.. there are a few improvements that
>>>>>>>>> I will make on it in the next few months. Right there's no way to make a
>>>>>>>>> mirror to go async if the backup fails. I also want to have a N Copy of the
>>>>>>>>> messages and some transactional control... I'm still on design for this..
>>>>>>>>> but perhaps we could keep in touch for when I'm designing this? I would do
>>>>>>>>> it through the dev-list
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2. you can combine classicHA and Mesh Mirror.. shoulddn't be an
>>>>>>>>> issue.
>>>>>>>>>
>>>>>>>>> 2.1. I'm not sure about the setup on Zookeeper. perhaps 2.2 would
>>>>>>>>> fix the issue for you.. you just have a set of ZK on each datacenter and
>>>>>>>>> use a manual procedure to activate a Datacenter.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2.2. There's one small change i'm going to make soon. have a way
>>>>>>>>> to manually start a LockManager. Usually DCs are disconnected... and
>>>>>>>>> require human intervention.
>>>>>>>>> I plan to make the lockmanager auto-start=false
>>>>>>>>>
>>>>>>>>> On Thu, May 28, 2026 at 6:52 AM M Lukyanov <lu...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thank you for the information.
>>>>>>>>>>
>>>>>>>>>> I'd like to clarify a few points regarding your answer.
>>>>>>>>>>
>>>>>>>>>> *1. Replacing classic HA with mirroring*
>>>>>>>>>>
>>>>>>>>>> If I drop classic active/passive HA, am I correct that the
>>>>>>>>>> replacement would be synchronous mirroring between two active nodes? That
>>>>>>>>>> is, I could keep the cluster for load and client distribution, and replace
>>>>>>>>>> the Artemis instance that used to be the backup of a primary with a node
>>>>>>>>>> that does synchronous mirroring but has client connections blocked via the
>>>>>>>>>> lock manager — with the former backup becoming an additional mirroring
>>>>>>>>>> target.
>>>>>>>>>>
>>>>>>>>>> This scheme raises several questions:
>>>>>>>>>>
>>>>>>>>>> 1.1 Can this active node (the former backup) participate in the
>>>>>>>>>> same cluster as the primary? Are there any pitfalls here?
>>>>>>>>>>
>>>>>>>>>> 1.2 Does this scheme require additional settings analogous to
>>>>>>>>>> classic HA so that specifically the primary in DataCenter1 acquires the
>>>>>>>>>> lock preferentially and reclaims it after recovery — something like `
>>>>>>>>>> *allow-failback*` and `*failover-on-shutdown*`?
>>>>>>>>>>
>>>>>>>>>> 1.3 Mirroring does not forward messages that already existed in
>>>>>>>>>> the broker at the time mirroring was configured ("Previously existing
>>>>>>>>>> messages will not be forwarded"). In that case, how do I correctly
>>>>>>>>>> synchronize the already-accumulated messages when migrating to this scheme?
>>>>>>>>>>
>>>>>>>>>> *2. Combining classic HA and Mesh Mirror*
>>>>>>>>>>
>>>>>>>>>> If I combine classic HA and Mesh Mirror, is my understanding of
>>>>>>>>>> the ZooKeeper layout correct? For classic HA, this is an ensemble of 3
>>>>>>>>>> ZooKeeper nodes inside DataCenter1 (DC1). Additionally, for Mesh Mirror of
>>>>>>>>>> the artemis1 group, I would need a separate ensemble of 3 ZooKeeper nodes
>>>>>>>>>> spread across data centers: one each on jvm1 in DC1, DC2, and DC3 — 3 nodes
>>>>>>>>>> to avoid split-brain.
>>>>>>>>>>
>>>>>>>>>> 2.1 Is this understanding correct? Can the number of ZooKeeper
>>>>>>>>>> nodes be reduced while still protecting against split-brain, or is it
>>>>>>>>>> needed to combine all ZooKeeper nodes into a single shared ensemble?
>>>>>>>>>>
>>>>>>>>>> 2.2 Same question as 1.2: how can I make artemis1 in DC1 the
>>>>>>>>>> preferred node so that it acquires the lock first while it is running? Or
>>>>>>>>>> is the only option to manually shut down the other nodes so that only one
>>>>>>>>>> remains?
>>>>>>>>>>
>>>>>>>>>> ср, 27 мая 2026 г. в 19:36, Clebert Suconic <
>>>>>>>>>> clebert.suconic@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> You probably seen this, there's an example here for mesh mirror:
>>>>>>>>>>> (it's good to point it out if someone else finds this thread anyway)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/apache/artemis-examples/tree/main/examples/features/broker-connection/ha-with-mesh-mirror
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 27, 2026 at 12:34 PM Clebert Suconic <
>>>>>>>>>>> clebert.suconic@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> When I wrote the MeshMirror example, I was trying to get rid of
>>>>>>>>>>>> replication and clustering... but it should be compatible...
>>>>>>>>>>>>
>>>>>>>>>>>> But yeah.. the lock manager is the solution for this case.
>>>>>>>>>>>>
>>>>>>>>>>>> You don't need a separate Zookeeper instance for any of this...
>>>>>>>>>>>> It just needs a ZK entry point where it can connect to.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I had in the past fixed issues with dual mirroring and
>>>>>>>>>>>> replication, and it should work fine with meshMirror. But it's always best
>>>>>>>>>>>> to test it, right?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If you could only use Mirror on your solution, it would be
>>>>>>>>>>>> simpler though.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 27, 2026 at 5:48 AM Mikhail Lukyanov <
>>>>>>>>>>>> lukyanovmiv@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello everyone.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would like to ask for advice and recommendations regarding a
>>>>>>>>>>>>> high-availability cluster with mirroring to another data centers.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Currently we have the following setup: in each data center
>>>>>>>>>>>>> there are 3 machines/JVMs, each running 2 Artemis instances — a primary of
>>>>>>>>>>>>> one group and a backup of another — along with ZooKeeper. In total there
>>>>>>>>>>>>> are 6 Artemis instances, forming 3 primary/backup pairs managed by
>>>>>>>>>>>>> CuratorManager within one data center. Each Artemis instance mirrors data
>>>>>>>>>>>>> to the other data center, so that if one data center goes down, we can
>>>>>>>>>>>>> bring up the full cluster and continue processing in the other data center.
>>>>>>>>>>>>> Mirroring is configured per pair failover: if the primary fails, we send
>>>>>>>>>>>>> data to its backup. In this scheme, all broker connections on the standby
>>>>>>>>>>>>> data center are disabled. The primary and backup configuration xml in
>>>>>>>>>>>>> attachment to the letter.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>>> Now there is a requirement to mirror data to a third,
>>>>>>>>>>>>> additional data center. This means each active Artemis instance in the
>>>>>>>>>>>>> first data center must mirror data to 2 standby data centers. However, this
>>>>>>>>>>>>> introduces significant new complications. For example, Artemis1 in standby
>>>>>>>>>>>>> DC2 has broker connections both to Artemis1 in active DC1 and to Artemis1
>>>>>>>>>>>>> in standby DC3. As a result, events received via mirroring from
>>>>>>>>>>>>> Artemis1.DC1 are then attempted to be re-mirrored from Artemis1.DC2 to
>>>>>>>>>>>>> Artemis1.DC3, and they accumulate in
>>>>>>>>>>>>> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3` — even though
>>>>>>>>>>>>> Artemis1.DC1 is already forwarding those events directly to Artemis1.DC3.
>>>>>>>>>>>>> Mirroring from Artemis1.DC2 to Artemis1.DC3 is disabled, yet messages
>>>>>>>>>>>>> continue to accumulate in
>>>>>>>>>>>>> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My understanding is that the solution to this problem is Mesh
>>>>>>>>>>>>> Mirror, as described in
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/ARTEMIS-5925.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This raises the question: are Cluster HA and Mesh Mirror
>>>>>>>>>>>>> compatible with each other, or are they mutually exclusive mechanisms?
>>>>>>>>>>>>>
>>>>>>>>>>>>> It also follows that ZooKeeper would need to be deployed
>>>>>>>>>>>>> separately for HA and for Mesh Mirror — how can this be configured and
>>>>>>>>>>>>> managed?
>>>>>>>>>>>>>
>>>>>>>>>>>>> By what mechanisms can I actually achieve the desired behavior?
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: users-help@artemis.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Clebert Suconic
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Clebert Suconic
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Clebert Suconic
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Clebert Suconic
>>>>>>
>>>>>
>>>
>>> --
>>> *With best regards, Lukyanov Mikhail*
>>> *Tel: **+7-909-69-71-547*
>>>
>>
>
> --
> *With best regards, Lukyanov Mikhail*
> *Tel: **+7-909-69-71-547*
>
--
Clebert Suconic
Re: Artemis Cluster with HA and with Mesh Mirror
Posted by Michael <lu...@gmail.com>.
Thanks, and enjoy your vacation.
I'll be around if anything needs reworking.
--
With best regards, Lukyanov Michael
ср, 22 июл. 2026 г. в 16:29, Clebert Suconic <cl...@gmail.com>:
> I had a brief look. Thank you so much for these.
>
> I’m in vacation this week. I will have a proper look on Monday. And I
> will merge it.
>
> Clebert Suconic
>
>
> On Tue, Jul 21, 2026 at 2:46 AM Michael <lu...@gmail.com> wrote:
>
>> Hello Clebert,
>>
>> Following up on your last message: you were right that the replica must
>> not start the
>> lock coordinator until it is activated, and I went ahead and wrote the
>> tests for it.
>>
>> Two things turned out to be broken in our setup (replication ha-policy
>> inside each DC,
>> mesh mirror between DCs, acceptors and broker connections bound to a
>> lock coordinator).
>>
>> First, ActiveMQServerImpl.internalStart() starts every configured lock
>> coordinator right
>> away, whatever the HA role is, so a passive backup contends for the
>> lock along with the
>> active primary. If the backup wins it, it holds the lock while none of
>> its acceptors or
>> broker connections exist yet, so the elements bound to that coordinator
>> are down on the
>> whole group.
>>
>> Second, even once that is fixed, all the active brokers still race for
>> the lock: after a
>> primary fails, its backup needs a few seconds to activate while the
>> brokers in the other
>> DCs keep polling every check-period, so the lock usually lands on a
>> remote broker rather
>> than on the backup taking over.
>>
>> I have a patch for both. The coordinators are still created in
>> internalStart(), since
>> acceptors and broker connections look them up by name, but polling now
>> starts in
>> completeActivation(). On top of that, the auto-start=false you were
>> planning, as an
>> attribute on lock-coordinator, plus
>> startLockCoordinator/stopLockCoordinator management
>> operations, so a standby DC only takes the lock over when somebody
>> decides so.
>>
>> Both are covered by tests that fail without the fix: a replicated pair
>> sharing one
>> coordinator, where the passive backup must stay out of the lock and
>> take it over once the
>> primary is gone, and the full start/stop cycle through management on an
>> acceptor bound to
>> an auto-start=false coordinator.
>>
>> ARTEMIS-6162: https://issues.apache.org/jira/browse/ARTEMIS-6162
>> ARTEMIS-6163: https://issues.apache.org/jira/browse/ARTEMIS-6163
>> PR (one commit per issue, the first one stands on its own):
>> https://github.com/apache/artemis/pull/6590
>>
>> Happy to rework anything you don't like, and to move the discussion to
>> the dev list if
>> you prefer.
>>
>> --
>> With best regards, Lukyanov Michael
>>
>> пн, 29 июн. 2026 г. в 06:02, Clebert Suconic <cl...@gmail.com>:
>>
>>> My intention was to replace replication with mirro. For that reason I
>>> didn’t look at any requirements in The replicated journal.
>>>
>>>
>>> I think I have to make sure the replica won’t start the lock coordinator
>>> until activated.
>>>
>>>
>>> I don’t know if it will work or not. But since I didn’t add a test I
>>> don’t know.
>>>
>>> Clebert Suconic
>>>
>>>
>>> On Thu, Jun 25, 2026 at 5:44 AM Michael <lu...@gmail.com> wrote:
>>>
>>>> Yes, <ha-policy> <replication>.
>>>>
>>>> вт, 23 июн. 2026 г., 17:57 Clebert Suconic <cl...@gmail.com>:
>>>>
>>>>> in high available setup you mean journal replication?
>>>>>
>>>>>
>>>>> On Fri, Jun 19, 2026 at 7:18 AM Michael <lu...@gmail.com> wrote:
>>>>>
>>>>>> Good day.
>>>>>>
>>>>>> We managed to set up a *mesh mirror* with synchronous replication
>>>>>> ha-policy with different groups of Zookeepers. The broker-connections and
>>>>>> acceptors stay active while the *lock-coordinator* lock is held.
>>>>>>
>>>>>> An additional question has come up: when the primary broker fails in
>>>>>> a high-availability setup, how can we ensure that the *lock is
>>>>>> acquired* specifically *by its backup* broker that has become
>>>>>> active? Is there a plan to add such functionality, or could this be
>>>>>> achieved some other way?
>>>>>>
>>>>>> чт, 28 мая 2026 г. в 19:57, M Lukyanov <lu...@gmail.com>:
>>>>>>
>>>>>>> Thank you for the clarification.
>>>>>>>
>>>>>>> Of course, I'll follow the design and implementation of these tasks
>>>>>>> through the dev-list. I'm ready to help if there's anything I can do.
>>>>>>>
>>>>>>> For now, I will test various configurations, including the ZK setup,
>>>>>>> on the latest version of Artemis.
>>>>>>>
>>>>>>> чт, 28 мая 2026 г. в 18:58, Clebert Suconic <
>>>>>>> clebert.suconic@gmail.com>:
>>>>>>>
>>>>>>>> 1. if you require synchronous.. there are a few improvements that I
>>>>>>>> will make on it in the next few months. Right there's no way to make a
>>>>>>>> mirror to go async if the backup fails. I also want to have a N Copy of the
>>>>>>>> messages and some transactional control... I'm still on design for this..
>>>>>>>> but perhaps we could keep in touch for when I'm designing this? I would do
>>>>>>>> it through the dev-list
>>>>>>>>
>>>>>>>>
>>>>>>>> 2. you can combine classicHA and Mesh Mirror.. shoulddn't be an
>>>>>>>> issue.
>>>>>>>>
>>>>>>>> 2.1. I'm not sure about the setup on Zookeeper. perhaps 2.2 would
>>>>>>>> fix the issue for you.. you just have a set of ZK on each datacenter and
>>>>>>>> use a manual procedure to activate a Datacenter.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2.2. There's one small change i'm going to make soon. have a way to
>>>>>>>> manually start a LockManager. Usually DCs are disconnected... and require
>>>>>>>> human intervention.
>>>>>>>> I plan to make the lockmanager auto-start=false
>>>>>>>>
>>>>>>>> On Thu, May 28, 2026 at 6:52 AM M Lukyanov <lu...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thank you for the information.
>>>>>>>>>
>>>>>>>>> I'd like to clarify a few points regarding your answer.
>>>>>>>>>
>>>>>>>>> *1. Replacing classic HA with mirroring*
>>>>>>>>>
>>>>>>>>> If I drop classic active/passive HA, am I correct that the
>>>>>>>>> replacement would be synchronous mirroring between two active nodes? That
>>>>>>>>> is, I could keep the cluster for load and client distribution, and replace
>>>>>>>>> the Artemis instance that used to be the backup of a primary with a node
>>>>>>>>> that does synchronous mirroring but has client connections blocked via the
>>>>>>>>> lock manager — with the former backup becoming an additional mirroring
>>>>>>>>> target.
>>>>>>>>>
>>>>>>>>> This scheme raises several questions:
>>>>>>>>>
>>>>>>>>> 1.1 Can this active node (the former backup) participate in the
>>>>>>>>> same cluster as the primary? Are there any pitfalls here?
>>>>>>>>>
>>>>>>>>> 1.2 Does this scheme require additional settings analogous to
>>>>>>>>> classic HA so that specifically the primary in DataCenter1 acquires the
>>>>>>>>> lock preferentially and reclaims it after recovery — something like `
>>>>>>>>> *allow-failback*` and `*failover-on-shutdown*`?
>>>>>>>>>
>>>>>>>>> 1.3 Mirroring does not forward messages that already existed in
>>>>>>>>> the broker at the time mirroring was configured ("Previously existing
>>>>>>>>> messages will not be forwarded"). In that case, how do I correctly
>>>>>>>>> synchronize the already-accumulated messages when migrating to this scheme?
>>>>>>>>>
>>>>>>>>> *2. Combining classic HA and Mesh Mirror*
>>>>>>>>>
>>>>>>>>> If I combine classic HA and Mesh Mirror, is my understanding of
>>>>>>>>> the ZooKeeper layout correct? For classic HA, this is an ensemble of 3
>>>>>>>>> ZooKeeper nodes inside DataCenter1 (DC1). Additionally, for Mesh Mirror of
>>>>>>>>> the artemis1 group, I would need a separate ensemble of 3 ZooKeeper nodes
>>>>>>>>> spread across data centers: one each on jvm1 in DC1, DC2, and DC3 — 3 nodes
>>>>>>>>> to avoid split-brain.
>>>>>>>>>
>>>>>>>>> 2.1 Is this understanding correct? Can the number of ZooKeeper
>>>>>>>>> nodes be reduced while still protecting against split-brain, or is it
>>>>>>>>> needed to combine all ZooKeeper nodes into a single shared ensemble?
>>>>>>>>>
>>>>>>>>> 2.2 Same question as 1.2: how can I make artemis1 in DC1 the
>>>>>>>>> preferred node so that it acquires the lock first while it is running? Or
>>>>>>>>> is the only option to manually shut down the other nodes so that only one
>>>>>>>>> remains?
>>>>>>>>>
>>>>>>>>> ср, 27 мая 2026 г. в 19:36, Clebert Suconic <
>>>>>>>>> clebert.suconic@gmail.com>:
>>>>>>>>>
>>>>>>>>>> You probably seen this, there's an example here for mesh mirror:
>>>>>>>>>> (it's good to point it out if someone else finds this thread anyway)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/artemis-examples/tree/main/examples/features/broker-connection/ha-with-mesh-mirror
>>>>>>>>>>
>>>>>>>>>> On Wed, May 27, 2026 at 12:34 PM Clebert Suconic <
>>>>>>>>>> clebert.suconic@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> When I wrote the MeshMirror example, I was trying to get rid of
>>>>>>>>>>> replication and clustering... but it should be compatible...
>>>>>>>>>>>
>>>>>>>>>>> But yeah.. the lock manager is the solution for this case.
>>>>>>>>>>>
>>>>>>>>>>> You don't need a separate Zookeeper instance for any of this...
>>>>>>>>>>> It just needs a ZK entry point where it can connect to.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I had in the past fixed issues with dual mirroring and
>>>>>>>>>>> replication, and it should work fine with meshMirror. But it's always best
>>>>>>>>>>> to test it, right?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If you could only use Mirror on your solution, it would be
>>>>>>>>>>> simpler though.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 27, 2026 at 5:48 AM Mikhail Lukyanov <
>>>>>>>>>>> lukyanovmiv@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello everyone.
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to ask for advice and recommendations regarding a
>>>>>>>>>>>> high-availability cluster with mirroring to another data centers.
>>>>>>>>>>>>
>>>>>>>>>>>> Currently we have the following setup: in each data center
>>>>>>>>>>>> there are 3 machines/JVMs, each running 2 Artemis instances — a primary of
>>>>>>>>>>>> one group and a backup of another — along with ZooKeeper. In total there
>>>>>>>>>>>> are 6 Artemis instances, forming 3 primary/backup pairs managed by
>>>>>>>>>>>> CuratorManager within one data center. Each Artemis instance mirrors data
>>>>>>>>>>>> to the other data center, so that if one data center goes down, we can
>>>>>>>>>>>> bring up the full cluster and continue processing in the other data center.
>>>>>>>>>>>> Mirroring is configured per pair failover: if the primary fails, we send
>>>>>>>>>>>> data to its backup. In this scheme, all broker connections on the standby
>>>>>>>>>>>> data center are disabled. The primary and backup configuration xml in
>>>>>>>>>>>> attachment to the letter.
>>>>>>>>>>>>
>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>> Now there is a requirement to mirror data to a third,
>>>>>>>>>>>> additional data center. This means each active Artemis instance in the
>>>>>>>>>>>> first data center must mirror data to 2 standby data centers. However, this
>>>>>>>>>>>> introduces significant new complications. For example, Artemis1 in standby
>>>>>>>>>>>> DC2 has broker connections both to Artemis1 in active DC1 and to Artemis1
>>>>>>>>>>>> in standby DC3. As a result, events received via mirroring from
>>>>>>>>>>>> Artemis1.DC1 are then attempted to be re-mirrored from Artemis1.DC2 to
>>>>>>>>>>>> Artemis1.DC3, and they accumulate in
>>>>>>>>>>>> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3` — even though
>>>>>>>>>>>> Artemis1.DC1 is already forwarding those events directly to Artemis1.DC3.
>>>>>>>>>>>> Mirroring from Artemis1.DC2 to Artemis1.DC3 is disabled, yet messages
>>>>>>>>>>>> continue to accumulate in
>>>>>>>>>>>> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
>>>>>>>>>>>>
>>>>>>>>>>>> My understanding is that the solution to this problem is Mesh
>>>>>>>>>>>> Mirror, as described in
>>>>>>>>>>>> https://issues.apache.org/jira/browse/ARTEMIS-5925.
>>>>>>>>>>>>
>>>>>>>>>>>> This raises the question: are Cluster HA and Mesh Mirror
>>>>>>>>>>>> compatible with each other, or are they mutually exclusive mechanisms?
>>>>>>>>>>>>
>>>>>>>>>>>> It also follows that ZooKeeper would need to be deployed
>>>>>>>>>>>> separately for HA and for Mesh Mirror — how can this be configured and
>>>>>>>>>>>> managed?
>>>>>>>>>>>>
>>>>>>>>>>>> By what mechanisms can I actually achieve the desired behavior?
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
>>>>>>>>>>>> For additional commands, e-mail: users-help@artemis.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Clebert Suconic
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Clebert Suconic
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Clebert Suconic
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Clebert Suconic
>>>>>
>>>>
>>
>> --
>> *With best regards, Lukyanov Mikhail*
>> *Tel: **+7-909-69-71-547*
>>
>
--
*With best regards, Lukyanov Mikhail*
*Tel: **+7-909-69-71-547*
Re: Artemis Cluster with HA and with Mesh Mirror
Posted by Clebert Suconic <cl...@gmail.com>.
I had a brief look. Thank you so much for these.
I’m in vacation this week. I will have a proper look on Monday. And I
will merge it.
Clebert Suconic
On Tue, Jul 21, 2026 at 2:46 AM Michael <lu...@gmail.com> wrote:
> Hello Clebert,
>
> Following up on your last message: you were right that the replica must
> not start the
> lock coordinator until it is activated, and I went ahead and wrote the
> tests for it.
>
> Two things turned out to be broken in our setup (replication ha-policy
> inside each DC,
> mesh mirror between DCs, acceptors and broker connections bound to a
> lock coordinator).
>
> First, ActiveMQServerImpl.internalStart() starts every configured lock
> coordinator right
> away, whatever the HA role is, so a passive backup contends for the lock
> along with the
> active primary. If the backup wins it, it holds the lock while none of
> its acceptors or
> broker connections exist yet, so the elements bound to that coordinator
> are down on the
> whole group.
>
> Second, even once that is fixed, all the active brokers still race for
> the lock: after a
> primary fails, its backup needs a few seconds to activate while the
> brokers in the other
> DCs keep polling every check-period, so the lock usually lands on a
> remote broker rather
> than on the backup taking over.
>
> I have a patch for both. The coordinators are still created in
> internalStart(), since
> acceptors and broker connections look them up by name, but polling now
> starts in
> completeActivation(). On top of that, the auto-start=false you were
> planning, as an
> attribute on lock-coordinator, plus
> startLockCoordinator/stopLockCoordinator management
> operations, so a standby DC only takes the lock over when somebody
> decides so.
>
> Both are covered by tests that fail without the fix: a replicated pair
> sharing one
> coordinator, where the passive backup must stay out of the lock and take
> it over once the
> primary is gone, and the full start/stop cycle through management on an
> acceptor bound to
> an auto-start=false coordinator.
>
> ARTEMIS-6162: https://issues.apache.org/jira/browse/ARTEMIS-6162
> ARTEMIS-6163: https://issues.apache.org/jira/browse/ARTEMIS-6163
> PR (one commit per issue, the first one stands on its own):
> https://github.com/apache/artemis/pull/6590
>
> Happy to rework anything you don't like, and to move the discussion to
> the dev list if
> you prefer.
>
> --
> With best regards, Lukyanov Michael
>
> пн, 29 июн. 2026 г. в 06:02, Clebert Suconic <cl...@gmail.com>:
>
>> My intention was to replace replication with mirro. For that reason I
>> didn’t look at any requirements in The replicated journal.
>>
>>
>> I think I have to make sure the replica won’t start the lock coordinator
>> until activated.
>>
>>
>> I don’t know if it will work or not. But since I didn’t add a test I
>> don’t know.
>>
>> Clebert Suconic
>>
>>
>> On Thu, Jun 25, 2026 at 5:44 AM Michael <lu...@gmail.com> wrote:
>>
>>> Yes, <ha-policy> <replication>.
>>>
>>> вт, 23 июн. 2026 г., 17:57 Clebert Suconic <cl...@gmail.com>:
>>>
>>>> in high available setup you mean journal replication?
>>>>
>>>>
>>>> On Fri, Jun 19, 2026 at 7:18 AM Michael <lu...@gmail.com> wrote:
>>>>
>>>>> Good day.
>>>>>
>>>>> We managed to set up a *mesh mirror* with synchronous replication
>>>>> ha-policy with different groups of Zookeepers. The broker-connections and
>>>>> acceptors stay active while the *lock-coordinator* lock is held.
>>>>>
>>>>> An additional question has come up: when the primary broker fails in a
>>>>> high-availability setup, how can we ensure that the *lock is acquired*
>>>>> specifically *by its backup* broker that has become active? Is there
>>>>> a plan to add such functionality, or could this be achieved some other way?
>>>>>
>>>>> чт, 28 мая 2026 г. в 19:57, M Lukyanov <lu...@gmail.com>:
>>>>>
>>>>>> Thank you for the clarification.
>>>>>>
>>>>>> Of course, I'll follow the design and implementation of these tasks
>>>>>> through the dev-list. I'm ready to help if there's anything I can do.
>>>>>>
>>>>>> For now, I will test various configurations, including the ZK setup,
>>>>>> on the latest version of Artemis.
>>>>>>
>>>>>> чт, 28 мая 2026 г. в 18:58, Clebert Suconic <
>>>>>> clebert.suconic@gmail.com>:
>>>>>>
>>>>>>> 1. if you require synchronous.. there are a few improvements that I
>>>>>>> will make on it in the next few months. Right there's no way to make a
>>>>>>> mirror to go async if the backup fails. I also want to have a N Copy of the
>>>>>>> messages and some transactional control... I'm still on design for this..
>>>>>>> but perhaps we could keep in touch for when I'm designing this? I would do
>>>>>>> it through the dev-list
>>>>>>>
>>>>>>>
>>>>>>> 2. you can combine classicHA and Mesh Mirror.. shoulddn't be an
>>>>>>> issue.
>>>>>>>
>>>>>>> 2.1. I'm not sure about the setup on Zookeeper. perhaps 2.2 would
>>>>>>> fix the issue for you.. you just have a set of ZK on each datacenter and
>>>>>>> use a manual procedure to activate a Datacenter.
>>>>>>>
>>>>>>>
>>>>>>> 2.2. There's one small change i'm going to make soon. have a way to
>>>>>>> manually start a LockManager. Usually DCs are disconnected... and require
>>>>>>> human intervention.
>>>>>>> I plan to make the lockmanager auto-start=false
>>>>>>>
>>>>>>> On Thu, May 28, 2026 at 6:52 AM M Lukyanov <lu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thank you for the information.
>>>>>>>>
>>>>>>>> I'd like to clarify a few points regarding your answer.
>>>>>>>>
>>>>>>>> *1. Replacing classic HA with mirroring*
>>>>>>>>
>>>>>>>> If I drop classic active/passive HA, am I correct that the
>>>>>>>> replacement would be synchronous mirroring between two active nodes? That
>>>>>>>> is, I could keep the cluster for load and client distribution, and replace
>>>>>>>> the Artemis instance that used to be the backup of a primary with a node
>>>>>>>> that does synchronous mirroring but has client connections blocked via the
>>>>>>>> lock manager — with the former backup becoming an additional mirroring
>>>>>>>> target.
>>>>>>>>
>>>>>>>> This scheme raises several questions:
>>>>>>>>
>>>>>>>> 1.1 Can this active node (the former backup) participate in the
>>>>>>>> same cluster as the primary? Are there any pitfalls here?
>>>>>>>>
>>>>>>>> 1.2 Does this scheme require additional settings analogous to
>>>>>>>> classic HA so that specifically the primary in DataCenter1 acquires the
>>>>>>>> lock preferentially and reclaims it after recovery — something like `
>>>>>>>> *allow-failback*` and `*failover-on-shutdown*`?
>>>>>>>>
>>>>>>>> 1.3 Mirroring does not forward messages that already existed in the
>>>>>>>> broker at the time mirroring was configured ("Previously existing messages
>>>>>>>> will not be forwarded"). In that case, how do I correctly synchronize the
>>>>>>>> already-accumulated messages when migrating to this scheme?
>>>>>>>>
>>>>>>>> *2. Combining classic HA and Mesh Mirror*
>>>>>>>>
>>>>>>>> If I combine classic HA and Mesh Mirror, is my understanding of the
>>>>>>>> ZooKeeper layout correct? For classic HA, this is an ensemble of 3
>>>>>>>> ZooKeeper nodes inside DataCenter1 (DC1). Additionally, for Mesh Mirror of
>>>>>>>> the artemis1 group, I would need a separate ensemble of 3 ZooKeeper nodes
>>>>>>>> spread across data centers: one each on jvm1 in DC1, DC2, and DC3 — 3 nodes
>>>>>>>> to avoid split-brain.
>>>>>>>>
>>>>>>>> 2.1 Is this understanding correct? Can the number of ZooKeeper
>>>>>>>> nodes be reduced while still protecting against split-brain, or is it
>>>>>>>> needed to combine all ZooKeeper nodes into a single shared ensemble?
>>>>>>>>
>>>>>>>> 2.2 Same question as 1.2: how can I make artemis1 in DC1 the
>>>>>>>> preferred node so that it acquires the lock first while it is running? Or
>>>>>>>> is the only option to manually shut down the other nodes so that only one
>>>>>>>> remains?
>>>>>>>>
>>>>>>>> ср, 27 мая 2026 г. в 19:36, Clebert Suconic <
>>>>>>>> clebert.suconic@gmail.com>:
>>>>>>>>
>>>>>>>>> You probably seen this, there's an example here for mesh mirror:
>>>>>>>>> (it's good to point it out if someone else finds this thread anyway)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/apache/artemis-examples/tree/main/examples/features/broker-connection/ha-with-mesh-mirror
>>>>>>>>>
>>>>>>>>> On Wed, May 27, 2026 at 12:34 PM Clebert Suconic <
>>>>>>>>> clebert.suconic@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> When I wrote the MeshMirror example, I was trying to get rid of
>>>>>>>>>> replication and clustering... but it should be compatible...
>>>>>>>>>>
>>>>>>>>>> But yeah.. the lock manager is the solution for this case.
>>>>>>>>>>
>>>>>>>>>> You don't need a separate Zookeeper instance for any of this...
>>>>>>>>>> It just needs a ZK entry point where it can connect to.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I had in the past fixed issues with dual mirroring and
>>>>>>>>>> replication, and it should work fine with meshMirror. But it's always best
>>>>>>>>>> to test it, right?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If you could only use Mirror on your solution, it would be
>>>>>>>>>> simpler though.
>>>>>>>>>>
>>>>>>>>>> On Wed, May 27, 2026 at 5:48 AM Mikhail Lukyanov <
>>>>>>>>>> lukyanovmiv@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello everyone.
>>>>>>>>>>>
>>>>>>>>>>> I would like to ask for advice and recommendations regarding a
>>>>>>>>>>> high-availability cluster with mirroring to another data centers.
>>>>>>>>>>>
>>>>>>>>>>> Currently we have the following setup: in each data center there
>>>>>>>>>>> are 3 machines/JVMs, each running 2 Artemis instances — a primary of one
>>>>>>>>>>> group and a backup of another — along with ZooKeeper. In total there are 6
>>>>>>>>>>> Artemis instances, forming 3 primary/backup pairs managed by CuratorManager
>>>>>>>>>>> within one data center. Each Artemis instance mirrors data to the other
>>>>>>>>>>> data center, so that if one data center goes down, we can bring up the full
>>>>>>>>>>> cluster and continue processing in the other data center. Mirroring is
>>>>>>>>>>> configured per pair failover: if the primary fails, we send data to its
>>>>>>>>>>> backup. In this scheme, all broker connections on the standby data center
>>>>>>>>>>> are disabled. The primary and backup configuration xml in attachment to the
>>>>>>>>>>> letter.
>>>>>>>>>>>
>>>>>>>>>>> [image: image.png]
>>>>>>>>>>> Now there is a requirement to mirror data to a third, additional
>>>>>>>>>>> data center. This means each active Artemis instance in the first data
>>>>>>>>>>> center must mirror data to 2 standby data centers. However, this introduces
>>>>>>>>>>> significant new complications. For example, Artemis1 in standby DC2 has
>>>>>>>>>>> broker connections both to Artemis1 in active DC1 and to Artemis1 in
>>>>>>>>>>> standby DC3. As a result, events received via mirroring from Artemis1.DC1
>>>>>>>>>>> are then attempted to be re-mirrored from Artemis1.DC2 to Artemis1.DC3, and
>>>>>>>>>>> they accumulate in `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`
>>>>>>>>>>> — even though Artemis1.DC1 is already forwarding those events directly to
>>>>>>>>>>> Artemis1.DC3. Mirroring from Artemis1.DC2 to Artemis1.DC3 is disabled, yet
>>>>>>>>>>> messages continue to accumulate in
>>>>>>>>>>> `$ACTIVEMQ_ARTEMIS_MIRROR_Artemis1.DC2_to_Artemis1.DC3`.
>>>>>>>>>>>
>>>>>>>>>>> My understanding is that the solution to this problem is Mesh
>>>>>>>>>>> Mirror, as described in
>>>>>>>>>>> https://issues.apache.org/jira/browse/ARTEMIS-5925.
>>>>>>>>>>>
>>>>>>>>>>> This raises the question: are Cluster HA and Mesh Mirror
>>>>>>>>>>> compatible with each other, or are they mutually exclusive mechanisms?
>>>>>>>>>>>
>>>>>>>>>>> It also follows that ZooKeeper would need to be deployed
>>>>>>>>>>> separately for HA and for Mesh Mirror — how can this be configured and
>>>>>>>>>>> managed?
>>>>>>>>>>>
>>>>>>>>>>> By what mechanisms can I actually achieve the desired behavior?
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
>>>>>>>>>>> For additional commands, e-mail: users-help@artemis.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Clebert Suconic
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Clebert Suconic
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Clebert Suconic
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *With best regards, Lukyanov Mikhail*
>>>>>> *Tel: **+7-909-69-71-547*
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *With best regards, Lukyanov Mikhail*
>>>>> *Tel: **+7-909-69-71-547*
>>>>>
>>>>
>>>>
>>>> --
>>>> Clebert Suconic
>>>>
>>>
>
> --
> *With best regards, Lukyanov Mikhail*
> *Tel: **+7-909-69-71-547*
>