You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Carbonneaux Yannick <ya...@skyguide.ch> on 2022/06/08 14:22:59 UTC
ActiveMQ Classic HA without SAN
Hi,
We are planning the deployment of an ActiveMQ Classic cluster and I am looking for the best way to implement HA in a VMWare environment on RHEL OS without a SAN.
It seems I'm down to Master/Slave with either something like GFS2 (so a RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that no messages are lost if we lose an ESX in our VMWare cluster.
Are there any recommendations on how to implement HA in this context? Performance is not really problematic today... but might in a near future.
Thanks for your help,
___
Yannick Carbonneaux
Re: ActiveMQ Classic HA without SAN
Posted by Matt Pavlovich <ma...@gmail.com>.
Hello Yannick-
When it comes to distributed systems and HA it generally boils down to a set of benefits and trade-offs. There is no ‘easy button’ to get perfect uptime. When you use replicated systems (filesystems, databases, messaging systems, etc) , you suffer from having to take a full-shutdown to fix ’split brain’ in the event the two nodes get out of sync— creating a generally undesirable amount of downtime to resolve.
Using well established and stable infrastructure is always wise, and leveraging ESX to do fault tolerant volumes is a strong option. I know of lots of messaging infrastructure (ActiveMQ, IBM MQ, and others) that use a lot of single broker architectures to reduce the overall amount of tech involved. Then focus energies on application performance to keep the queues as close to empty as possible vs making the messaging broker HA. Remember— if the queues are empty, there is no need to replicate anything!
Echoing what Justin said his follow-up email, if the broker can’t write to disk the safest thing for the broker to do is to error to the clients— this way clients can decide whether to bubble up the error message to the sender, or take some sort of compensating action.
Thanks,
Matt
> On Jun 10, 2022, at 1:38 AM, Carbonneaux Yannick <ya...@skyguide.ch> wrote:
>
> We were actually discussing this possibility yesterday, but wondered what would be the behavior of the broker in case the shared FS fails, and have two options there:
> - let the VM exposing the NFS fail and be restarted on another ESX host by the cluster
> - or set the VM as fault tolerant on the ESX so that it is readily available shall the primary host fail.
>
> But I'm especially wondering what would be the best practice in case of failure of the shared storage with KahaDB ? Just piles up in memory until KahaDB comes back ? Or plainly fail ? Seems to be configurable via LeaseLockerIOExceptionHandler if I understood well.
>
> Yannick
>
>
> -----Original Message-----
> From: Matt Pavlovich <ma...@gmail.com>
> Sent: Thursday, 9 June 2022 22:18
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Classic HA without SAN
>
> A NFSv4 server running on a dedicated NAS (or other Linux server running on ESX) can provide the shared-storage option for HA data storage using KahaDB
>
> Two (or more) ActiveMQ servers can mount the same NFSv4 file share and the first one to obtain the lock “wins” and becomes the primary (aka master) broker.
>
> -Matt
>
>> On Wed, Jun 8, 2022 at 9:23 AM Carbonneaux Yannick < yannick.carbonneaux@skyguide.ch> wrote:
>>
>>> Hi,
>>>
>>> We are planning the deployment of an ActiveMQ Classic cluster and I
>>> am looking for the best way to implement HA in a VMWare environment
>>> on RHEL OS without a SAN.
>>> It seems I'm down to Master/Slave with either something like GFS2 (so
>>> a RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that
>>> no messages are lost if we lose an ESX in our VMWare cluster.
>>>
>>> Are there any recommendations on how to implement HA in this context?
>>> Performance is not really problematic today... but might in a near future.
>>>
>>> Thanks for your help,
>>>
>>> ___
>>> Yannick Carbonneaux
>>>
>>>
>
Re: ActiveMQ Classic HA without SAN
Posted by Justin Bertram <jb...@apache.org>.
I may have misunderstood, but I assumed based on your description of your
problem (and the potential solutions) that you wanted a storage backend
that could replicate to avoid a single point of failure. A replicating
filesystem (e.g. GFS2) or database (e.g. MySQL) are the only ways to get
that with ActiveMQ "Classic." You asked about alternative options in
Artemis and Artemis does offer native replication [1] which might be worth
exploring.
In any case, when the back-end store fails and the broker is unable to
actually durably store or acknowledge persistent messages then the safest
thing to do is for the broker to shutdown. Continuing normal broker
operations while waiting for the storage back-end to become available is
dangerous and could lead to both lost and duplicated messages.
Justin
[1]
https://activemq.apache.org/components/artemis/documentation/latest/ha.html#data-replication
On Fri, Jun 10, 2022 at 1:38 AM Carbonneaux Yannick <
yannick.carbonneaux@skyguide.ch> wrote:
> We were actually discussing this possibility yesterday, but wondered what
> would be the behavior of the broker in case the shared FS fails, and have
> two options there:
> - let the VM exposing the NFS fail and be restarted on another ESX host
> by the cluster
> - or set the VM as fault tolerant on the ESX so that it is readily
> available shall the primary host fail.
>
> But I'm especially wondering what would be the best practice in case of
> failure of the shared storage with KahaDB ? Just piles up in memory until
> KahaDB comes back ? Or plainly fail ? Seems to be configurable via
> LeaseLockerIOExceptionHandler if I understood well.
>
> Yannick
>
>
> -----Original Message-----
> From: Matt Pavlovich <ma...@gmail.com>
> Sent: Thursday, 9 June 2022 22:18
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Classic HA without SAN
>
> A NFSv4 server running on a dedicated NAS (or other Linux server running
> on ESX) can provide the shared-storage option for HA data storage using
> KahaDB
>
> Two (or more) ActiveMQ servers can mount the same NFSv4 file share and the
> first one to obtain the lock “wins” and becomes the primary (aka master)
> broker.
>
> -Matt
>
> > On Wed, Jun 8, 2022 at 9:23 AM Carbonneaux Yannick <
> yannick.carbonneaux@skyguide.ch> wrote:
> >
> >> Hi,
> >>
> >> We are planning the deployment of an ActiveMQ Classic cluster and I
> >> am looking for the best way to implement HA in a VMWare environment
> >> on RHEL OS without a SAN.
> >> It seems I'm down to Master/Slave with either something like GFS2 (so
> >> a RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that
> >> no messages are lost if we lose an ESX in our VMWare cluster.
> >>
> >> Are there any recommendations on how to implement HA in this context?
> >> Performance is not really problematic today... but might in a near
> future.
> >>
> >> Thanks for your help,
> >>
> >> ___
> >> Yannick Carbonneaux
> >>
> >>
>
>
RE: ActiveMQ Classic HA without SAN
Posted by Carbonneaux Yannick <ya...@skyguide.ch>.
We were actually discussing this possibility yesterday, but wondered what would be the behavior of the broker in case the shared FS fails, and have two options there:
- let the VM exposing the NFS fail and be restarted on another ESX host by the cluster
- or set the VM as fault tolerant on the ESX so that it is readily available shall the primary host fail.
But I'm especially wondering what would be the best practice in case of failure of the shared storage with KahaDB ? Just piles up in memory until KahaDB comes back ? Or plainly fail ? Seems to be configurable via LeaseLockerIOExceptionHandler if I understood well.
Yannick
-----Original Message-----
From: Matt Pavlovich <ma...@gmail.com>
Sent: Thursday, 9 June 2022 22:18
To: users@activemq.apache.org
Subject: Re: ActiveMQ Classic HA without SAN
A NFSv4 server running on a dedicated NAS (or other Linux server running on ESX) can provide the shared-storage option for HA data storage using KahaDB
Two (or more) ActiveMQ servers can mount the same NFSv4 file share and the first one to obtain the lock “wins” and becomes the primary (aka master) broker.
-Matt
> On Wed, Jun 8, 2022 at 9:23 AM Carbonneaux Yannick < yannick.carbonneaux@skyguide.ch> wrote:
>
>> Hi,
>>
>> We are planning the deployment of an ActiveMQ Classic cluster and I
>> am looking for the best way to implement HA in a VMWare environment
>> on RHEL OS without a SAN.
>> It seems I'm down to Master/Slave with either something like GFS2 (so
>> a RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that
>> no messages are lost if we lose an ESX in our VMWare cluster.
>>
>> Are there any recommendations on how to implement HA in this context?
>> Performance is not really problematic today... but might in a near future.
>>
>> Thanks for your help,
>>
>> ___
>> Yannick Carbonneaux
>>
>>
Re: ActiveMQ Classic HA without SAN
Posted by Matt Pavlovich <ma...@gmail.com>.
A NFSv4 server running on a dedicated NAS (or other Linux server running on ESX) can provide the shared-storage option for HA data storage using KahaDB
Two (or more) ActiveMQ servers can mount the same NFSv4 file share and the first one to obtain the lock “wins” and becomes the primary (aka master) broker.
-Matt
> On Wed, Jun 8, 2022 at 9:23 AM Carbonneaux Yannick < yannick.carbonneaux@skyguide.ch> wrote:
>
>> Hi,
>>
>> We are planning the deployment of an ActiveMQ Classic cluster and I am
>> looking for the best way to implement HA in a VMWare environment on
>> RHEL OS without a SAN.
>> It seems I'm down to Master/Slave with either something like GFS2 (so
>> a RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that no
>> messages are lost if we lose an ESX in our VMWare cluster.
>>
>> Are there any recommendations on how to implement HA in this context?
>> Performance is not really problematic today... but might in a near future.
>>
>> Thanks for your help,
>>
>> ___
>> Yannick Carbonneaux
>>
>>
RE: ActiveMQ Classic HA without SAN
Posted by Carbonneaux Yannick <ya...@skyguide.ch>.
Thanks for this insight. Would by any chance Artemis offer better options for HA ?
Yannick
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Wednesday, 8 June 2022 18:38
To: users@activemq.apache.org
Subject: Re: ActiveMQ Classic HA without SAN
> It seems I'm down to Master/Slave with either something like GFS2 (so
> a
RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that no messages are lost if we lose an ESX in our VMWare cluster.
It sounds to me like you've got a good handle on your options.
Justin
On Wed, Jun 8, 2022 at 9:23 AM Carbonneaux Yannick < yannick.carbonneaux@skyguide.ch> wrote:
> Hi,
>
> We are planning the deployment of an ActiveMQ Classic cluster and I am
> looking for the best way to implement HA in a VMWare environment on
> RHEL OS without a SAN.
> It seems I'm down to Master/Slave with either something like GFS2 (so
> a RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that no
> messages are lost if we lose an ESX in our VMWare cluster.
>
> Are there any recommendations on how to implement HA in this context?
> Performance is not really problematic today... but might in a near future.
>
> Thanks for your help,
>
> ___
> Yannick Carbonneaux
>
>
Re: ActiveMQ Classic HA without SAN
Posted by Justin Bertram <jb...@apache.org>.
> It seems I'm down to Master/Slave with either something like GFS2 (so a
RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that no
messages are lost if we lose an ESX in our VMWare cluster.
It sounds to me like you've got a good handle on your options.
Justin
On Wed, Jun 8, 2022 at 9:23 AM Carbonneaux Yannick <
yannick.carbonneaux@skyguide.ch> wrote:
> Hi,
>
> We are planning the deployment of an ActiveMQ Classic cluster and I am
> looking for the best way to implement HA in a VMWare environment on RHEL OS
> without a SAN.
> It seems I'm down to Master/Slave with either something like GFS2 (so a
> RHEL Cluster) or JDBC with MySQL (or any other db) to ensure that no
> messages are lost if we lose an ESX in our VMWare cluster.
>
> Are there any recommendations on how to implement HA in this context?
> Performance is not really problematic today... but might in a near future.
>
> Thanks for your help,
>
> ___
> Yannick Carbonneaux
>
>