You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Gavin Alexander <ga...@gmail.com> on 2012/07/26 17:02:10 UTC
2 Node HA Cluster
Hi All,
I'm looking to set up a durable HA broker and had a few questions.
First off, sorry if these questions are more appropriate on a Linux-HA
forum... just trying to see if other people have come across the same
issues.
After setting up a two node cluster and being quite happy with the
ease of config - I realised that I would have to solve the split-brain
problem.
So, using 3 virtual machines, I followed the guide @
https://cwiki.apache.org/confluence/display/qpid/Configuring+qpidd+with+Cluster+Manager
which works reasonable well - (I've had a few issues with rgmanager
hanging occasionally during failover... but I digress...)
The environment that I'm using at the moment is on a bunch of Linux
KVM's, but it will eventually be moved to physical infrastructure.
I have some reasonable performance requirements and the hardware I'm
using is quite expensive - so adding another node for HA just to
maintain a quorum seems "wasteful".
So, I'm investigating if a 2 node setup is possible.
Is it possible to do HA with two nodes (without a "real" qdisk)?
I've tried adding a 3rd node (a cheap virtual machine) for quorum only
- I.e. it doesn't run qpidd and is only used for quorum. It seems to
work - although I haven't thoroughly tested all failure scenarios yet.
Other idea's I've had are
- Using Qdisk heuristics to generate more votes - everything I read
seems to suggest not to do this...
- Integrating with a load balancer (there's an existing LVS cluster
on the same network - that I can piggy-back on) to protect against
split-brain "somehow" :)
- Leave it as a two node cluster and rely on corosync totem protocol,
or interface bonding for fault tolerance. Then, if a split brain does
occur, it's pretty certain (about 99.9% :) that clients wouldn't be
able to access the cluster anyway.
A couple of questions...
Do cluster messages require acknowledgement from cluster nodes before
sending ack's back to the client (if the clients require
acknowledgement)? And hence, does adding more nodes degrade
performance?
Has anyone ever tried running qpid over something like drbd?
Is it possible to specify a bind address for qpidd?
Look forward to any replies/guidance/commentary
Thanks,
/gav
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: 2 Node HA Cluster
Posted by Alan Conway <ac...@redhat.com>.
On Fri, 2012-07-27 at 01:02 +1000, Gavin Alexander wrote:
> Hi All,
>
> I'm looking to set up a durable HA broker and had a few questions.
> First off, sorry if these questions are more appropriate on a Linux-HA
> forum... just trying to see if other people have come across the same
> issues.
>
> After setting up a two node cluster and being quite happy with the
> ease of config - I realised that I would have to solve the split-brain
> problem.
> So, using 3 virtual machines, I followed the guide @
> https://cwiki.apache.org/confluence/display/qpid/Configuring+qpidd+with+Cluster+Manager
> which works reasonable well - (I've had a few issues with rgmanager
> hanging occasionally during failover... but I digress...)
>
> The environment that I'm using at the moment is on a bunch of Linux
> KVM's, but it will eventually be moved to physical infrastructure.
> I have some reasonable performance requirements and the hardware I'm
> using is quite expensive - so adding another node for HA just to
> maintain a quorum seems "wasteful".
>
> So, I'm investigating if a 2 node setup is possible.
>
> Is it possible to do HA with two nodes (without a "real" qdisk)?
> I've tried adding a 3rd node (a cheap virtual machine) for quorum only
> - I.e. it doesn't run qpidd and is only used for quorum. It seems to
> work - although I haven't thoroughly tested all failure scenarios yet.
> Other idea's I've had are
> - Using Qdisk heuristics to generate more votes - everything I read
> seems to suggest not to do this...
> - Integrating with a load balancer (there's an existing LVS cluster
> on the same network - that I can piggy-back on) to protect against
> split-brain "somehow" :)
> - Leave it as a two node cluster and rely on corosync totem protocol,
> or interface bonding for fault tolerance. Then, if a split brain does
> occur, it's pretty certain (about 99.9% :) that clients wouldn't be
> able to access the cluster anyway.
>
There is a special cman configuration option to set up a 2 node cluster
(check the cman docs for "two_node" or something like that.) It requires
hardware fencing. It doesn't do quorum, instead it works like a
shoot-out: in a partition each node tries to fence the other and only
one survives.
> A couple of questions...
> Do cluster messages require acknowledgement from cluster nodes before
> sending ack's back to the client (if the clients require
> acknowledgement)? And hence, does adding more nodes degrade
> performance?
There are no acks as such between the brokers. The cluster relies on
virtual synchrony, provided by corosync, to ensure each broker gets a
consistent set of messages from all its peers. Brokers use multicast to
exchange messages so there should not be a big performance difference
with more brokers.
> Has anyone ever tried running qpid over something like drbd?
Not that I know of.
> Is it possible to specify a bind address for qpidd?
I'm not sure what you mean by a "bind address".
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
RE: 2 Node HA Cluster
Posted by Jan Bares <ja...@wood.cz>.
Thanks for pointing me to altname tag. I changed node name (previously it matched uname -n) and added new line to hosts with the IP I want to use for cluster communications. It seems to solve the problem. In the case I will have backup (multihome) for cluster communicaton, I will add them as altname.
Regards, Jan
> -----Original Message-----
> From: Gavin Alexander [mailto:gavin.alexander@gmail.com]
> Sent: Tuesday, August 28, 2012 10:28 AM
> To: users@qpid.apache.org
> Subject: Re: 2 Node HA Cluster
>
> To be honest, I've only tested this setup on KVM's. About to do some
> prototyping on actual hardware next week - with the above setup.
>
> What does your cluster.conf look like?
> If I set the altname parameter under <clusternodes> in cluster.conf -
> corosync-objctl seems to show the right thing.
>
> Selection of the interface, I believe, is based on the address
> specified by "altname" above.
>
> Cheers,
>
> /gav
>
> On 28 August 2012 17:45, Jan Bares <ja...@wood.cz> wrote:
> > Hi,
> >
> > Just out of curiosity:
> >
> >> > For those interested.
> >> > The configuration that I've settled on is
> >> > Two Nodes running qpidd + 1 Quorum Node
> >> > Each node has 2 Nics, allocated on separate switching
> >> infrastructure/networks.
> >> > One network is for cluster comms the other to serve requests (its
> >> also
> >> > used as a backup interface - I'm using Totem - RRP (Redundant Ring
> >> > Protocol)).
> >
> > CMAN doesn't support binding totem to specific interface, are you
> sure that your setup works as described? I have eth0 for requests and
> management and eth1 dedicated for cluster network. Unfortunately CMAN
> uses eth0 for cluster communication, I can verify it with iptables and
> corosync-objctl. There is no way how to configure eth1 as totem bind
> address in cluster.conf.
> >
> > Can someone verify that the first of ethX will be used? I can
> mitigate the problem above by switching eth0 and eht1 but I want to be
> sure the bind interfaces is not selected randomly.
> >
> > Thanks, Jan
> >
> >
> >
> >
> >
> > DISCLAIMER
> > WOOD & Company Financial Services, a.s. and its branches are
> authorized and regulated by the CNB as Home State regulator and in
> Poland by the KNF, in Romania by the CNVM, in Slovakia by the NBS and
> in the UK by the FSA as Host State regulators. For further information
> about WOOD & Co., its investment services, financial instruments and
> associated risks, safeguard client assets (incl. compensation schemes)
> and contractual relationship please see our website at www.wood.cz
> under section Corporate Governance.
> >
> > Unless otherwise stated, this transmission is neither an offer nor
> the solicitation of an offer to sell or purchase any investment. All
> estimates, opinions and other information contained herein are subject
> to change without notice and are provided in good faith but without
> legal responsibility or liability. Opinion may be personal to the
> author and may not reflect the opinions of WOOD & Co. Communications
> from sales persons, sales traders or traders should not be regarded as
> investment research and may contain opinions or trading ideas which are
> different from WOOD & Co. investment research opinions.
> >
> > This e-mail and any attachments are confidential and may be
> privileged or otherwise protected from disclosure. If you are not a
> named addressee you must not use, disclose, distribute, copy, print or
> rely on this e-mail and any of its attachments. Please notify the
> sender that you have received this email by mistake by replying to the
> email, and then delete the email and any copies of it. Although WOOD &
> Co. routinely screens e-mails for viruses, addressees should scan this
> e-mail and any attachments for viruses. WOOD & Co. makes no
> representation or warranty as to the absence of viruses in this e-mail
> or any attachments. Please note that to ensure regulatory compliance
> and for the protection of our clients and business, we may monitor and
> read e-mails sent to and from our server(s).
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
DISCLAIMER
WOOD & Company Financial Services, a.s. and its branches are authorized and regulated by the CNB as Home State regulator and in Poland by the KNF, in Romania by the CNVM, in Slovakia by the NBS and in the UK by the FSA as Host State regulators. For further information about WOOD & Co., its investment services, financial instruments and associated risks, safeguard client assets (incl. compensation schemes) and contractual relationship please see our website at www.wood.cz under section Corporate Governance.
Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. All estimates, opinions and other information contained herein are subject to change without notice and are provided in good faith but without legal responsibility or liability. Opinion may be personal to the author and may not reflect the opinions of WOOD & Co. Communications from sales persons, sales traders or traders should not be regarded as investment research and may contain opinions or trading ideas which are different from WOOD & Co. investment research opinions.
This e-mail and any attachments are confidential and may be privileged or otherwise protected from disclosure. If you are not a named addressee you must not use, disclose, distribute, copy, print or rely on this e-mail and any of its attachments. Please notify the sender that you have received this email by mistake by replying to the email, and then delete the email and any copies of it. Although WOOD & Co. routinely screens e-mails for viruses, addressees should scan this e-mail and any attachments for viruses. WOOD & Co. makes no representation or warranty as to the absence of viruses in this e-mail or any attachments. Please note that to ensure regulatory compliance and for the protection of our clients and business, we may monitor and read e-mails sent to and from our server(s).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: 2 Node HA Cluster
Posted by Gavin Alexander <ga...@gmail.com>.
To be honest, I've only tested this setup on KVM's. About to do some
prototyping on actual hardware next week - with the above setup.
What does your cluster.conf look like?
If I set the altname parameter under <clusternodes> in cluster.conf -
corosync-objctl seems to show the right thing.
Selection of the interface, I believe, is based on the address
specified by "altname" above.
Cheers,
/gav
On 28 August 2012 17:45, Jan Bares <ja...@wood.cz> wrote:
> Hi,
>
> Just out of curiosity:
>
>> > For those interested.
>> > The configuration that I've settled on is
>> > Two Nodes running qpidd + 1 Quorum Node
>> > Each node has 2 Nics, allocated on separate switching
>> infrastructure/networks.
>> > One network is for cluster comms the other to serve requests (its
>> also
>> > used as a backup interface - I'm using Totem - RRP (Redundant Ring
>> > Protocol)).
>
> CMAN doesn't support binding totem to specific interface, are you sure that your setup works as described? I have eth0 for requests and management and eth1 dedicated for cluster network. Unfortunately CMAN uses eth0 for cluster communication, I can verify it with iptables and corosync-objctl. There is no way how to configure eth1 as totem bind address in cluster.conf.
>
> Can someone verify that the first of ethX will be used? I can mitigate the problem above by switching eth0 and eht1 but I want to be sure the bind interfaces is not selected randomly.
>
> Thanks, Jan
>
>
>
>
>
> DISCLAIMER
> WOOD & Company Financial Services, a.s. and its branches are authorized and regulated by the CNB as Home State regulator and in Poland by the KNF, in Romania by the CNVM, in Slovakia by the NBS and in the UK by the FSA as Host State regulators. For further information about WOOD & Co., its investment services, financial instruments and associated risks, safeguard client assets (incl. compensation schemes) and contractual relationship please see our website at www.wood.cz under section Corporate Governance.
>
> Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. All estimates, opinions and other information contained herein are subject to change without notice and are provided in good faith but without legal responsibility or liability. Opinion may be personal to the author and may not reflect the opinions of WOOD & Co. Communications from sales persons, sales traders or traders should not be regarded as investment research and may contain opinions or trading ideas which are different from WOOD & Co. investment research opinions.
>
> This e-mail and any attachments are confidential and may be privileged or otherwise protected from disclosure. If you are not a named addressee you must not use, disclose, distribute, copy, print or rely on this e-mail and any of its attachments. Please notify the sender that you have received this email by mistake by replying to the email, and then delete the email and any copies of it. Although WOOD & Co. routinely screens e-mails for viruses, addressees should scan this e-mail and any attachments for viruses. WOOD & Co. makes no representation or warranty as to the absence of viruses in this e-mail or any attachments. Please note that to ensure regulatory compliance and for the protection of our clients and business, we may monitor and read e-mails sent to and from our server(s).
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
RE: 2 Node HA Cluster
Posted by Jan Bares <ja...@wood.cz>.
Hi,
Just out of curiosity:
> > For those interested.
> > The configuration that I've settled on is
> > Two Nodes running qpidd + 1 Quorum Node
> > Each node has 2 Nics, allocated on separate switching
> infrastructure/networks.
> > One network is for cluster comms the other to serve requests (its
> also
> > used as a backup interface - I'm using Totem - RRP (Redundant Ring
> > Protocol)).
CMAN doesn't support binding totem to specific interface, are you sure that your setup works as described? I have eth0 for requests and management and eth1 dedicated for cluster network. Unfortunately CMAN uses eth0 for cluster communication, I can verify it with iptables and corosync-objctl. There is no way how to configure eth1 as totem bind address in cluster.conf.
Can someone verify that the first of ethX will be used? I can mitigate the problem above by switching eth0 and eht1 but I want to be sure the bind interfaces is not selected randomly.
Thanks, Jan
DISCLAIMER
WOOD & Company Financial Services, a.s. and its branches are authorized and regulated by the CNB as Home State regulator and in Poland by the KNF, in Romania by the CNVM, in Slovakia by the NBS and in the UK by the FSA as Host State regulators. For further information about WOOD & Co., its investment services, financial instruments and associated risks, safeguard client assets (incl. compensation schemes) and contractual relationship please see our website at www.wood.cz under section Corporate Governance.
Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. All estimates, opinions and other information contained herein are subject to change without notice and are provided in good faith but without legal responsibility or liability. Opinion may be personal to the author and may not reflect the opinions of WOOD & Co. Communications from sales persons, sales traders or traders should not be regarded as investment research and may contain opinions or trading ideas which are different from WOOD & Co. investment research opinions.
This e-mail and any attachments are confidential and may be privileged or otherwise protected from disclosure. If you are not a named addressee you must not use, disclose, distribute, copy, print or rely on this e-mail and any of its attachments. Please notify the sender that you have received this email by mistake by replying to the email, and then delete the email and any copies of it. Although WOOD & Co. routinely screens e-mails for viruses, addressees should scan this e-mail and any attachments for viruses. WOOD & Co. makes no representation or warranty as to the absence of viruses in this e-mail or any attachments. Please note that to ensure regulatory compliance and for the protection of our clients and business, we may monitor and read e-mails sent to and from our server(s).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: 2 Node HA Cluster
Posted by Gordon Sim <gs...@redhat.com>.
On 08/10/2012 02:26 PM, Alan Conway wrote:
> On Thu, 2012-08-09 at 20:01 +1000, Gavin Alexander wrote:
>> Hi Alan,
>>
>> Thanks for getting back to me.
>> Some of limitations would make this a no go for me at the moment.
>> I'm guessing the "older cluster" module will be phased out gradually?
>
> Yes that's the plan.
Specifically, the upcoming 0.18 release will be the last release to have
the old cluster module in it.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: 2 Node HA Cluster
Posted by Alan Conway <ac...@redhat.com>.
On Thu, 2012-08-09 at 20:01 +1000, Gavin Alexander wrote:
> Hi Alan,
>
> Thanks for getting back to me.
> Some of limitations would make this a no go for me at the moment.
> I'm guessing the "older cluster" module will be phased out gradually?
Yes that's the plan.
> Obviously the burden of maintaining both strategies long term will be
> a pain...
> (any). Is it possible to bind to a specific local IP?
Not yet, see https://issues.apache.org/jira/browse/QPID-3351
> For those interested.
> The configuration that I've settled on is
> Two Nodes running qpidd + 1 Quorum Node
> Each node has 2 Nics, allocated on separate switching infrastructure/networks.
> One network is for cluster comms the other to serve requests (its also
> used as a backup interface - I'm using Totem - RRP (Redundant Ring
> Protocol)).
> Now for some performance testing...... :)
>
Sounds like a good setup, good luck!
> On 2 August 2012 06:01, Alan Conway <ac...@redhat.com> wrote:
> > I should have also said: There is a new HA module in Qpid that will
> > eventually replace the older cluster module. You should take a look at
> > that as well, if you're starting out on a new project.
> >
> > http://qpid.apache.org/books/trunk/AMQP-Messaging-Broker-CPP-Book/html/chap-Messaging_User_Guide-Active_Passive_Cluster.html
> >
> > On Fri, 2012-07-27 at 01:02 +1000, Gavin Alexander wrote:
> >> Hi All,
> >>
> >> I'm looking to set up a durable HA broker and had a few questions.
> >> First off, sorry if these questions are more appropriate on a Linux-HA
> >> forum... just trying to see if other people have come across the same
> >> issues.
> >>
> >> After setting up a two node cluster and being quite happy with the
> >> ease of config - I realised that I would have to solve the split-brain
> >> problem.
> >> So, using 3 virtual machines, I followed the guide @
> >> https://cwiki.apache.org/confluence/display/qpid/Configuring+qpidd+with+Cluster+Manager
> >> which works reasonable well - (I've had a few issues with rgmanager
> >> hanging occasionally during failover... but I digress...)
> >>
> >> The environment that I'm using at the moment is on a bunch of Linux
> >> KVM's, but it will eventually be moved to physical infrastructure.
> >> I have some reasonable performance requirements and the hardware I'm
> >> using is quite expensive - so adding another node for HA just to
> >> maintain a quorum seems "wasteful".
> >>
> >> So, I'm investigating if a 2 node setup is possible.
> >>
> >> Is it possible to do HA with two nodes (without a "real" qdisk)?
> >> I've tried adding a 3rd node (a cheap virtual machine) for quorum only
> >> - I.e. it doesn't run qpidd and is only used for quorum. It seems to
> >> work - although I haven't thoroughly tested all failure scenarios yet.
> >> Other idea's I've had are
> >> - Using Qdisk heuristics to generate more votes - everything I read
> >> seems to suggest not to do this...
> >> - Integrating with a load balancer (there's an existing LVS cluster
> >> on the same network - that I can piggy-back on) to protect against
> >> split-brain "somehow" :)
> >> - Leave it as a two node cluster and rely on corosync totem protocol,
> >> or interface bonding for fault tolerance. Then, if a split brain does
> >> occur, it's pretty certain (about 99.9% :) that clients wouldn't be
> >> able to access the cluster anyway.
> >>
> >>
> >> A couple of questions...
> >> Do cluster messages require acknowledgement from cluster nodes before
> >> sending ack's back to the client (if the clients require
> >> acknowledgement)? And hence, does adding more nodes degrade
> >> performance?
> >> Has anyone ever tried running qpid over something like drbd?
> >> Is it possible to specify a bind address for qpidd?
> >>
> >> Look forward to any replies/guidance/commentary
> >>
> >> Thanks,
> >>
> >> /gav
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> For additional commands, e-mail: users-help@qpid.apache.org
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: 2 Node HA Cluster
Posted by Gavin Alexander <ga...@gmail.com>.
Hi Alan,
Thanks for getting back to me.
Some of limitations would make this a no go for me at the moment.
I'm guessing the "older cluster" module will be phased out gradually?
Obviously the burden of maintaining both strategies long term will be
a pain...
By "bind address" - I mean, when qpidd starts up it listens on 0.0.0.0
(any). Is it possible to bind to a specific local IP?
For those interested.
The configuration that I've settled on is
Two Nodes running qpidd + 1 Quorum Node
Each node has 2 Nics, allocated on separate switching infrastructure/networks.
One network is for cluster comms the other to serve requests (its also
used as a backup interface - I'm using Totem - RRP (Redundant Ring
Protocol)).
Now for some performance testing...... :)
/gav
On 2 August 2012 06:01, Alan Conway <ac...@redhat.com> wrote:
> I should have also said: There is a new HA module in Qpid that will
> eventually replace the older cluster module. You should take a look at
> that as well, if you're starting out on a new project.
>
> http://qpid.apache.org/books/trunk/AMQP-Messaging-Broker-CPP-Book/html/chap-Messaging_User_Guide-Active_Passive_Cluster.html
>
> On Fri, 2012-07-27 at 01:02 +1000, Gavin Alexander wrote:
>> Hi All,
>>
>> I'm looking to set up a durable HA broker and had a few questions.
>> First off, sorry if these questions are more appropriate on a Linux-HA
>> forum... just trying to see if other people have come across the same
>> issues.
>>
>> After setting up a two node cluster and being quite happy with the
>> ease of config - I realised that I would have to solve the split-brain
>> problem.
>> So, using 3 virtual machines, I followed the guide @
>> https://cwiki.apache.org/confluence/display/qpid/Configuring+qpidd+with+Cluster+Manager
>> which works reasonable well - (I've had a few issues with rgmanager
>> hanging occasionally during failover... but I digress...)
>>
>> The environment that I'm using at the moment is on a bunch of Linux
>> KVM's, but it will eventually be moved to physical infrastructure.
>> I have some reasonable performance requirements and the hardware I'm
>> using is quite expensive - so adding another node for HA just to
>> maintain a quorum seems "wasteful".
>>
>> So, I'm investigating if a 2 node setup is possible.
>>
>> Is it possible to do HA with two nodes (without a "real" qdisk)?
>> I've tried adding a 3rd node (a cheap virtual machine) for quorum only
>> - I.e. it doesn't run qpidd and is only used for quorum. It seems to
>> work - although I haven't thoroughly tested all failure scenarios yet.
>> Other idea's I've had are
>> - Using Qdisk heuristics to generate more votes - everything I read
>> seems to suggest not to do this...
>> - Integrating with a load balancer (there's an existing LVS cluster
>> on the same network - that I can piggy-back on) to protect against
>> split-brain "somehow" :)
>> - Leave it as a two node cluster and rely on corosync totem protocol,
>> or interface bonding for fault tolerance. Then, if a split brain does
>> occur, it's pretty certain (about 99.9% :) that clients wouldn't be
>> able to access the cluster anyway.
>>
>>
>> A couple of questions...
>> Do cluster messages require acknowledgement from cluster nodes before
>> sending ack's back to the client (if the clients require
>> acknowledgement)? And hence, does adding more nodes degrade
>> performance?
>> Has anyone ever tried running qpid over something like drbd?
>> Is it possible to specify a bind address for qpidd?
>>
>> Look forward to any replies/guidance/commentary
>>
>> Thanks,
>>
>> /gav
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: 2 Node HA Cluster
Posted by Alan Conway <ac...@redhat.com>.
I should have also said: There is a new HA module in Qpid that will
eventually replace the older cluster module. You should take a look at
that as well, if you're starting out on a new project.
http://qpid.apache.org/books/trunk/AMQP-Messaging-Broker-CPP-Book/html/chap-Messaging_User_Guide-Active_Passive_Cluster.html
On Fri, 2012-07-27 at 01:02 +1000, Gavin Alexander wrote:
> Hi All,
>
> I'm looking to set up a durable HA broker and had a few questions.
> First off, sorry if these questions are more appropriate on a Linux-HA
> forum... just trying to see if other people have come across the same
> issues.
>
> After setting up a two node cluster and being quite happy with the
> ease of config - I realised that I would have to solve the split-brain
> problem.
> So, using 3 virtual machines, I followed the guide @
> https://cwiki.apache.org/confluence/display/qpid/Configuring+qpidd+with+Cluster+Manager
> which works reasonable well - (I've had a few issues with rgmanager
> hanging occasionally during failover... but I digress...)
>
> The environment that I'm using at the moment is on a bunch of Linux
> KVM's, but it will eventually be moved to physical infrastructure.
> I have some reasonable performance requirements and the hardware I'm
> using is quite expensive - so adding another node for HA just to
> maintain a quorum seems "wasteful".
>
> So, I'm investigating if a 2 node setup is possible.
>
> Is it possible to do HA with two nodes (without a "real" qdisk)?
> I've tried adding a 3rd node (a cheap virtual machine) for quorum only
> - I.e. it doesn't run qpidd and is only used for quorum. It seems to
> work - although I haven't thoroughly tested all failure scenarios yet.
> Other idea's I've had are
> - Using Qdisk heuristics to generate more votes - everything I read
> seems to suggest not to do this...
> - Integrating with a load balancer (there's an existing LVS cluster
> on the same network - that I can piggy-back on) to protect against
> split-brain "somehow" :)
> - Leave it as a two node cluster and rely on corosync totem protocol,
> or interface bonding for fault tolerance. Then, if a split brain does
> occur, it's pretty certain (about 99.9% :) that clients wouldn't be
> able to access the cluster anyway.
>
>
> A couple of questions...
> Do cluster messages require acknowledgement from cluster nodes before
> sending ack's back to the client (if the clients require
> acknowledgement)? And hence, does adding more nodes degrade
> performance?
> Has anyone ever tried running qpid over something like drbd?
> Is it possible to specify a bind address for qpidd?
>
> Look forward to any replies/guidance/commentary
>
> Thanks,
>
> /gav
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org