You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Francois Gaudreault <fg...@cloudops.com> on 2014/01/26 03:33:47 UTC
Connect to a MySQL Galera Cluster
We are trying to make CloudStack to work with a Galera two node
master/master cluster. We have a NetScaler doing DataStream LB in front
of the cluster, and we have two CloudStack management servers in
active/active.
Anyone had success with this kind of scenario? As soon as I shut a MySQL
node, the CloudStack JDBC panics, and the management server fence
themselves. Why this is happening? Is there any way to handle database
failures in HA fashion? Any large cloud providers using CloudStack
handle this?
Thanks.
--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC H3J 1S6
www.cloudops.com
@CloudOps_
Re: Connect to a MySQL Galera Cluster
Posted by Nux! <nu...@li.nux.ro>.
On 26.01.2014 04:40, Francois Gaudreault wrote:
> Yes, we tried with a NetScaler, but as soon as one MySQL dies, both
> management servers fence themselves on DB exceptions :S
I think part of your problem is you run only 2 galera nodes.
"Thus it is strongly advised that the minimum Galera cluster
configuration is 3 nodes." -
http://www.codership.com/wiki/doku.php?id=info
This is how we run it and it does indeed seem to be crucial for quorum.
Additionally we're using this software to deploy and manage it
(partially), it can help with failover and recovery.
http://www.severalnines.com/clustercontrol
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
Re: Connect to a MySQL Galera Cluster
Posted by Francois Gaudreault <fg...@cloudops.com>.
On 1/26/2014, 5:34 AM, Shanker Balan wrote:
> The other way would be to make the whole fault tolerance an app problem. If
> the app could automagically reconfigure itself to do round robin writes
> across the galera cluster based on health checks, we will get “HA” automatically.
> A failed node would end up being blacklisted by the app.
JDBC is supposed to be able to do that out of the box. But that means a
code recompile. Something like:
jdbc:mysql:loadbalance://node1,node2,nodeN/database?loadBalanceBlacklistTimeout=5000
--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC H3J 1S6
www.cloudops.com
@CloudOps_
Re: Connect to a MySQL Galera Cluster
Posted by Francois Gaudreault <fg...@cloudops.com>.
I just noticed some DB HA parameters in the db.properties.in file.
Anyone ever tried them?
It looks like CloudStack is able to handle slaves and autoReconnectForPool?
Francois
> I was not able to get much information but Looks like it does not manage sessions
> across the cluster. So if a node fails, all sessions currently handled by
> the node will eventually timeout and fail.
>
> It is up to the client now to open new connections.
>
> One way to work around the session handling issue would be to use a
> L7 proxy that can talk MySQL. I think mysql-proxy can do that but
> its alpha(?) s/w. Also, mysql-proxy itself is not HA.
>
> The other way would be to make the whole fault tolerance an app problem. If
> the app could automagically reconfigure itself to do round robin writes
> across the galera cluster based on health checks, we will get “HA” automatically.
> A failed node would end up being blacklisted by the app.
>
> As of now, I don’t see any other way other than to force a CloudStack restart
> in response to a galera node failure.
>
> Hoping someone with more db-foo can tell us more.
>
> --
> @shankerbalan
>
> M: +91 98860 60539 | O: +91 (80) 67935867
> shanker.balan@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
> ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
>
> Need Enterprise Grade Support for Apache CloudStack?
> Our CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> offers the best 24/7 SLA for CloudStack Environments.
>
> Apache CloudStack Bootcamp training courses
>
> **NEW!** CloudStack 4.2.1 training<http://shapeblue.com/cloudstack-training/>
> 18th-19th February 2014, Brazil. Classroom<http://shapeblue.com/cloudstack-training/>
> 17th-23rd March 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
> 24th-28th March 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
> 16th-20th June 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
> 23rd-27th June 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>
--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC H3J 1S6
www.cloudops.com
@CloudOps_
Re: Connect to a MySQL Galera Cluster
Posted by Shanker Balan <sh...@shapeblue.com>.
Comments inline.
On 26-Jan-2014, at 11:14 am, Shanker Balan <sh...@shapeblue.com> wrote:
> Comments inline.
>
> On 26-Jan-2014, at 10:10 am, Francois Gaudreault <fg...@cloudops.com> wrote:
>
>> Yes, we tried with a NetScaler, but as soon as one MySQL dies, both management servers fence themselves on DB exceptions :S
>
>
> Does Galera maintain client connection state across the failover?
I was not able to get much information but Looks like it does not manage sessions
across the cluster. So if a node fails, all sessions currently handled by
the node will eventually timeout and fail.
It is up to the client now to open new connections.
One way to work around the session handling issue would be to use a
L7 proxy that can talk MySQL. I think mysql-proxy can do that but
its alpha(?) s/w. Also, mysql-proxy itself is not HA.
The other way would be to make the whole fault tolerance an app problem. If
the app could automagically reconfigure itself to do round robin writes
across the galera cluster based on health checks, we will get “HA” automatically.
A failed node would end up being blacklisted by the app.
As of now, I don’t see any other way other than to force a CloudStack restart
in response to a galera node failure.
Hoping someone with more db-foo can tell us more.
--
@shankerbalan
M: +91 98860 60539 | O: +91 (80) 67935867
shanker.balan@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
Need Enterprise Grade Support for Apache CloudStack?
Our CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> offers the best 24/7 SLA for CloudStack Environments.
Apache CloudStack Bootcamp training courses
**NEW!** CloudStack 4.2.1 training<http://shapeblue.com/cloudstack-training/>
18th-19th February 2014, Brazil. Classroom<http://shapeblue.com/cloudstack-training/>
17th-23rd March 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
24th-28th March 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
16th-20th June 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
23rd-27th June 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: Connect to a MySQL Galera Cluster
Posted by Shanker Balan <sh...@shapeblue.com>.
Comments inline.
On 26-Jan-2014, at 10:10 am, Francois Gaudreault <fg...@cloudops.com> wrote:
> Yes, we tried with a NetScaler, but as soon as one MySQL dies, both management servers fence themselves on DB exceptions :S
Does Galera maintain client connection state across the failover?
--
@shankerbalan
M: +91 98860 60539 | O: +91 (80) 67935867
shanker.balan@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
Need Enterprise Grade Support for Apache CloudStack?
Our CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> offers the best 24/7 SLA for CloudStack Environments.
Apache CloudStack Bootcamp training courses
**NEW!** CloudStack 4.2.1 training<http://shapeblue.com/cloudstack-training/>
18th-19th February 2014, Brazil. Classroom<http://shapeblue.com/cloudstack-training/>
17th-23rd March 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
24th-28th March 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
16th-20th June 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
23rd-27th June 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: Connect to a MySQL Galera Cluster
Posted by Francois Gaudreault <fg...@cloudops.com>.
Yes, we tried with a NetScaler, but as soon as one MySQL dies, both
management servers fence themselves on DB exceptions :S
Francois
On 1/25/2014, 10:08 PM, Nux! wrote:
> On 26.01.2014 02:33, Francois Gaudreault wrote:
>> We are trying to make CloudStack to work with a Galera two node
>> master/master cluster. We have a NetScaler doing DataStream LB in
>> front of the cluster, and we have two CloudStack management servers in
>> active/active.
>>
>> Anyone had success with this kind of scenario? As soon as I shut a
>> MySQL node, the CloudStack JDBC panics, and the management server
>> fence themselves. Why this is happening? Is there any way to handle
>> database failures in HA fashion? Any large cloud providers using
>> CloudStack handle this?
>
> Hello,
>
> I know there are people on the ml who are using galera and I'm
> planning on using it myself for Cloudstack.
> We already have a 3 node deployment, but it is loadbalanced on virtual
> IPs and load balancer (we run VIP on each node as well as Haproxy),
> hopefully this will keep Cloudstack happy, I suggest you do a similar
> thing.
>
> Lucian
>
--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC H3J 1S6
www.cloudops.com
@CloudOps_
Re: Connect to a MySQL Galera Cluster
Posted by Nux! <nu...@li.nux.ro>.
On 26.01.2014 02:33, Francois Gaudreault wrote:
> We are trying to make CloudStack to work with a Galera two node
> master/master cluster. We have a NetScaler doing DataStream LB in
> front of the cluster, and we have two CloudStack management servers in
> active/active.
>
> Anyone had success with this kind of scenario? As soon as I shut a
> MySQL node, the CloudStack JDBC panics, and the management server
> fence themselves. Why this is happening? Is there any way to handle
> database failures in HA fashion? Any large cloud providers using
> CloudStack handle this?
Hello,
I know there are people on the ml who are using galera and I'm planning
on using it myself for Cloudstack.
We already have a 3 node deployment, but it is loadbalanced on virtual
IPs and load balancer (we run VIP on each node as well as Haproxy),
hopefully this will keep Cloudstack happy, I suggest you do a similar
thing.
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro