You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Andrija Panic <an...@gmail.com> on 2015/06/04 16:08:30 UTC
Re: database high availability question vs haproxy
Anyone :) ?
On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
> Hi,
>
> I would have a question on database HA feature in db.properties (
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> )
>
> If I understand correctly, it is up to the admin to provide appropriate
> mysql HA (active-active, galera, etc) and ACS management server will JUST
> try to connect to slaves if the master is down ?
>
> We are running Galera, with haproxy/keepalived, and by using stoping
> haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and
> failover IP to another host.
>
> During these 6 seconds, ACS managemnt server goes dead, because of this DB
> unavailability.
>
> So my wondering, is better to use ACS db HA feature, instead of
> loadbalancer for this specific purpose ?
> (we are also using haproxy/keepalived for management server loadbalancing
> - 2 servers in backend...)
>
> Any experience shared is really appreciated !
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Andrija Panic <an...@gmail.com>.
True..Thanks Simon...
On 5 June 2015 at 16:28, Simon Weller <sw...@ena.com> wrote:
> I'm not sure whether that's configurable and I haven't seen any examples
> out there that point to it being configurable. Having said that, there are
> people a lot more familiar with that than I am on the list.
>
> One option you do have for this particular failure situation, is to use
> monit to monitor the pid of the CS management service and if it's not
> running, force it to start up again.
>
> Losing a load balancer completely is very rare, so it's not likely to
> happen often at all.
>
> - Si
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 9:07 AM
> To: users@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Exactly...thanks Simon for the time and help :)
>
> Is there any mysql timeout/retry parameter that is set in db.properties
> (like db.cloud.queriesBeforeRetryMaster=5000), but I'm wondering since I'm
> using keepalived/haproxy setup, it takes up to 5-6 sec for keepalive to
> detect haproxy is down (while testing I shutdown haproxu on active node),
> and then keepalvied moves IP to another node - meaning I have 5-6sec of no
> connection between ACS mgmt servers and database/haproxy.
>
> Is this timeout configurabile on ACS/mysql connector side ?
>
> Thanks again,
> Andrija
>
> On 5 June 2015 at 15:54, Simon Weller <sw...@ena.com> wrote:
>
> > I think haproxy is a better design in my opinion. You're going to have to
> > use haproxy to balance the host agents to multiple management servers
> > anyway, so you'll already be using it. haproxy can then manage the health
> > checks to Galera rather than over complicating it with the CS Management
> > configuration.
> >
> > - Si
> >
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Friday, June 5, 2015 8:37 AM
> > To: users@cloudstack.apache.org
> > Cc: dev@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Thanks Simon - that is what I thought...
> >
> > So my question would be then, haproxy vs native ACS/mysql connector going
> > to galera1/galera2/etc...will figure out, for now we use haproxy for
> > mysql/galera loadbalancing...
> >
> >
> > THanks a lot Simon,
> > Andrija
> >
> > On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
> >
> > >
> > > Personally, I think that Gallera is always going to be a safer option,
> as
> > > it handles conflict resolution natively. Having said that, it appears
> > care
> > > has been taken in designing the ACS MGMT DB integration so that the
> > chance
> > > of conflicts is very low. Galera requires a 3 nodes minimum, so it's a
> > lot
> > > of hardware unless you've got plans to use it elsewhere in your
> > > organisation.
> > >
> > > The downside to Galera, is that it's synchronous replication, so it
> needs
> > > very low latency between nodes. That doesn't make it a good candidate
> for
> > > geographic separation between DB nodes for a DR scenario.
> > >
> > > You're understanding of the replication structure, as based on the
> design
> > > document is correct. MySQL (or Galera) handles all the replication. ACS
> > > just handles which node it's writing and reading from. In a 2 node
> native
> > > MySQL cluster, it's expected that you are setup for cross master-master
> > > replication.
> > >
> > > - Si
> > > ________________________________________
> > > From: Andrija Panic <an...@gmail.com>
> > > Sent: Friday, June 5, 2015 2:41 AM
> > > To: dev@cloudstack.apache.org
> > > Cc: users@cloudstack.apache.org
> > > Subject: Re: database high availability question vs haproxy
> > >
> > > Hi Simon,
> > >
> > > thanks for the link - actually I have already read this - but Im still
> > > seaking for some answeres :) :
> > >
> > > - real world experience with DB HA in general - is i better to use
> > > haproxy(clustered/redudant) for mysql towards Galera cluster - or
> simply
> > to
> > > reference 2 nodes (1 as master, another as slave) with native ACS DB HA
> > -
> > > silly question but anyway...
> > > - my understanding - ACS just pings and connects to master or slave
> (all
> > > replication etc, is done from my side, not from ACS) ?
> > >
> > > Thanks again and any info is greatly appreciated.
> > >
> > > Andrija
> > >
> > > On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> > >
> > > > Andrija,
> > > >
> > > > Here is the original design document, and it should give you a better
> > > idea
> > > > of what is implemented today:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> > > >
> > > > We have plans to test this in our lab soon, but just haven't got
> around
> > > to
> > > > it yet.
> > > >
> > > > - Si
> > > >
> > > > ________________________________________
> > > > From: Andrija Panic <an...@gmail.com>
> > > > Sent: Thursday, June 4, 2015 9:08 AM
> > > > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > > > Subject: Re: database high availability question vs haproxy
> > > >
> > > > Anyone :) ?
> > > >
> > > > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I would have a question on database HA feature in db.properties (
> > > > >
> > > >
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > > > )
> > > > >
> > > > > If I understand correctly, it is up to the admin to provide
> > appropriate
> > > > > mysql HA (active-active, galera, etc) and ACS management server
> will
> > > > JUST
> > > > > try to connect to slaves if the master is down ?
> > > > >
> > > > > We are running Galera, with haproxy/keepalived, and by using
> stoping
> > > > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is
> down,
> > > and
> > > > > failover IP to another host.
> > > > >
> > > > > During these 6 seconds, ACS managemnt server goes dead, because of
> > this
> > > > DB
> > > > > unavailability.
> > > > >
> > > > > So my wondering, is better to use ACS db HA feature, instead of
> > > > > loadbalancer for this specific purpose ?
> > > > > (we are also using haproxy/keepalived for management server
> > > loadbalancing
> > > > > - 2 servers in backend...)
> > > > >
> > > > > Any experience shared is really appreciated !
> > > > > --
> > > > >
> > > > > Andrija Panić
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrija Panić
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Andrija Panic <an...@gmail.com>.
True..Thanks Simon...
On 5 June 2015 at 16:28, Simon Weller <sw...@ena.com> wrote:
> I'm not sure whether that's configurable and I haven't seen any examples
> out there that point to it being configurable. Having said that, there are
> people a lot more familiar with that than I am on the list.
>
> One option you do have for this particular failure situation, is to use
> monit to monitor the pid of the CS management service and if it's not
> running, force it to start up again.
>
> Losing a load balancer completely is very rare, so it's not likely to
> happen often at all.
>
> - Si
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 9:07 AM
> To: users@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Exactly...thanks Simon for the time and help :)
>
> Is there any mysql timeout/retry parameter that is set in db.properties
> (like db.cloud.queriesBeforeRetryMaster=5000), but I'm wondering since I'm
> using keepalived/haproxy setup, it takes up to 5-6 sec for keepalive to
> detect haproxy is down (while testing I shutdown haproxu on active node),
> and then keepalvied moves IP to another node - meaning I have 5-6sec of no
> connection between ACS mgmt servers and database/haproxy.
>
> Is this timeout configurabile on ACS/mysql connector side ?
>
> Thanks again,
> Andrija
>
> On 5 June 2015 at 15:54, Simon Weller <sw...@ena.com> wrote:
>
> > I think haproxy is a better design in my opinion. You're going to have to
> > use haproxy to balance the host agents to multiple management servers
> > anyway, so you'll already be using it. haproxy can then manage the health
> > checks to Galera rather than over complicating it with the CS Management
> > configuration.
> >
> > - Si
> >
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Friday, June 5, 2015 8:37 AM
> > To: users@cloudstack.apache.org
> > Cc: dev@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Thanks Simon - that is what I thought...
> >
> > So my question would be then, haproxy vs native ACS/mysql connector going
> > to galera1/galera2/etc...will figure out, for now we use haproxy for
> > mysql/galera loadbalancing...
> >
> >
> > THanks a lot Simon,
> > Andrija
> >
> > On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
> >
> > >
> > > Personally, I think that Gallera is always going to be a safer option,
> as
> > > it handles conflict resolution natively. Having said that, it appears
> > care
> > > has been taken in designing the ACS MGMT DB integration so that the
> > chance
> > > of conflicts is very low. Galera requires a 3 nodes minimum, so it's a
> > lot
> > > of hardware unless you've got plans to use it elsewhere in your
> > > organisation.
> > >
> > > The downside to Galera, is that it's synchronous replication, so it
> needs
> > > very low latency between nodes. That doesn't make it a good candidate
> for
> > > geographic separation between DB nodes for a DR scenario.
> > >
> > > You're understanding of the replication structure, as based on the
> design
> > > document is correct. MySQL (or Galera) handles all the replication. ACS
> > > just handles which node it's writing and reading from. In a 2 node
> native
> > > MySQL cluster, it's expected that you are setup for cross master-master
> > > replication.
> > >
> > > - Si
> > > ________________________________________
> > > From: Andrija Panic <an...@gmail.com>
> > > Sent: Friday, June 5, 2015 2:41 AM
> > > To: dev@cloudstack.apache.org
> > > Cc: users@cloudstack.apache.org
> > > Subject: Re: database high availability question vs haproxy
> > >
> > > Hi Simon,
> > >
> > > thanks for the link - actually I have already read this - but Im still
> > > seaking for some answeres :) :
> > >
> > > - real world experience with DB HA in general - is i better to use
> > > haproxy(clustered/redudant) for mysql towards Galera cluster - or
> simply
> > to
> > > reference 2 nodes (1 as master, another as slave) with native ACS DB HA
> > -
> > > silly question but anyway...
> > > - my understanding - ACS just pings and connects to master or slave
> (all
> > > replication etc, is done from my side, not from ACS) ?
> > >
> > > Thanks again and any info is greatly appreciated.
> > >
> > > Andrija
> > >
> > > On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> > >
> > > > Andrija,
> > > >
> > > > Here is the original design document, and it should give you a better
> > > idea
> > > > of what is implemented today:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> > > >
> > > > We have plans to test this in our lab soon, but just haven't got
> around
> > > to
> > > > it yet.
> > > >
> > > > - Si
> > > >
> > > > ________________________________________
> > > > From: Andrija Panic <an...@gmail.com>
> > > > Sent: Thursday, June 4, 2015 9:08 AM
> > > > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > > > Subject: Re: database high availability question vs haproxy
> > > >
> > > > Anyone :) ?
> > > >
> > > > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I would have a question on database HA feature in db.properties (
> > > > >
> > > >
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > > > )
> > > > >
> > > > > If I understand correctly, it is up to the admin to provide
> > appropriate
> > > > > mysql HA (active-active, galera, etc) and ACS management server
> will
> > > > JUST
> > > > > try to connect to slaves if the master is down ?
> > > > >
> > > > > We are running Galera, with haproxy/keepalived, and by using
> stoping
> > > > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is
> down,
> > > and
> > > > > failover IP to another host.
> > > > >
> > > > > During these 6 seconds, ACS managemnt server goes dead, because of
> > this
> > > > DB
> > > > > unavailability.
> > > > >
> > > > > So my wondering, is better to use ACS db HA feature, instead of
> > > > > loadbalancer for this specific purpose ?
> > > > > (we are also using haproxy/keepalived for management server
> > > loadbalancing
> > > > > - 2 servers in backend...)
> > > > >
> > > > > Any experience shared is really appreciated !
> > > > > --
> > > > >
> > > > > Andrija Panić
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrija Panić
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Simon Weller <sw...@ena.com>.
I'm not sure whether that's configurable and I haven't seen any examples out there that point to it being configurable. Having said that, there are people a lot more familiar with that than I am on the list.
One option you do have for this particular failure situation, is to use monit to monitor the pid of the CS management service and if it's not running, force it to start up again.
Losing a load balancer completely is very rare, so it's not likely to happen often at all.
- Si
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Friday, June 5, 2015 9:07 AM
To: users@cloudstack.apache.org
Cc: dev@cloudstack.apache.org
Subject: Re: database high availability question vs haproxy
Exactly...thanks Simon for the time and help :)
Is there any mysql timeout/retry parameter that is set in db.properties
(like db.cloud.queriesBeforeRetryMaster=5000), but I'm wondering since I'm
using keepalived/haproxy setup, it takes up to 5-6 sec for keepalive to
detect haproxy is down (while testing I shutdown haproxu on active node),
and then keepalvied moves IP to another node - meaning I have 5-6sec of no
connection between ACS mgmt servers and database/haproxy.
Is this timeout configurabile on ACS/mysql connector side ?
Thanks again,
Andrija
On 5 June 2015 at 15:54, Simon Weller <sw...@ena.com> wrote:
> I think haproxy is a better design in my opinion. You're going to have to
> use haproxy to balance the host agents to multiple management servers
> anyway, so you'll already be using it. haproxy can then manage the health
> checks to Galera rather than over complicating it with the CS Management
> configuration.
>
> - Si
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 8:37 AM
> To: users@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Thanks Simon - that is what I thought...
>
> So my question would be then, haproxy vs native ACS/mysql connector going
> to galera1/galera2/etc...will figure out, for now we use haproxy for
> mysql/galera loadbalancing...
>
>
> THanks a lot Simon,
> Andrija
>
> On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
>
> >
> > Personally, I think that Gallera is always going to be a safer option, as
> > it handles conflict resolution natively. Having said that, it appears
> care
> > has been taken in designing the ACS MGMT DB integration so that the
> chance
> > of conflicts is very low. Galera requires a 3 nodes minimum, so it's a
> lot
> > of hardware unless you've got plans to use it elsewhere in your
> > organisation.
> >
> > The downside to Galera, is that it's synchronous replication, so it needs
> > very low latency between nodes. That doesn't make it a good candidate for
> > geographic separation between DB nodes for a DR scenario.
> >
> > You're understanding of the replication structure, as based on the design
> > document is correct. MySQL (or Galera) handles all the replication. ACS
> > just handles which node it's writing and reading from. In a 2 node native
> > MySQL cluster, it's expected that you are setup for cross master-master
> > replication.
> >
> > - Si
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Friday, June 5, 2015 2:41 AM
> > To: dev@cloudstack.apache.org
> > Cc: users@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Hi Simon,
> >
> > thanks for the link - actually I have already read this - but Im still
> > seaking for some answeres :) :
> >
> > - real world experience with DB HA in general - is i better to use
> > haproxy(clustered/redudant) for mysql towards Galera cluster - or simply
> to
> > reference 2 nodes (1 as master, another as slave) with native ACS DB HA
> -
> > silly question but anyway...
> > - my understanding - ACS just pings and connects to master or slave (all
> > replication etc, is done from my side, not from ACS) ?
> >
> > Thanks again and any info is greatly appreciated.
> >
> > Andrija
> >
> > On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> >
> > > Andrija,
> > >
> > > Here is the original design document, and it should give you a better
> > idea
> > > of what is implemented today:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> > >
> > > We have plans to test this in our lab soon, but just haven't got around
> > to
> > > it yet.
> > >
> > > - Si
> > >
> > > ________________________________________
> > > From: Andrija Panic <an...@gmail.com>
> > > Sent: Thursday, June 4, 2015 9:08 AM
> > > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > > Subject: Re: database high availability question vs haproxy
> > >
> > > Anyone :) ?
> > >
> > > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I would have a question on database HA feature in db.properties (
> > > >
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > > )
> > > >
> > > > If I understand correctly, it is up to the admin to provide
> appropriate
> > > > mysql HA (active-active, galera, etc) and ACS management server will
> > > JUST
> > > > try to connect to slaves if the master is down ?
> > > >
> > > > We are running Galera, with haproxy/keepalived, and by using stoping
> > > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down,
> > and
> > > > failover IP to another host.
> > > >
> > > > During these 6 seconds, ACS managemnt server goes dead, because of
> this
> > > DB
> > > > unavailability.
> > > >
> > > > So my wondering, is better to use ACS db HA feature, instead of
> > > > loadbalancer for this specific purpose ?
> > > > (we are also using haproxy/keepalived for management server
> > loadbalancing
> > > > - 2 servers in backend...)
> > > >
> > > > Any experience shared is really appreciated !
> > > > --
> > > >
> > > > Andrija Panić
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Simon Weller <sw...@ena.com>.
I'm not sure whether that's configurable and I haven't seen any examples out there that point to it being configurable. Having said that, there are people a lot more familiar with that than I am on the list.
One option you do have for this particular failure situation, is to use monit to monitor the pid of the CS management service and if it's not running, force it to start up again.
Losing a load balancer completely is very rare, so it's not likely to happen often at all.
- Si
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Friday, June 5, 2015 9:07 AM
To: users@cloudstack.apache.org
Cc: dev@cloudstack.apache.org
Subject: Re: database high availability question vs haproxy
Exactly...thanks Simon for the time and help :)
Is there any mysql timeout/retry parameter that is set in db.properties
(like db.cloud.queriesBeforeRetryMaster=5000), but I'm wondering since I'm
using keepalived/haproxy setup, it takes up to 5-6 sec for keepalive to
detect haproxy is down (while testing I shutdown haproxu on active node),
and then keepalvied moves IP to another node - meaning I have 5-6sec of no
connection between ACS mgmt servers and database/haproxy.
Is this timeout configurabile on ACS/mysql connector side ?
Thanks again,
Andrija
On 5 June 2015 at 15:54, Simon Weller <sw...@ena.com> wrote:
> I think haproxy is a better design in my opinion. You're going to have to
> use haproxy to balance the host agents to multiple management servers
> anyway, so you'll already be using it. haproxy can then manage the health
> checks to Galera rather than over complicating it with the CS Management
> configuration.
>
> - Si
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 8:37 AM
> To: users@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Thanks Simon - that is what I thought...
>
> So my question would be then, haproxy vs native ACS/mysql connector going
> to galera1/galera2/etc...will figure out, for now we use haproxy for
> mysql/galera loadbalancing...
>
>
> THanks a lot Simon,
> Andrija
>
> On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
>
> >
> > Personally, I think that Gallera is always going to be a safer option, as
> > it handles conflict resolution natively. Having said that, it appears
> care
> > has been taken in designing the ACS MGMT DB integration so that the
> chance
> > of conflicts is very low. Galera requires a 3 nodes minimum, so it's a
> lot
> > of hardware unless you've got plans to use it elsewhere in your
> > organisation.
> >
> > The downside to Galera, is that it's synchronous replication, so it needs
> > very low latency between nodes. That doesn't make it a good candidate for
> > geographic separation between DB nodes for a DR scenario.
> >
> > You're understanding of the replication structure, as based on the design
> > document is correct. MySQL (or Galera) handles all the replication. ACS
> > just handles which node it's writing and reading from. In a 2 node native
> > MySQL cluster, it's expected that you are setup for cross master-master
> > replication.
> >
> > - Si
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Friday, June 5, 2015 2:41 AM
> > To: dev@cloudstack.apache.org
> > Cc: users@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Hi Simon,
> >
> > thanks for the link - actually I have already read this - but Im still
> > seaking for some answeres :) :
> >
> > - real world experience with DB HA in general - is i better to use
> > haproxy(clustered/redudant) for mysql towards Galera cluster - or simply
> to
> > reference 2 nodes (1 as master, another as slave) with native ACS DB HA
> -
> > silly question but anyway...
> > - my understanding - ACS just pings and connects to master or slave (all
> > replication etc, is done from my side, not from ACS) ?
> >
> > Thanks again and any info is greatly appreciated.
> >
> > Andrija
> >
> > On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> >
> > > Andrija,
> > >
> > > Here is the original design document, and it should give you a better
> > idea
> > > of what is implemented today:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> > >
> > > We have plans to test this in our lab soon, but just haven't got around
> > to
> > > it yet.
> > >
> > > - Si
> > >
> > > ________________________________________
> > > From: Andrija Panic <an...@gmail.com>
> > > Sent: Thursday, June 4, 2015 9:08 AM
> > > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > > Subject: Re: database high availability question vs haproxy
> > >
> > > Anyone :) ?
> > >
> > > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I would have a question on database HA feature in db.properties (
> > > >
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > > )
> > > >
> > > > If I understand correctly, it is up to the admin to provide
> appropriate
> > > > mysql HA (active-active, galera, etc) and ACS management server will
> > > JUST
> > > > try to connect to slaves if the master is down ?
> > > >
> > > > We are running Galera, with haproxy/keepalived, and by using stoping
> > > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down,
> > and
> > > > failover IP to another host.
> > > >
> > > > During these 6 seconds, ACS managemnt server goes dead, because of
> this
> > > DB
> > > > unavailability.
> > > >
> > > > So my wondering, is better to use ACS db HA feature, instead of
> > > > loadbalancer for this specific purpose ?
> > > > (we are also using haproxy/keepalived for management server
> > loadbalancing
> > > > - 2 servers in backend...)
> > > >
> > > > Any experience shared is really appreciated !
> > > > --
> > > >
> > > > Andrija Panić
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Andrija Panic <an...@gmail.com>.
Exactly...thanks Simon for the time and help :)
Is there any mysql timeout/retry parameter that is set in db.properties
(like db.cloud.queriesBeforeRetryMaster=5000), but I'm wondering since I'm
using keepalived/haproxy setup, it takes up to 5-6 sec for keepalive to
detect haproxy is down (while testing I shutdown haproxu on active node),
and then keepalvied moves IP to another node - meaning I have 5-6sec of no
connection between ACS mgmt servers and database/haproxy.
Is this timeout configurabile on ACS/mysql connector side ?
Thanks again,
Andrija
On 5 June 2015 at 15:54, Simon Weller <sw...@ena.com> wrote:
> I think haproxy is a better design in my opinion. You're going to have to
> use haproxy to balance the host agents to multiple management servers
> anyway, so you'll already be using it. haproxy can then manage the health
> checks to Galera rather than over complicating it with the CS Management
> configuration.
>
> - Si
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 8:37 AM
> To: users@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Thanks Simon - that is what I thought...
>
> So my question would be then, haproxy vs native ACS/mysql connector going
> to galera1/galera2/etc...will figure out, for now we use haproxy for
> mysql/galera loadbalancing...
>
>
> THanks a lot Simon,
> Andrija
>
> On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
>
> >
> > Personally, I think that Gallera is always going to be a safer option, as
> > it handles conflict resolution natively. Having said that, it appears
> care
> > has been taken in designing the ACS MGMT DB integration so that the
> chance
> > of conflicts is very low. Galera requires a 3 nodes minimum, so it's a
> lot
> > of hardware unless you've got plans to use it elsewhere in your
> > organisation.
> >
> > The downside to Galera, is that it's synchronous replication, so it needs
> > very low latency between nodes. That doesn't make it a good candidate for
> > geographic separation between DB nodes for a DR scenario.
> >
> > You're understanding of the replication structure, as based on the design
> > document is correct. MySQL (or Galera) handles all the replication. ACS
> > just handles which node it's writing and reading from. In a 2 node native
> > MySQL cluster, it's expected that you are setup for cross master-master
> > replication.
> >
> > - Si
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Friday, June 5, 2015 2:41 AM
> > To: dev@cloudstack.apache.org
> > Cc: users@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Hi Simon,
> >
> > thanks for the link - actually I have already read this - but Im still
> > seaking for some answeres :) :
> >
> > - real world experience with DB HA in general - is i better to use
> > haproxy(clustered/redudant) for mysql towards Galera cluster - or simply
> to
> > reference 2 nodes (1 as master, another as slave) with native ACS DB HA
> -
> > silly question but anyway...
> > - my understanding - ACS just pings and connects to master or slave (all
> > replication etc, is done from my side, not from ACS) ?
> >
> > Thanks again and any info is greatly appreciated.
> >
> > Andrija
> >
> > On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> >
> > > Andrija,
> > >
> > > Here is the original design document, and it should give you a better
> > idea
> > > of what is implemented today:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> > >
> > > We have plans to test this in our lab soon, but just haven't got around
> > to
> > > it yet.
> > >
> > > - Si
> > >
> > > ________________________________________
> > > From: Andrija Panic <an...@gmail.com>
> > > Sent: Thursday, June 4, 2015 9:08 AM
> > > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > > Subject: Re: database high availability question vs haproxy
> > >
> > > Anyone :) ?
> > >
> > > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I would have a question on database HA feature in db.properties (
> > > >
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > > )
> > > >
> > > > If I understand correctly, it is up to the admin to provide
> appropriate
> > > > mysql HA (active-active, galera, etc) and ACS management server will
> > > JUST
> > > > try to connect to slaves if the master is down ?
> > > >
> > > > We are running Galera, with haproxy/keepalived, and by using stoping
> > > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down,
> > and
> > > > failover IP to another host.
> > > >
> > > > During these 6 seconds, ACS managemnt server goes dead, because of
> this
> > > DB
> > > > unavailability.
> > > >
> > > > So my wondering, is better to use ACS db HA feature, instead of
> > > > loadbalancer for this specific purpose ?
> > > > (we are also using haproxy/keepalived for management server
> > loadbalancing
> > > > - 2 servers in backend...)
> > > >
> > > > Any experience shared is really appreciated !
> > > > --
> > > >
> > > > Andrija Panić
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Andrija Panic <an...@gmail.com>.
Exactly...thanks Simon for the time and help :)
Is there any mysql timeout/retry parameter that is set in db.properties
(like db.cloud.queriesBeforeRetryMaster=5000), but I'm wondering since I'm
using keepalived/haproxy setup, it takes up to 5-6 sec for keepalive to
detect haproxy is down (while testing I shutdown haproxu on active node),
and then keepalvied moves IP to another node - meaning I have 5-6sec of no
connection between ACS mgmt servers and database/haproxy.
Is this timeout configurabile on ACS/mysql connector side ?
Thanks again,
Andrija
On 5 June 2015 at 15:54, Simon Weller <sw...@ena.com> wrote:
> I think haproxy is a better design in my opinion. You're going to have to
> use haproxy to balance the host agents to multiple management servers
> anyway, so you'll already be using it. haproxy can then manage the health
> checks to Galera rather than over complicating it with the CS Management
> configuration.
>
> - Si
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 8:37 AM
> To: users@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Thanks Simon - that is what I thought...
>
> So my question would be then, haproxy vs native ACS/mysql connector going
> to galera1/galera2/etc...will figure out, for now we use haproxy for
> mysql/galera loadbalancing...
>
>
> THanks a lot Simon,
> Andrija
>
> On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
>
> >
> > Personally, I think that Gallera is always going to be a safer option, as
> > it handles conflict resolution natively. Having said that, it appears
> care
> > has been taken in designing the ACS MGMT DB integration so that the
> chance
> > of conflicts is very low. Galera requires a 3 nodes minimum, so it's a
> lot
> > of hardware unless you've got plans to use it elsewhere in your
> > organisation.
> >
> > The downside to Galera, is that it's synchronous replication, so it needs
> > very low latency between nodes. That doesn't make it a good candidate for
> > geographic separation between DB nodes for a DR scenario.
> >
> > You're understanding of the replication structure, as based on the design
> > document is correct. MySQL (or Galera) handles all the replication. ACS
> > just handles which node it's writing and reading from. In a 2 node native
> > MySQL cluster, it's expected that you are setup for cross master-master
> > replication.
> >
> > - Si
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Friday, June 5, 2015 2:41 AM
> > To: dev@cloudstack.apache.org
> > Cc: users@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Hi Simon,
> >
> > thanks for the link - actually I have already read this - but Im still
> > seaking for some answeres :) :
> >
> > - real world experience with DB HA in general - is i better to use
> > haproxy(clustered/redudant) for mysql towards Galera cluster - or simply
> to
> > reference 2 nodes (1 as master, another as slave) with native ACS DB HA
> -
> > silly question but anyway...
> > - my understanding - ACS just pings and connects to master or slave (all
> > replication etc, is done from my side, not from ACS) ?
> >
> > Thanks again and any info is greatly appreciated.
> >
> > Andrija
> >
> > On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> >
> > > Andrija,
> > >
> > > Here is the original design document, and it should give you a better
> > idea
> > > of what is implemented today:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> > >
> > > We have plans to test this in our lab soon, but just haven't got around
> > to
> > > it yet.
> > >
> > > - Si
> > >
> > > ________________________________________
> > > From: Andrija Panic <an...@gmail.com>
> > > Sent: Thursday, June 4, 2015 9:08 AM
> > > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > > Subject: Re: database high availability question vs haproxy
> > >
> > > Anyone :) ?
> > >
> > > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I would have a question on database HA feature in db.properties (
> > > >
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > > )
> > > >
> > > > If I understand correctly, it is up to the admin to provide
> appropriate
> > > > mysql HA (active-active, galera, etc) and ACS management server will
> > > JUST
> > > > try to connect to slaves if the master is down ?
> > > >
> > > > We are running Galera, with haproxy/keepalived, and by using stoping
> > > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down,
> > and
> > > > failover IP to another host.
> > > >
> > > > During these 6 seconds, ACS managemnt server goes dead, because of
> this
> > > DB
> > > > unavailability.
> > > >
> > > > So my wondering, is better to use ACS db HA feature, instead of
> > > > loadbalancer for this specific purpose ?
> > > > (we are also using haproxy/keepalived for management server
> > loadbalancing
> > > > - 2 servers in backend...)
> > > >
> > > > Any experience shared is really appreciated !
> > > > --
> > > >
> > > > Andrija Panić
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Simon Weller <sw...@ena.com>.
I think haproxy is a better design in my opinion. You're going to have to use haproxy to balance the host agents to multiple management servers anyway, so you'll already be using it. haproxy can then manage the health checks to Galera rather than over complicating it with the CS Management configuration.
- Si
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Friday, June 5, 2015 8:37 AM
To: users@cloudstack.apache.org
Cc: dev@cloudstack.apache.org
Subject: Re: database high availability question vs haproxy
Thanks Simon - that is what I thought...
So my question would be then, haproxy vs native ACS/mysql connector going
to galera1/galera2/etc...will figure out, for now we use haproxy for
mysql/galera loadbalancing...
THanks a lot Simon,
Andrija
On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
>
> Personally, I think that Gallera is always going to be a safer option, as
> it handles conflict resolution natively. Having said that, it appears care
> has been taken in designing the ACS MGMT DB integration so that the chance
> of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot
> of hardware unless you've got plans to use it elsewhere in your
> organisation.
>
> The downside to Galera, is that it's synchronous replication, so it needs
> very low latency between nodes. That doesn't make it a good candidate for
> geographic separation between DB nodes for a DR scenario.
>
> You're understanding of the replication structure, as based on the design
> document is correct. MySQL (or Galera) handles all the replication. ACS
> just handles which node it's writing and reading from. In a 2 node native
> MySQL cluster, it's expected that you are setup for cross master-master
> replication.
>
> - Si
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 2:41 AM
> To: dev@cloudstack.apache.org
> Cc: users@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Hi Simon,
>
> thanks for the link - actually I have already read this - but Im still
> seaking for some answeres :) :
>
> - real world experience with DB HA in general - is i better to use
> haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to
> reference 2 nodes (1 as master, another as slave) with native ACS DB HA -
> silly question but anyway...
> - my understanding - ACS just pings and connects to master or slave (all
> replication etc, is done from my side, not from ACS) ?
>
> Thanks again and any info is greatly appreciated.
>
> Andrija
>
> On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
>
> > Andrija,
> >
> > Here is the original design document, and it should give you a better
> idea
> > of what is implemented today:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> >
> > We have plans to test this in our lab soon, but just haven't got around
> to
> > it yet.
> >
> > - Si
> >
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Thursday, June 4, 2015 9:08 AM
> > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Anyone :) ?
> >
> > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I would have a question on database HA feature in db.properties (
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > )
> > >
> > > If I understand correctly, it is up to the admin to provide appropriate
> > > mysql HA (active-active, galera, etc) and ACS management server will
> > JUST
> > > try to connect to slaves if the master is down ?
> > >
> > > We are running Galera, with haproxy/keepalived, and by using stoping
> > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down,
> and
> > > failover IP to another host.
> > >
> > > During these 6 seconds, ACS managemnt server goes dead, because of this
> > DB
> > > unavailability.
> > >
> > > So my wondering, is better to use ACS db HA feature, instead of
> > > loadbalancer for this specific purpose ?
> > > (we are also using haproxy/keepalived for management server
> loadbalancing
> > > - 2 servers in backend...)
> > >
> > > Any experience shared is really appreciated !
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Simon Weller <sw...@ena.com>.
I think haproxy is a better design in my opinion. You're going to have to use haproxy to balance the host agents to multiple management servers anyway, so you'll already be using it. haproxy can then manage the health checks to Galera rather than over complicating it with the CS Management configuration.
- Si
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Friday, June 5, 2015 8:37 AM
To: users@cloudstack.apache.org
Cc: dev@cloudstack.apache.org
Subject: Re: database high availability question vs haproxy
Thanks Simon - that is what I thought...
So my question would be then, haproxy vs native ACS/mysql connector going
to galera1/galera2/etc...will figure out, for now we use haproxy for
mysql/galera loadbalancing...
THanks a lot Simon,
Andrija
On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
>
> Personally, I think that Gallera is always going to be a safer option, as
> it handles conflict resolution natively. Having said that, it appears care
> has been taken in designing the ACS MGMT DB integration so that the chance
> of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot
> of hardware unless you've got plans to use it elsewhere in your
> organisation.
>
> The downside to Galera, is that it's synchronous replication, so it needs
> very low latency between nodes. That doesn't make it a good candidate for
> geographic separation between DB nodes for a DR scenario.
>
> You're understanding of the replication structure, as based on the design
> document is correct. MySQL (or Galera) handles all the replication. ACS
> just handles which node it's writing and reading from. In a 2 node native
> MySQL cluster, it's expected that you are setup for cross master-master
> replication.
>
> - Si
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 2:41 AM
> To: dev@cloudstack.apache.org
> Cc: users@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Hi Simon,
>
> thanks for the link - actually I have already read this - but Im still
> seaking for some answeres :) :
>
> - real world experience with DB HA in general - is i better to use
> haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to
> reference 2 nodes (1 as master, another as slave) with native ACS DB HA -
> silly question but anyway...
> - my understanding - ACS just pings and connects to master or slave (all
> replication etc, is done from my side, not from ACS) ?
>
> Thanks again and any info is greatly appreciated.
>
> Andrija
>
> On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
>
> > Andrija,
> >
> > Here is the original design document, and it should give you a better
> idea
> > of what is implemented today:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> >
> > We have plans to test this in our lab soon, but just haven't got around
> to
> > it yet.
> >
> > - Si
> >
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Thursday, June 4, 2015 9:08 AM
> > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Anyone :) ?
> >
> > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I would have a question on database HA feature in db.properties (
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > )
> > >
> > > If I understand correctly, it is up to the admin to provide appropriate
> > > mysql HA (active-active, galera, etc) and ACS management server will
> > JUST
> > > try to connect to slaves if the master is down ?
> > >
> > > We are running Galera, with haproxy/keepalived, and by using stoping
> > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down,
> and
> > > failover IP to another host.
> > >
> > > During these 6 seconds, ACS managemnt server goes dead, because of this
> > DB
> > > unavailability.
> > >
> > > So my wondering, is better to use ACS db HA feature, instead of
> > > loadbalancer for this specific purpose ?
> > > (we are also using haproxy/keepalived for management server
> loadbalancing
> > > - 2 servers in backend...)
> > >
> > > Any experience shared is really appreciated !
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Andrija Panic <an...@gmail.com>.
Thanks Simon - that is what I thought...
So my question would be then, haproxy vs native ACS/mysql connector going
to galera1/galera2/etc...will figure out, for now we use haproxy for
mysql/galera loadbalancing...
THanks a lot Simon,
Andrija
On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
>
> Personally, I think that Gallera is always going to be a safer option, as
> it handles conflict resolution natively. Having said that, it appears care
> has been taken in designing the ACS MGMT DB integration so that the chance
> of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot
> of hardware unless you've got plans to use it elsewhere in your
> organisation.
>
> The downside to Galera, is that it's synchronous replication, so it needs
> very low latency between nodes. That doesn't make it a good candidate for
> geographic separation between DB nodes for a DR scenario.
>
> You're understanding of the replication structure, as based on the design
> document is correct. MySQL (or Galera) handles all the replication. ACS
> just handles which node it's writing and reading from. In a 2 node native
> MySQL cluster, it's expected that you are setup for cross master-master
> replication.
>
> - Si
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 2:41 AM
> To: dev@cloudstack.apache.org
> Cc: users@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Hi Simon,
>
> thanks for the link - actually I have already read this - but Im still
> seaking for some answeres :) :
>
> - real world experience with DB HA in general - is i better to use
> haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to
> reference 2 nodes (1 as master, another as slave) with native ACS DB HA -
> silly question but anyway...
> - my understanding - ACS just pings and connects to master or slave (all
> replication etc, is done from my side, not from ACS) ?
>
> Thanks again and any info is greatly appreciated.
>
> Andrija
>
> On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
>
> > Andrija,
> >
> > Here is the original design document, and it should give you a better
> idea
> > of what is implemented today:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> >
> > We have plans to test this in our lab soon, but just haven't got around
> to
> > it yet.
> >
> > - Si
> >
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Thursday, June 4, 2015 9:08 AM
> > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Anyone :) ?
> >
> > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I would have a question on database HA feature in db.properties (
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > )
> > >
> > > If I understand correctly, it is up to the admin to provide appropriate
> > > mysql HA (active-active, galera, etc) and ACS management server will
> > JUST
> > > try to connect to slaves if the master is down ?
> > >
> > > We are running Galera, with haproxy/keepalived, and by using stoping
> > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down,
> and
> > > failover IP to another host.
> > >
> > > During these 6 seconds, ACS managemnt server goes dead, because of this
> > DB
> > > unavailability.
> > >
> > > So my wondering, is better to use ACS db HA feature, instead of
> > > loadbalancer for this specific purpose ?
> > > (we are also using haproxy/keepalived for management server
> loadbalancing
> > > - 2 servers in backend...)
> > >
> > > Any experience shared is really appreciated !
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Andrija Panic <an...@gmail.com>.
Thanks Simon - that is what I thought...
So my question would be then, haproxy vs native ACS/mysql connector going
to galera1/galera2/etc...will figure out, for now we use haproxy for
mysql/galera loadbalancing...
THanks a lot Simon,
Andrija
On 5 June 2015 at 15:18, Simon Weller <sw...@ena.com> wrote:
>
> Personally, I think that Gallera is always going to be a safer option, as
> it handles conflict resolution natively. Having said that, it appears care
> has been taken in designing the ACS MGMT DB integration so that the chance
> of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot
> of hardware unless you've got plans to use it elsewhere in your
> organisation.
>
> The downside to Galera, is that it's synchronous replication, so it needs
> very low latency between nodes. That doesn't make it a good candidate for
> geographic separation between DB nodes for a DR scenario.
>
> You're understanding of the replication structure, as based on the design
> document is correct. MySQL (or Galera) handles all the replication. ACS
> just handles which node it's writing and reading from. In a 2 node native
> MySQL cluster, it's expected that you are setup for cross master-master
> replication.
>
> - Si
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, June 5, 2015 2:41 AM
> To: dev@cloudstack.apache.org
> Cc: users@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Hi Simon,
>
> thanks for the link - actually I have already read this - but Im still
> seaking for some answeres :) :
>
> - real world experience with DB HA in general - is i better to use
> haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to
> reference 2 nodes (1 as master, another as slave) with native ACS DB HA -
> silly question but anyway...
> - my understanding - ACS just pings and connects to master or slave (all
> replication etc, is done from my side, not from ACS) ?
>
> Thanks again and any info is greatly appreciated.
>
> Andrija
>
> On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
>
> > Andrija,
> >
> > Here is the original design document, and it should give you a better
> idea
> > of what is implemented today:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
> >
> > We have plans to test this in our lab soon, but just haven't got around
> to
> > it yet.
> >
> > - Si
> >
> > ________________________________________
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Thursday, June 4, 2015 9:08 AM
> > To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> > Subject: Re: database high availability question vs haproxy
> >
> > Anyone :) ?
> >
> > On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I would have a question on database HA feature in db.properties (
> > >
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > > )
> > >
> > > If I understand correctly, it is up to the admin to provide appropriate
> > > mysql HA (active-active, galera, etc) and ACS management server will
> > JUST
> > > try to connect to slaves if the master is down ?
> > >
> > > We are running Galera, with haproxy/keepalived, and by using stoping
> > > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down,
> and
> > > failover IP to another host.
> > >
> > > During these 6 seconds, ACS managemnt server goes dead, because of this
> > DB
> > > unavailability.
> > >
> > > So my wondering, is better to use ACS db HA feature, instead of
> > > loadbalancer for this specific purpose ?
> > > (we are also using haproxy/keepalived for management server
> loadbalancing
> > > - 2 servers in backend...)
> > >
> > > Any experience shared is really appreciated !
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Simon Weller <sw...@ena.com>.
Personally, I think that Gallera is always going to be a safer option, as it handles conflict resolution natively. Having said that, it appears care has been taken in designing the ACS MGMT DB integration so that the chance of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot of hardware unless you've got plans to use it elsewhere in your organisation.
The downside to Galera, is that it's synchronous replication, so it needs very low latency between nodes. That doesn't make it a good candidate for geographic separation between DB nodes for a DR scenario.
You're understanding of the replication structure, as based on the design document is correct. MySQL (or Galera) handles all the replication. ACS just handles which node it's writing and reading from. In a 2 node native MySQL cluster, it's expected that you are setup for cross master-master replication.
- Si
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Friday, June 5, 2015 2:41 AM
To: dev@cloudstack.apache.org
Cc: users@cloudstack.apache.org
Subject: Re: database high availability question vs haproxy
Hi Simon,
thanks for the link - actually I have already read this - but Im still
seaking for some answeres :) :
- real world experience with DB HA in general - is i better to use
haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to
reference 2 nodes (1 as master, another as slave) with native ACS DB HA -
silly question but anyway...
- my understanding - ACS just pings and connects to master or slave (all
replication etc, is done from my side, not from ACS) ?
Thanks again and any info is greatly appreciated.
Andrija
On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> Andrija,
>
> Here is the original design document, and it should give you a better idea
> of what is implemented today:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
>
> We have plans to test this in our lab soon, but just haven't got around to
> it yet.
>
> - Si
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, June 4, 2015 9:08 AM
> To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Anyone :) ?
>
> On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
>
> > Hi,
> >
> > I would have a question on database HA feature in db.properties (
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > )
> >
> > If I understand correctly, it is up to the admin to provide appropriate
> > mysql HA (active-active, galera, etc) and ACS management server will
> JUST
> > try to connect to slaves if the master is down ?
> >
> > We are running Galera, with haproxy/keepalived, and by using stoping
> > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and
> > failover IP to another host.
> >
> > During these 6 seconds, ACS managemnt server goes dead, because of this
> DB
> > unavailability.
> >
> > So my wondering, is better to use ACS db HA feature, instead of
> > loadbalancer for this specific purpose ?
> > (we are also using haproxy/keepalived for management server loadbalancing
> > - 2 servers in backend...)
> >
> > Any experience shared is really appreciated !
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Simon Weller <sw...@ena.com>.
Personally, I think that Gallera is always going to be a safer option, as it handles conflict resolution natively. Having said that, it appears care has been taken in designing the ACS MGMT DB integration so that the chance of conflicts is very low. Galera requires a 3 nodes minimum, so it's a lot of hardware unless you've got plans to use it elsewhere in your organisation.
The downside to Galera, is that it's synchronous replication, so it needs very low latency between nodes. That doesn't make it a good candidate for geographic separation between DB nodes for a DR scenario.
You're understanding of the replication structure, as based on the design document is correct. MySQL (or Galera) handles all the replication. ACS just handles which node it's writing and reading from. In a 2 node native MySQL cluster, it's expected that you are setup for cross master-master replication.
- Si
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Friday, June 5, 2015 2:41 AM
To: dev@cloudstack.apache.org
Cc: users@cloudstack.apache.org
Subject: Re: database high availability question vs haproxy
Hi Simon,
thanks for the link - actually I have already read this - but Im still
seaking for some answeres :) :
- real world experience with DB HA in general - is i better to use
haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to
reference 2 nodes (1 as master, another as slave) with native ACS DB HA -
silly question but anyway...
- my understanding - ACS just pings and connects to master or slave (all
replication etc, is done from my side, not from ACS) ?
Thanks again and any info is greatly appreciated.
Andrija
On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> Andrija,
>
> Here is the original design document, and it should give you a better idea
> of what is implemented today:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
>
> We have plans to test this in our lab soon, but just haven't got around to
> it yet.
>
> - Si
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, June 4, 2015 9:08 AM
> To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Anyone :) ?
>
> On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
>
> > Hi,
> >
> > I would have a question on database HA feature in db.properties (
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > )
> >
> > If I understand correctly, it is up to the admin to provide appropriate
> > mysql HA (active-active, galera, etc) and ACS management server will
> JUST
> > try to connect to slaves if the master is down ?
> >
> > We are running Galera, with haproxy/keepalived, and by using stoping
> > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and
> > failover IP to another host.
> >
> > During these 6 seconds, ACS managemnt server goes dead, because of this
> DB
> > unavailability.
> >
> > So my wondering, is better to use ACS db HA feature, instead of
> > loadbalancer for this specific purpose ?
> > (we are also using haproxy/keepalived for management server loadbalancing
> > - 2 servers in backend...)
> >
> > Any experience shared is really appreciated !
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Andrija Panic <an...@gmail.com>.
Hi Simon,
thanks for the link - actually I have already read this - but Im still
seaking for some answeres :) :
- real world experience with DB HA in general - is i better to use
haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to
reference 2 nodes (1 as master, another as slave) with native ACS DB HA -
silly question but anyway...
- my understanding - ACS just pings and connects to master or slave (all
replication etc, is done from my side, not from ACS) ?
Thanks again and any info is greatly appreciated.
Andrija
On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> Andrija,
>
> Here is the original design document, and it should give you a better idea
> of what is implemented today:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
>
> We have plans to test this in our lab soon, but just haven't got around to
> it yet.
>
> - Si
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, June 4, 2015 9:08 AM
> To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Anyone :) ?
>
> On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
>
> > Hi,
> >
> > I would have a question on database HA feature in db.properties (
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > )
> >
> > If I understand correctly, it is up to the admin to provide appropriate
> > mysql HA (active-active, galera, etc) and ACS management server will
> JUST
> > try to connect to slaves if the master is down ?
> >
> > We are running Galera, with haproxy/keepalived, and by using stoping
> > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and
> > failover IP to another host.
> >
> > During these 6 seconds, ACS managemnt server goes dead, because of this
> DB
> > unavailability.
> >
> > So my wondering, is better to use ACS db HA feature, instead of
> > loadbalancer for this specific purpose ?
> > (we are also using haproxy/keepalived for management server loadbalancing
> > - 2 servers in backend...)
> >
> > Any experience shared is really appreciated !
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Andrija Panic <an...@gmail.com>.
Hi Simon,
thanks for the link - actually I have already read this - but Im still
seaking for some answeres :) :
- real world experience with DB HA in general - is i better to use
haproxy(clustered/redudant) for mysql towards Galera cluster - or simply to
reference 2 nodes (1 as master, another as slave) with native ACS DB HA -
silly question but anyway...
- my understanding - ACS just pings and connects to master or slave (all
replication etc, is done from my side, not from ACS) ?
Thanks again and any info is greatly appreciated.
Andrija
On 4 June 2015 at 16:23, Simon Weller <sw...@ena.com> wrote:
> Andrija,
>
> Here is the original design document, and it should give you a better idea
> of what is implemented today:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
>
> We have plans to test this in our lab soon, but just haven't got around to
> it yet.
>
> - Si
>
> ________________________________________
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, June 4, 2015 9:08 AM
> To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> Subject: Re: database high availability question vs haproxy
>
> Anyone :) ?
>
> On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
>
> > Hi,
> >
> > I would have a question on database HA feature in db.properties (
> >
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> > )
> >
> > If I understand correctly, it is up to the admin to provide appropriate
> > mysql HA (active-active, galera, etc) and ACS management server will
> JUST
> > try to connect to slaves if the master is down ?
> >
> > We are running Galera, with haproxy/keepalived, and by using stoping
> > haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and
> > failover IP to another host.
> >
> > During these 6 seconds, ACS managemnt server goes dead, because of this
> DB
> > unavailability.
> >
> > So my wondering, is better to use ACS db HA feature, instead of
> > loadbalancer for this specific purpose ?
> > (we are also using haproxy/keepalived for management server loadbalancing
> > - 2 servers in backend...)
> >
> > Any experience shared is really appreciated !
> > --
> >
> > Andrija Panić
> >
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Simon Weller <sw...@ena.com>.
Andrija,
Here is the original design document, and it should give you a better idea of what is implemented today:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
We have plans to test this in our lab soon, but just haven't got around to it yet.
- Si
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Thursday, June 4, 2015 9:08 AM
To: dev@cloudstack.apache.org; users@cloudstack.apache.org
Subject: Re: database high availability question vs haproxy
Anyone :) ?
On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
> Hi,
>
> I would have a question on database HA feature in db.properties (
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> )
>
> If I understand correctly, it is up to the admin to provide appropriate
> mysql HA (active-active, galera, etc) and ACS management server will JUST
> try to connect to slaves if the master is down ?
>
> We are running Galera, with haproxy/keepalived, and by using stoping
> haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and
> failover IP to another host.
>
> During these 6 seconds, ACS managemnt server goes dead, because of this DB
> unavailability.
>
> So my wondering, is better to use ACS db HA feature, instead of
> loadbalancer for this specific purpose ?
> (we are also using haproxy/keepalived for management server loadbalancing
> - 2 servers in backend...)
>
> Any experience shared is really appreciated !
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: database high availability question vs haproxy
Posted by Simon Weller <sw...@ena.com>.
Andrija,
Here is the original design document, and it should give you a better idea of what is implemented today:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34838207
We have plans to test this in our lab soon, but just haven't got around to it yet.
- Si
________________________________________
From: Andrija Panic <an...@gmail.com>
Sent: Thursday, June 4, 2015 9:08 AM
To: dev@cloudstack.apache.org; users@cloudstack.apache.org
Subject: Re: database high availability question vs haproxy
Anyone :) ?
On 31 May 2015 at 00:26, Andrija Panic <an...@gmail.com> wrote:
> Hi,
>
> I would have a question on database HA feature in db.properties (
> http://cloudstack-administration.readthedocs.org/en/latest/reliability.html#configuring-database-high-availability
> )
>
> If I understand correctly, it is up to the admin to provide appropriate
> mysql HA (active-active, galera, etc) and ACS management server will JUST
> try to connect to slaves if the master is down ?
>
> We are running Galera, with haproxy/keepalived, and by using stoping
> haproxy, it takes i.e. 6sec for keepalived to detect haproxy is down, and
> failover IP to another host.
>
> During these 6 seconds, ACS managemnt server goes dead, because of this DB
> unavailability.
>
> So my wondering, is better to use ACS db HA feature, instead of
> loadbalancer for this specific purpose ?
> (we are also using haproxy/keepalived for management server loadbalancing
> - 2 servers in backend...)
>
> Any experience shared is really appreciated !
> --
>
> Andrija Panić
>
--
Andrija Panić