You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Julien Garet <ju...@inria.fr> on 2012/10/22 16:53:46 UTC

backup a cloudstack deployment

Hello, 


I'm currently deploying CloudStack for a private IaaS cloud, and I have a question about how and what I should back up. 
I've read quite a lot of forums/blogs posts, telling I should only backup the mysql database, and that's it... But what happens if my primary storage servers burn ? If my secondary storage gets corrupted ? I think I can't just tell my users : "Oh you've lost your VMs, that's so sad but I can't do anything for you..." 

I'm planning to do backup/replication of my primary storage and secondary storage volumes, so that I can restore entire volumes in case of big crashes. 


What are people doing ? 



Julien 

Re: backup a cloudstack deployment

Posted by Gavin Lee <ga...@gmail.com>.
Below is just my assumption:
Storage hardware should make sure no data lost, the solution in this
layer may be easy but expensive in CloudStack's perspective.
CloudStack provides schedule snapshot[1] is just for your convenience,
with backuped DB, you can easily restore the whole cloud. But hardware
failure may still cause some data lose in 2 scheduled snapshots or
even corruptted for incremental snapshot.  The progress of Take
snapshot and restore to VM is also time consumption.

Using DFS(eg: ceph) as primary/secondary storage is similiar,  the DFS
itself needs to keep data intact. And currently it's not fully
supported.


[1] snapshot in cloudstack is volume backup rather than vm snapshot at
this moment.


On Tue, Oct 23, 2012 at 2:22 PM, Paco Orozco <pa...@upcnet.es> wrote:
> Hi Julien,
>
> Good point! We have a private IaaS cloud and we have the same question.
> At the beginning documentation and forums recommends to backup MySQL DB
> and the secondary storage.
>
> I'm using Cloudstack and Xen servers with LVM as a primary storage.
>
> I'm doing it but I've got some not solved questions.
>
> 1) Geoff says "You should have every VM running a Scheduled Snapshot
> routine, that way if you have a disaster, you can recreate your
> architecture, and restore VMs from Snapshots / Templates". When you
> schedule this snapshot routine it will create snaps on secondary
> storage. Does this snaps count as a consumed resources for users? Must
> they will pay for it? Can users delete them?
>
> 2) I'm now testing a solution for PHD in order to do VM backups from Xen
> Server. Are you using Xen or KVM/VMware?
>
> 3) I need to protect myself from harware failures (mirroring) and
> software or human errors (backup). Are you only using mirror?
>
> Thanks
> --
> Paco Orozco (paco.orozco@upcnet.es)
> SIP: paco.orozco@upcnet.es
> Backoffice. Àrea de Serveis TIC
> UPCNet
> Edifici Vértex - Pl. Eusebi Güell, 6
> Teléfon centraleta: 93.40.11600
> GPG Key ID: 0x3EDEC0AC
>



-- 
Gavin

Re: backup a cloudstack deployment

Posted by Julien Garet <ju...@inria.fr>.

De: "Paco Orozco" <pa...@upcnet.es> 
À: cloudstack-users@incubator.apache.org 
Envoyé: Jeudi 25 Octobre 2012 11:19:54 
Objet: Re: backup a cloudstack deployment 

Hello, 

I comment between lines. 

On 25/10/12 00:22, Julien Garet wrote: 
> [...] 
>> 1) Geoff says "You should have every VM running a Scheduled Snapshot 
>> routine, that way if you have a disaster, you can recreate your 
>> architecture, and restore VMs from Snapshots / Templates". When you 
>> schedule this snapshot routine it will create snaps on secondary 
>> storage. Does this snaps count as a consumed resources for users? 
>> Must 
>> they will pay for it? Can users delete them? 
> As far as I know, a snapshot is a resource as others, it has its own limits, different from the volumes. Users can delete their own snapshots. The service offer we are planning to provide is to tell users they have all the tools to plan their own backup (via the snapshots) and are responsible to set it up for their VMs. We ensure that in case of disaster on primary storage, they will have tools to recover failed VMs. 

Yes, but I understand you were looking a way to make a backup for your 
IaaS Cloud Service. I'm worried that by system admin error can lose all 
data on primary storage. So I'd like to have a recovery point. 

I've been thinking in do scheduled snaps of users VMs, in order to be 
able to recover them in case of disaster. But this way has two problems: 

Firstly, these snapshots consumes user resources, so I must give one or 
two snapshots free. This isn't a problem after all. 

Secondly, user should be able to delete this snaps. So I can assure I 
will be able to recover all cloud in case of disaster. 

Imagine one of your admins is installing a patch or upgrading one of my 
XenServers and it executes a wrong command that deletes all primary 
storage data. It would be difficult to explain to my customers that i 
can't recover its VMs. 

The problem when using snapshots from the domain admin, is that, if you set them up within cloudstack, users can't set another schedule policy, say you put a daily schedule for that, your customers won't be able to put another daily snapshot, so you remove a feature. Perhaps you can schedule a batch with the api to snapshot manually all the VMs, but you can't be sure people are not doing a snapshot at the same time which will lead to the second snapshot not working... 


But your problem is a real one, and errors from operators can always occur. After thinking about it, I guess I will do snapshots at the array level, which I think can also be done with lvm but I don't know how easy it is to schedule and recover from it. A problem still remains, how can one prevent from general hardware failure of the raid array of primary storage ? We just hope it won't happen ? 
<blockquote>

Customers are responsible for do its own backup task. But I need to 
assure some point of recovery. I hope you can understand my point of 
view (English, as you has note, is not my native language :-) 

> [...] 
> Does anyone face this kind of snapshots problems ? 

Yes, I've got the same. The fail a lot, and sometimes they are very slow. 

Thanks 
-- 
Paco Orozco (paco.orozco@upcnet.es) 
SIP: paco.orozco@upcnet.es 
Backoffice. Àrea de Serveis TIC 
UPCNet 
Edifici Vértex - Pl. Eusebi Güell, 6 
Teléfon centraleta: 93.40.11600 
GPG Key ID: 0x3EDEC0AC 


</blockquote>


Re: backup a cloudstack deployment

Posted by Paco Orozco <pa...@upcnet.es>.
Hello,

I comment between lines.

On 25/10/12 00:22, Julien Garet wrote:
> [...]
>> 1) Geoff says "You should have every VM running a Scheduled Snapshot
>> routine, that way if you have a disaster, you can recreate your
>> architecture, and restore VMs from Snapshots / Templates". When you
>> schedule this snapshot routine it will create snaps on secondary
>> storage. Does this snaps count as a consumed resources for users?
>> Must
>> they will pay for it? Can users delete them?
> As far as I know, a snapshot is a resource as others, it has its own limits, different from the volumes. Users can delete their own snapshots. The service offer we are planning to provide is to tell users they have all the tools to plan their own backup (via the snapshots) and are responsible to set it up for their VMs. We ensure that in case of disaster on primary storage, they will have tools to recover failed VMs. 

Yes, but I understand you were looking a way to make a backup for your
IaaS Cloud Service. I'm worried that by system admin error can lose all
data on primary storage. So I'd like to have a recovery point.

I've been thinking in do scheduled snaps of users VMs, in order to be
able to recover them in case of disaster. But this way has two problems:

Firstly, these snapshots consumes user resources, so I must give one or
two snapshots free. This isn't a problem after all.

Secondly, user should be able to delete this snaps. So I can assure I
will be able to recover all cloud in case of disaster.

Imagine one of your admins is installing a patch or upgrading one of my
XenServers and it executes a wrong command that deletes all primary
storage data. It would be difficult to explain to my customers that i
can't recover its VMs.

Customers are responsible for do its own backup task. But I need to
assure some point of recovery. I hope you can understand my point of
view (English, as you has note, is not my native language :-)

> [...]
> Does anyone face this kind of snapshots problems ? 

Yes, I've got the same. The fail a lot, and sometimes they are very slow.

Thanks
-- 
Paco Orozco (paco.orozco@upcnet.es)
SIP: paco.orozco@upcnet.es
Backoffice. Àrea de Serveis TIC
UPCNet
Edifici Vértex - Pl. Eusebi Güell, 6
Teléfon centraleta: 93.40.11600
GPG Key ID: 0x3EDEC0AC


Re: backup a cloudstack deployment

Posted by Julien Garet <ju...@inria.fr>.
Hello, 

> De: "Paco Orozco" <pa...@upcnet.es>
> À: cloudstack-users@incubator.apache.org
> Envoyé: Mardi 23 Octobre 2012 08:22:29
> Objet: Re: backup a cloudstack deployment

> Hi Julien,

> Good point! We have a private IaaS cloud and we have the same
> question.
> At the beginning documentation and forums recommends to backup MySQL
> DB
> and the secondary storage.

> I'm using Cloudstack and Xen servers with LVM as a primary storage.
I'm using KVM along with Cloudstack with iSCSI and GFS2 (on Dell Equallogic) as primary storage. 
As secondary storage, I am thinking at using disk based backup appliance with inline deduplication (Quantum, EMC DataDomain, ...), has anyone tried such a setup ? Are performances good enough or do I need some kind of speedy NFS server ? 

> I'm doing it but I've got some not solved questions.

> 1) Geoff says "You should have every VM running a Scheduled Snapshot
> routine, that way if you have a disaster, you can recreate your
> architecture, and restore VMs from Snapshots / Templates". When you
> schedule this snapshot routine it will create snaps on secondary
> storage. Does this snaps count as a consumed resources for users?
> Must
> they will pay for it? Can users delete them?
As far as I know, a snapshot is a resource as others, it has its own limits, different from the volumes. Users can delete their own snapshots. The service offer we are planning to provide is to tell users they have all the tools to plan their own backup (via the snapshots) and are responsible to set it up for their VMs. We ensure that in case of disaster on primary storage, they will have tools to recover failed VMs. 

I've done some testing around snapshots (manual or scheduled) and it appears that, vm running or not, I have a lot of errors. For example, I've scheduled hourly snapshots on a VM, keeping the last 3, and I currently have : the last backup is in error, the two others are OK on the root disk, and for the data disk only the last is good. 

Does anyone face this kind of snapshots problems ? 
Another question around that, are there means to have a copy of a volume on secondary storage or only snapshots are stored on it ? 

> 2) I'm now testing a solution for PHD in order to do VM backups from
> Xen
> Server. Are you using Xen or KVM/VMware?
I did not look for kvm specific backup solutions, I have a little background with XenServer and last time I looked at it, I finished with a perl script that did vm snapshoting before moving the volume to a dedup appliance (Quantum DXi) 
> 3) I need to protect myself from harware failures (mirroring) and
> software or human errors (backup). Are you only using mirror?
For human errors, in Cloudstack, I think that the cleanup delay before removing destroyed VMs can help a lot. I'm also quite afraid of data corruption on the raid array, e.g. many disks failing at the same time because of a bad series... How long will it take to recover the whole platform if only from secondary storage ? And recover will only be on the user behalf ? 

How will Cloudstack behave if a lot of VMs on a primary storage are not available ? 

> Thanks
> --
> Paco Orozco (paco.orozco@upcnet.es)
> SIP: paco.orozco@upcnet.es
> Backoffice. Àrea de Serveis TIC
> UPCNet
> Edifici Vértex - Pl. Eusebi Güell, 6
> Teléfon centraleta: 93.40.11600
> GPG Key ID: 0x3EDEC0AC

Re: backup a cloudstack deployment

Posted by Paco Orozco <pa...@upcnet.es>.
Hi Julien,

Good point! We have a private IaaS cloud and we have the same question.
At the beginning documentation and forums recommends to backup MySQL DB
and the secondary storage.

I'm using Cloudstack and Xen servers with LVM as a primary storage.

I'm doing it but I've got some not solved questions.

1) Geoff says "You should have every VM running a Scheduled Snapshot
routine, that way if you have a disaster, you can recreate your
architecture, and restore VMs from Snapshots / Templates". When you
schedule this snapshot routine it will create snaps on secondary
storage. Does this snaps count as a consumed resources for users? Must
they will pay for it? Can users delete them?

2) I'm now testing a solution for PHD in order to do VM backups from Xen
Server. Are you using Xen or KVM/VMware?

3) I need to protect myself from harware failures (mirroring) and
software or human errors (backup). Are you only using mirror?

Thanks
-- 
Paco Orozco (paco.orozco@upcnet.es)
SIP: paco.orozco@upcnet.es
Backoffice. Àrea de Serveis TIC
UPCNet
Edifici Vértex - Pl. Eusebi Güell, 6
Teléfon centraleta: 93.40.11600
GPG Key ID: 0x3EDEC0AC


RE: backup a cloudstack deployment

Posted by Geoff Higginbottom <ge...@shapeblue.com>.
Hi Nik,

The whole point of Cloud is that the applications running on them are Cloud Aware, and should be written to cope with an element of failure.

I recommend that you do regular backups of your MySQL DBs, and also Secondary Storage.

You should have every VM running a Scheduled Snapshot routine, that way if you have a disaster, you can recreate your architecture, and restore VMs from Snapshots / Templates

Regards

Geoff


-----Original Message-----
From: Nik Martin [mailto:nik.martin@nfinausa.com]
Sent: 22 October 2012 16:05
To: cloudstack-users@incubator.apache.org
Subject: Re: backup a cloudstack deployment

On 10/22/2012 09:53 AM, Julien Garet wrote:
> Hello,
>
>
> I'm currently deploying CloudStack for a private IaaS cloud, and I have a question about how and what I should back up.
> I've read quite a lot of forums/blogs posts, telling I should only backup the mysql database, and that's it... But what happens if my primary storage servers burn ? If my secondary storage gets corrupted ? I think I can't just tell my users : "Oh you've lost your VMs, that's so sad but I can't do anything for you..."
>
> I'm planning to do backup/replication of my primary storage and secondary storage volumes, so that I can restore entire volumes in case of big crashes.
>
>
> What are people doing ?
>
>
>
> Julien
>

I depend on secondary storage and the Volume snapshotting utility to backup important VMs, but if you want more than that, you'll want to do something like backup your secondary storage to tape or a tertiary disk storage volume.  But I also run dual active-active SAN, so my vm data is already on two different SAN nodes at all times, PLUS a secondary storage backup

--
Regards,

Nik

Nik Martin
VP Business Development
Nfina Technologies, Inc.
+1.251.243.0043 x1003
Relentless Reliability

ShapeBlue provides a range of strategic and technical consulting and implementation services to help IT Service Providers and Enterprises to build a true IaaS compute cloud. ShapeBlue’s expertise, combined with CloudStack technology, allows IT Service Providers and Enterprises to deliver true, utility based, IaaS to the customer or end-user.

________________________________

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales.

Re: backup a cloudstack deployment

Posted by Nik Martin <ni...@nfinausa.com>.
On 10/22/2012 09:53 AM, Julien Garet wrote:
> Hello,
>
>
> I'm currently deploying CloudStack for a private IaaS cloud, and I have a question about how and what I should back up.
> I've read quite a lot of forums/blogs posts, telling I should only backup the mysql database, and that's it... But what happens if my primary storage servers burn ? If my secondary storage gets corrupted ? I think I can't just tell my users : "Oh you've lost your VMs, that's so sad but I can't do anything for you..."
>
> I'm planning to do backup/replication of my primary storage and secondary storage volumes, so that I can restore entire volumes in case of big crashes.
>
>
> What are people doing ?
>
>
>
> Julien
>

I depend on secondary storage and the Volume snapshotting utility to 
backup important VMs, but if you want more than that, you'll want to do 
something like backup your secondary storage to tape or a tertiary disk 
storage volume.  But I also run dual active-active SAN, so my vm data is 
already on two different SAN nodes at all times, PLUS a secondary 
storage backup

-- 
Regards,

Nik

Nik Martin
VP Business Development
Nfina Technologies, Inc.
+1.251.243.0043 x1003
Relentless Reliability