You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Simon Murphy <si...@vifx.co.nz> on 2013/10/08 21:28:27 UTC

VM Backup plug-in framework

Hi,

Can anyone tell me if there are plans to implement a more comprehensive mechanism to backup data within guest VM's, or to provide a framework for backup vendors to plug into?

This is needed because:

1. The native volume snapshot functionality is slow and does not provide the required feature set. (Incremental forever, Dedup, file level indexing, tape support etc)
2. Integrating 3rd party tools is painful and can only be done at the guest/hypervisor layers which is outside of CCP. Restoring entire VM's is problematic and billing even more so.

What would be ideal is a plug-in framework that lets backup vendors integrate their software with CS, so that CS aware backup and restore can be scheduled and exectued by end users.

Im hoping we are not the only people out there who are struggling with this!

Many thanks,
Simon Murphy
Solutions Architect

ViFX | Cloud Infrastructure
Level 7, 57 Fort Street, Auckland, New Zealand 1010
PO Box 106700, Auckland, New Zealand 1143
M +64 21 285 4519 | S simon_a_murphy
www.vifx.co.nz<http://www.vifx.co.nz/> follow us on twitter<https://twitter.com/ViFX>
Auckland | Wellington | Christchurch

[cid:image003.jpg@01CDDF95.815BF160]

experience. expertise. execution.

This email and any files transmitted with it are confidential, without prejudice and may contain information that is subject to legal privilege. It is intended solely for the use of the individual/s to whom it is addressed in accordance with the provisions of the Privacy Act (1993). The content contained in this email does not, necessarily, reflect the official policy position of ViFX nor does ViFX have any responsibility for any alterations to the contents of this email that may occur following transmission. If you are not the addressee it may be unlawful for you to read, copy, distribute, disclose or otherwise use the information contained within this email. If you are not the intended recipient, please notify the sender prior to deleting this email message from your system. Please note ViFX reserves the right to monitor, from time to time, the communications sent to and from its email network.

Re: VM Backup plug-in framework

Posted by "SuichII, Christopher" <Ch...@netapp.com>.

Sorry it has taken me a bit to reply, Simon.

Right now, it sounds like you have a couple options.

1) There is an interface called 'SnapshotStrategy', which would allow you to write a plugin becoming the authority on how to take, backup and revert snapshots. This should give you the appropriate hooks to do some of what you are asking for. However, this is fairly new territory. One issue you might run in to is how that would potentially interfere if a storage provider created there own SnapshotStrategy to take advantage of storage level backup and recovery operations.

2) You can listen in on the CloudStack EventBus for operations that you are interested in. This would allow you to perform whatever operations you need, without tying directly into directly and becoming an authority on the B&R lifecycle.

That said, the community is open to feedback on how these options do or do not fit your needs. I think (1) would be an interesting approach, but it would currently require that you be the authority on the entire lifecycle, not just take over the backup portion.

-Chris
-- 
Chris Suich
chris.suich@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

On Oct 8, 2013, at 5:29 PM, Simon Murphy <si...@vifx.co.nz> wrote:

> Hi Chris,
> 
> Correct, I am talking about a way to orchestrate backing up and restoring
> an entire VM (or set of VM's), but using more efficient technique at the
> back end to transfer data.
> 
> Today we do the following:
> 
> - When a CS account is created we create a corresponding vSphere folder,
> and we ensure that all VM's for that account are placed in the folder.
> Access is restricted to that customer.
> - We configure Veeam to back up all VM's in the vSphere folder using a
> schedule that we have agreed with the customer. CS has no knowledge that
> these backups are being taken.
> - Veeam uses vSphere integrated backups (VADP, CBT) and performs
> incremental forever backups to our backup server
> - To restore, we give customers access the Veeam Enterprise Manager and
> they can perform file level or entire VM restores
> - CS seems to reference the VM name and not the UID so this does appear to
> work, however it hasn't been tested at scale
> 
> 
> This is less than ideal for the following reasons:
> - VM placement is prone to user error until we can write a script that
> automates this
> - Veeam backups occur without knowledge of CS
> - We can't integrate billing (we use CPBM)
> - There is a separate console for Self service restore
> - We need to implement the backup schedule on behalf of the customer which
> can be time consuming
> 
> What I feel is missing is an integrated way to provide 'enterprise grade'
> data protection for CS VM's. Our Customers expect this, and some have long
> retention requirements (up to 7 years!) so the native snapshot function
> just isn't fit for purpose.
> 
> It makes sense to me that CS would orchestrate the backup and restore
> operations, and hand off to a 3rd party system (Commvault, Veeam, snap
> manager etc) for the actual data transfer and long term storage
> 
> Thanks!
> 
> 
> Simon Murphy
> Solutions Architect
> 
> ViFX | Cloud Infrastructure
> Level 7, 57 Fort Street, Auckland, New Zealand 1010
> PO Box 106700, Auckland, New Zealand 1143
> M +64 21 285 4519 | S simon_a_murphy
> www.vifx.co.nz <http://www.vifx.co.nz/> follow us on twitter
> <https://twitter.com/ViFX>
> Auckland | Wellington | Christchurch
> 
> 
> 
> experience. expertise. execution.
> 
> This email and any files transmitted with it are confidential, without
> prejudice and may contain information that is subject to legal privilege.
> It is intended solely for the use of the individual/s to whom it is
> addressed in accordance with the provisions of the Privacy Act (1993). The
> content contained in this email does not, necessarily, reflect the
> official policy position of ViFX nor does ViFX have any responsibility for
> any alterations to the contents of this email that may occur following
> transmission. If you are not the addressee it may be unlawful for you to
> read, copy, distribute, disclose or otherwise use the information
> contained within this email. If you are not the intended recipient, please
> notify the sender prior to deleting this email message from your system.
> Please note ViFX reserves the right to monitor, from time to time, the
> communications sent to and from its email network.
> 
> 
> 
> 
> 
> 
> On 9/10/13 10:02 AM, "SuichII, Christopher" <Ch...@netapp.com> wrote:
> 
>> http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201309.mbox/%3C18A
>> 67ED7-9CB0-486A-BE80-E16152F33C00@netapp.com%3E
>> <http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201309.mbox/%3C18
>> A67ED7-9CB0-486A-BE80-E16152F33C00@netapp.com%3E<http://mail-archives.apac
>> he.org/mod_mbox/cloudstack-dev/201309.mbox/<18A67ED7-9CB0-486A-BE80-E16152
>> F33C00@netapp.com>>>
>

Re: VM Backup plug-in framework

Posted by Simon Murphy <si...@vifx.co.nz>.

Hi Chris,

Correct, I am talking about a way to orchestrate backing up and restoring
an entire VM (or set of VM's), but using more efficient technique at the
back end to transfer data.

Today we do the following:

- When a CS account is created we create a corresponding vSphere folder,
and we ensure that all VM's for that account are placed in the folder.
Access is restricted to that customer.
- We configure Veeam to back up all VM's in the vSphere folder using a
schedule that we have agreed with the customer. CS has no knowledge that
these backups are being taken.
- Veeam uses vSphere integrated backups (VADP, CBT) and performs
incremental forever backups to our backup server
- To restore, we give customers access the Veeam Enterprise Manager and
they can perform file level or entire VM restores
- CS seems to reference the VM name and not the UID so this does appear to
work, however it hasn't been tested at scale

This is less than ideal for the following reasons:
- VM placement is prone to user error until we can write a script that
automates this
- Veeam backups occur without knowledge of CS
- We can't integrate billing (we use CPBM)
- There is a separate console for Self service restore
- We need to implement the backup schedule on behalf of the customer which
can be time consuming

What I feel is missing is an integrated way to provide 'enterprise grade'
data protection for CS VM's. Our Customers expect this, and some have long
retention requirements (up to 7 years!) so the native snapshot function
just isn't fit for purpose.

It makes sense to me that CS would orchestrate the backup and restore
operations, and hand off to a 3rd party system (Commvault, Veeam, snap
manager etc) for the actual data transfer and long term storage

Thanks!

Simon Murphy
Solutions Architect

ViFX | Cloud Infrastructure
Level 7, 57 Fort Street, Auckland, New Zealand 1010
PO Box 106700, Auckland, New Zealand 1143
M +64 21 285 4519 | S simon_a_murphy
www.vifx.co.nz <http://www.vifx.co.nz/> follow us on twitter
<https://twitter.com/ViFX>
Auckland | Wellington | Christchurch

experience. expertise. execution.

This email and any files transmitted with it are confidential, without
prejudice and may contain information that is subject to legal privilege.
It is intended solely for the use of the individual/s to whom it is
addressed in accordance with the provisions of the Privacy Act (1993). The
content contained in this email does not, necessarily, reflect the
official policy position of ViFX nor does ViFX have any responsibility for
any alterations to the contents of this email that may occur following
transmission. If you are not the addressee it may be unlawful for you to
read, copy, distribute, disclose or otherwise use the information
contained within this email. If you are not the intended recipient, please
notify the sender prior to deleting this email message from your system.
Please note ViFX reserves the right to monitor, from time to time, the
communications sent to and from its email network.

On 9/10/13 10:02 AM, "SuichII, Christopher" <Ch...@netapp.com> wrote:

>http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201309.mbox/%3C18A
>67ED7-9CB0-486A-BE80-E16152F33C00@netapp.com%3E
><http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201309.mbox/%3C18
>A67ED7-9CB0-486A-BE80-E16152F33C00@netapp.com%3E<http://mail-archives.apac
>he.org/mod_mbox/cloudstack-dev/201309.mbox/<18A67ED7-9CB0-486A-BE80-E16152
>F33C00@netapp.com>>>

Re: VM Backup plug-in framework

Posted by "SuichII, Christopher" <Ch...@netapp.com>.

Hi Simon,

The answer to your question is 'kind of'. The Storage Subsystem API (https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+subsystem+2.0) allows storage providers to implement a suite of methods for the purpose of storage management, including taking and reverting snapshots of VM volumes.

However, it sounds like you aren't talking at the level of storage providers. It sounds like you would like to hook in third party software between the API call to perform a snapshot and the hypervisor actually taking the snapshot.

It also sounds like you are talking about easier scheduling of entire VM backups. This, on the other hand, does have an effort behind it. Take a look at the email thread I started here: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201309.mbox/%3C18A67ED7-9CB0-486A-BE80-E16152F33C00@netapp.com%3E<ht...@netapp.com>>

All that said, you are definitely not the only people struggling with this. Would you mind elaborating a bit on what you have in mind for non-native backup features (1) and the 3rd party tools you'd like to integrate (2)? That may help us provide a better answer than 'kind of'. =)

-Chris
--
Chris Suich
chris.suich@netapp.com<ma...@netapp.com>
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat

On Oct 8, 2013, at 3:28 PM, Simon Murphy <si...@vifx.co.nz>> wrote:

Hi,

Can anyone tell me if there are plans to implement a more comprehensive mechanism to backup data within guest VM's, or to provide a framework for backup vendors to plug into?

This is needed because:

What would be ideal is a plug-in framework that lets backup vendors integrate their software with CS, so that CS aware backup and restore can be scheduled and exectued by end users.

Im hoping we are not the only people out there who are struggling with this!

Many thanks,
Simon Murphy
Solutions Architect

experience. expertise. execution.