[Propose][New Feature] Storage Snapshots

[Syed Ahmed <sa...@cloudops.com>]

Design Spec:

https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API

Jira Ticket

https://issues.apache.org/jira/browse/CLOUDSTACK-9278


Hi All,

We plan to propose a new set of APIs to do snapshots on managed storage
backends like SolidFire. Snapshots on current managed storage stay on the
device which is contrary to what CloudStack calls snpshots. But taking
snapshots
on storage and keeping it there has its own advantages and we would ideally
like
to have both ways of doing snapshots. This proposal adds 4 new APIs to
create snapshots on backend storage.

What do you guys think of this feature? I would love to have some feedback.
I am working on making the design spec more concrete but wanted to have
a high level feedback first before starting to work on it.

Thanks,
-Syed

Re: [Propose][New Feature] Storage Snapshots

[Erik Weber <te...@gmail.com>]

Volume backup only works if you rely on a single volume.

Any system using LVM (or dynamic disks in Windows) that spans more than one
volume is going to have problems using Volume Snapshots.
Or cases where your application stores data in different disks, perhaps
your data and your journal are laying on different volumes but you need
them to be consistent.

And so on.

-- 
Erik

On Fri, Feb 5, 2016 at 5:57 PM, Syed Mushtaq <sy...@gmail.com>
wrote:

> Paul,
>
> When you say actual backups, how would it be different from the Volume
> Snapshots that exist currently. My understanding is that Backups end up in
> Sec Storage whereas Snapshots are just a point-in-time state of your volume
> which can be restored back correct?
>
> -Syed
>
>
> On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <pa...@shapeblue.com>
> wrote:
>
> > Hi Syed,
> >
> > As I understand it, the SolidFire plugin will export the snapshot to
> > secondary storage if the user requests a template from the snapshot or
> > wants to download the snapshot from the cloud. This is a good, pragmatic
> > approach and yes Mike the SolidFire storage is super reliable and
> snapshots
> > on SolidFire arrays take up next to no space. BUT I think that we are
> > talking about a more general purpose API, and other storage systems may
> not
> > be as awesome as Mike's. That's my concern. Also, the time to transfer
> for
> > say 1TB to move from primary to sec storage and then create a VM template
> > out of it may be too long for users.
> >
> > @Mike I don’t think 'we' use the term volume snapshot for backup, it's
> > just that users want to do backups and a volume snapshot is the only type
> > of snapshot that copies the disk elsewhere and can be scheduled.
> >
> > I'm 'pondering' the implications of enabling actual backups (through
> > recognised backup providers) and the user requirements around them
> > (particularly restoration use cases) as a separate thread of work.
> >
> >
> >
> >
> >
> > [image: ShapeBlue] <http://www.shapeblue.com>
> > Paul Angus
> > VP Technology ,  ShapeBlue
> > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > *+44 7711 418784* <+44%207711%20418784>
> > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > <pa...@cloudyangus>  |  w:
> > *www.shapeblue.com* <http://www.shapeblue.com>
> > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> > Services India LLP is a company incorporated in India and is operated
> under
> > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > company incorporated in Brasil and is operated under license from Shape
> > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
> is
> > a registered trademark.
> > This email and any attachments to it may be confidential and are intended
> > solely for the use of the individual to whom it is addressed. Any views
> or
> > opinions expressed are solely those of the author and do not necessarily
> > represent those of Shape Blue Ltd or related companies. If you are not
> the
> > intended recipient of this email, you must neither take any action based
> > upon its contents, nor copy or show it to anyone. Please contact the
> sender
> > if you believe you have received this email in error.
> >
> >
> > -----Original Message-----
> > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> > Sent: 05 February 2016 15:31
> > To: dev@cloudstack.apache.org
> > Subject: Re: [Propose][New Feature] Storage Snapshots
> >
> > I think the terminology confusion comes from AWS where they do EBS
> > snapshots backed up to S3 and CloudStack sort of followed that. And as an
> > end user who is oblivious to the internals of my provider, my expectation
> > would be something similar to what AWS as that is my biggest reference
> > point.
> >
> > To your point Mike, I agree that a Primary Storage failure on SolidFire
> is
> > unlikely, there are other motivations for us to push data to secondary
> > storage. Primary storage (atleast for us) costs around 3 times as much as
> > secondary storage and snapshots on primary storage are rarely used
> > (especially for some of our customers who do daily backups).
> >
> >
> > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com> wrote:
> >
> > > Some of the weirdness is around terminology.
> > >
> > > For most systems I've worked on, a snapshot and a backup are two
> > > completely different things (but CloudStack has traditionally used the
> > > term "volume snapshot" to mean backup).
> > >
> > > I will put in a SolidFire "plug" here and say, though, that if your
> > > primary storage is running on SolidFire that it is unlikely you'll
> > > encounter an issue where your primary storage goes offline (and you'll
> > > even maintain your performance guarantees during failure scenarios and
> > > upgrades, as well). That being the case, it is less useful to require
> > > a backup to Swift (but it's perfectly OK if that's what we want to do
> > here).
> > >
> > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq <sy...@gmail.com>
> > > wrote:
> > >
> > > > Paul,
> > > >
> > > > I believe with the current implementation of Snapshots on managed
> > > > storage
> > > > (SolidFire) the snapshots are never exported to the secondary
> storage.
> > > > While this solves the problem of having snapshots taking forever to
> > > > get to sec storage, this leaves us with a
> > > huge
> > > > liability if our primary storage goes down. We see snapshots as our
> > > > recovery path because we store them in Swift which is reliable and
> > > > resilient to failures.
> > > >
> > > > With Storage snpashots our goal is to have Volume snapshots always
> > > > backed up to secondary storage and Storage Snapshots stay on the
> > > > primary
> > > storage.
> > > > A provider could potentially mix both these and solve the problem
> > > > that you mentioned where you want to meet user's expectation of a
> > > > snapshot (ie backup to sec storage) while having an ability to
> > > > utilize faster sanpshots (i.e. on the device)
> > > >
> > > > Hope this clarifies things.
> > > >
> > > > Thanks,
> > > > -Syed
> > > >
> > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > <pa...@shapeblue.com>
> > > > wrote:
> > > >
> > > > > HI guys,
> > > > >
> > > > > Could someone point me to the Jira bug of FS for the SAN-snapshot
> > > feature
> > > > > in 4.6 which is mentioned.
> > > > >
> > > > > From my discussions with users and operators around snapshots I'd
> > > > > make
> > > > the
> > > > > following observations:
> > > > > a. 'users' use snapshots as backups (both long-term and short
> > > > > term)
> > > with
> > > > > the expectation that they can use them for recovery if required.
> > > > > b. operators fall back to snapshots if something has gone wrong
> > > > > with primary storage.
> > > > > c. users sometimes want to be able to export snapshots as well as
> > > create
> > > > > new VMs from their snapshots
> > > > > d. snapshots are a currently a massive pain for operators, I know
> > > > > at
> > > > least
> > > > > one public cloud who have snapshots which take 2 days to complete.
> > > > > e. snapshots (as they are) can't be used for multiple LVM disks.
> > > > >
> > > > > I think the process Mike has used in the SolidFire plugin (only
> > > > > moving
> > > > the
> > > > > disk image to secondary storage when you absolutely have to) is a
> > > > > very
> > > > good
> > > > > and pragmatic solution. I wonder what problems an operator might
> > > > experience
> > > > > if they have an issue with a given primary storage pool in a
> cluster.
> > > (I
> > > > > know that that is REALLY unlikely in the SolidFire case Mike :) )
> > > > > And
> > > if
> > > > > the transfer from primary to secondary is slow, the time to being
> > > > > able
> > > to
> > > > > create a template or export the volume will be slow.
> > > > >
> > > > > So for me the issue is around making sure that the end users
> > > expectations
> > > > > are met (while improving the speed/efficiency of the back end)
> > > > >
> > > > >
> > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > > Technology , ShapeBlue
> > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > > <pa...@cloudyangus> | w:
> > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue
> > > > > Ltd is a company incorporated in England & Wales. ShapeBlue
> > > > > Services India LLP is a company incorporated in India and is
> > > > > operated
> > > > under
> > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
> > > > > a company incorporated in Brasil and is operated under license
> > > > > from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> > > > > by The Republic
> > > of
> > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > ShapeBlue
> > > > is
> > > > > a registered trademark.
> > > > > This email and any attachments to it may be confidential and are
> > > intended
> > > > > solely for the use of the individual to whom it is addressed. Any
> > > > > views
> > > > or
> > > > > opinions expressed are solely those of the author and do not
> > > necessarily
> > > > > represent those of Shape Blue Ltd or related companies. If you are
> > > > > not
> > > > the
> > > > > intended recipient of this email, you must neither take any action
> > > based
> > > > > upon its contents, nor copy or show it to anyone. Please contact
> > > > > the
> > > > sender
> > > > > if you believe you have received this email in error.
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > >
> > > > > Hi Mike,
> > > > >
> > > > > The idea of introducing a new API: StorageSnapshot for managed
> > > > > storage
> > > is
> > > > > because the VolumeSnapshot default, or expected, behavior is to
> > > > > archive snapshots into the Secondary Storage. So a StorageSnapshot
> > > > > API would be
> > > > for
> > > > > snapshot that remain on the managed storage appliance.
> > > > >
> > > > > Quickly looking at the API doc and I don't see a strong
> > > > > requirement for volume snapshots to be moved into secondary
> > > > > storage. So, maybe StorageSnapshot API is not useful, but both use
> > > > > cases are required. A snapshot that remain on the managed storage,
> > > > > and another type of
> > > snapshot
> > > > > that end up into the secondary storage. Since you've done a lot of
> > > work,
> > > > > might easier to just add a parameter to the current snapshot API
> > > > > that
> > > > would
> > > > > trigger an extraction of the storage snapshot into the secondary
> > > storage?
> > > > >
> > > > > PL
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > I think that all sounds reasonable then - thanks!
> > > > > >
> > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > syed1.mushtaq@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> You are correct Mike in terms of the requirements. One of our
> > > earlier
> > > > > >> iterations on this was to have an argument to the create
> > > > > >> snapshot
> > > API
> > > > > >> which decides whether to backup the volume to sec storage but
> > > > > >> we realized it would make management of snapshots quite messy
> > > > > >> so we proposed a new api instead.
> > > > > >>
> > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > >> <mi...@solidfire.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> Just to make sure I understand all the requirements here:
> > > > > >>>
> > > > > >>> 1) This relates only to managed storage (1:1 mapping between a
> > > > > >>> virtual disk and a backend SAN volume).
> > > > > >>>
> > > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > > >>> functionality, which creates a snapshot on the SAN, and extend
> > > > > >>> it via a config option (or
> > > > > >>> something) to not only take the SAN snapshot, but to copy the
> > > > > >>> underlying VHD (XenServer only) to NFS.
> > > > > >>>
> > > > > >>> 3) The SAN snapshot is always taken. It's the backup to NFS
> > > > > >>> that is optional.
> > > > > >>>
> > > > > >>> 4) Templates can be created from the snapshot that's on the
> > > > > >>> SAN (already works).
> > > > > >>>
> > > > > >>> 5) CloudStack volumes can be created from the snapshot that's
> > > > > >>> on
> > > the
> > > > > >>> SAN (already works as long as the new CloudStack volume ends
> > > > > >>> up on the same primary storage).
> > > > > >>>
> > > > > >>> Would we have a need for a storage snapshot API then or would
> > > > > >>> that just be the standard volume snapshot without the backup to
> > NFS?
> > > > > >>>
> > > > > >>> Thanks!
> > > > > >>> Mike
> > > > > >>>
> > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > >>> <sy...@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Is it possible to have both functionalities (snapshot on SAN
> > > > > >>>> & Sec
> > > > > >>>> Storage) coexist? Because Ideally, we would like to have both.
> > > > > >>>> For example, some of our customers want to implement their
> > > > > >>>> own backup strategies and do encryption to their backups
> > > > > >>>> which is a perfect use case for Storage Snapshot while our
> > > > > >>>> other customers will still keep using the standard volume
> > snapshot.
> > > > > >>>>
> > > > > >>>> To keep things backward compatible, we can add a setting
> > > > > >>>> which
> > > says
> > > > > >>>> to not upload on secondary storage, because, after all, you
> > > > > >>>> would take a SAN snapshot first when doing a Volume Snapshot.
> > > > > >>>> You could stop the process there and not do the upload.
> > > > > >>>>
> > > > > >>>> What do you think about this approach?
> > > > > >>>>
> > > > > >>>> Thanks,
> > > > > >>>> -Syed
> > > > > >>>>
> > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > >>>> mike.tutkowski@solidfire.com> wrote:
> > > > > >>>>
> > > > > >>>>> So, this is just me thinking out load here, but if a given
> > > > > >>>>> CloudStack cloud doesn't actually need to provide both the
> > > ability
> > > > > >>>>> to take a SAN snapshot and export it to NFS (if just taking
> > > > > >>>>> a SAN snapshot is OK), then we might be able to get away
> > > > > >>>>> with no new
> > > API
> > > > > >>>>> calls and simply implement a new custom snapshot strategy
> > > > > >>>>> and
> > > data
> > > > > >>>>> motion strategy to handle the case where the CloudStack
> > > > > >>>>> cloud
> > > does
> > > > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > > > >>>>>
> > > > > >>>>> In other words, the "default" behavior would be to use the
> > > > > >>>>> snapshot strategy and data motion strategy that we already
> > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > >>>>>
> > > > > >>>>> If your CloudStack cloud, however, wants to take a SAN
> > > > > >>>>> snapshot and have the data exported to NFS, then we could
> > > > > >>>>> have you manipulate a Swing config file to make use of a new
> > > > > >>>>> snapshot strategy and data motion strategy that performs
> > > > > >>>>> both of these
> > > > > activities.
> > > > > >>>>>
> > > > > >>>>> This way, the old behavior is still the default for users,
> > > > > >>>>> but CloudStack admins can change this behavior via
> > configuration.
> > > > > >>>>>
> > > > > >>>>> Thoughts?
> > > > > >>>>>
> > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > > >>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > >>>>>
> > > > > >>>>>> Right...I think we will need to come up with a viable
> > > > > >>>>>> upgrade path or some reasonable way for them to move from
> > > > > >>>>>> the old way to the new way (and some obvious way that they
> > > > > >>>>>> will know they need
> > > to
> > > > > do this).
> > > > > >>>>>>
> > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> I'm not really sure about the upgrade path however,
> > > > > >>>>>>> customers who are using 4.6 and are on a managed storage
> > > > > >>>>>>> would no longer have the same functionality with Volume
> > Snapshots.
> > > > > >>>>>>>
> > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> So if I understand correctly, currently taking a Volume
> > > > > >>>>>>>> Snapshots of a volume on a managed storage keeps it on
> > > > > >>>>>>>> the storage array. As a part of this feature, we can make
> > > > > >>>>>>>> sure
> > > that
> > > > > >>>>>>>> Volume Snapshots on managed storage are uploaded to the
> > > > > >>>>>>>> secondary storage. This would make the Volume Snapshot
> > > > > >>>>>>>> feature behave the same regardless of the storage
> > > > > >>>>>>>> (managed or
> > > > > >>>>>>>> non-managed) And, for utilizing the efficient backend
> > > > > >>>>>>>> storage
> > > > > capabilities, we can use the new storage snapshots API.
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > > > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Hi everyone,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Whatever we do here, we need to have a plan to deal with
> > > > > >>>>>>>>> the fact that we already have a feature (in 4.6 and
> > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > >>>>>>>>> volume-snapshot APIs to create a volume snapshot (for
> > > > > >>>>>>>>> managed
> > > > > >>>>>>>>> storage) that resides on a backend SAN (using a custom
> > > > > >>>>>>>>> snapshot strategy and a custom data motion strategy).
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> If these new APIs go in, then how should the original
> > > > > >>>>>>>>> implementation (present in 4.6 and later) be changed? If
> > > > > >>>>>>>>> it
> > > is
> > > > > >>>>>>>>> changed, how do we support customers who are already
> > > > > >>>>>>>>> using
> > > the
> > > > > >>>>>>>>> original volume-snapshot API to take snapshots on a
> > > > > >>>>>>>>> backend
> > > > SAN?
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks,
> > > > > >>>>>>>>> Mike
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > > > >>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Will you be able to create a Template from a
> > > StorageSnapshot?
> > > > > >>>>>>>>>> If yes, will the template be stored in the secondary
> > > > > >>>>>>>>>> storage like normal templates or will that be handled
> > > > > >>>>>>>>>> somehow on the
> > > > > vendor side?
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> *Will STEVENS*
> > > > > >>>>>>>>>> Lead Developer
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > > > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > > > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira
> > > > > >>>>>>>>>>>> ticket because it was not clear in the 'mention'
> > > > > >>>>>>>>>>>> section because it shows as a page 'you do not have
> > permission to'.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > >>>>>>>>>>>> Lead Developer
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > >>>>>>>>>>>> > wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Design Spec:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-927
> > > > > >>>>>>>>>>>>> 8
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Hi All,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots
> > > > > >>>>>>>>>>>>> on managed storage backends like SolidFire.
> > > > > >>>>>>>>>>>>> Snapshots on current managed storage stay on the
> > > > > >>>>>>>>>>>>> device which is contrary to what CloudStack calls
> > snpshots.
> > > > > >>>>>>>>>>>>> But taking snapshots on storage and keeping it there
> > > > > >>>>>>>>>>>>> has its own advantages
> > > and
> > > > > >>>>>>>>>>>>> we would ideally like to have both ways of doing
> > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
> > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> What do you guys think of this feature? I would love
> > > > > >>>>>>>>>>>>> to have some feedback. I am working on making the
> > > > > >>>>>>>>>>>>> design
> > > spec
> > > > > >>>>>>>>>>>>> more concrete but wanted to have a high level
> > > > > >>>>>>>>>>>>> feedback first before starting to work on it.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>> -Syed
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> --
> > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > > > >>>>>>>>> o: 303.746.7302
> > > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > > >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> --
> > > > > >>>>>> *Mike Tutkowski*
> > > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > > > >>>>>> o: 303.746.7302
> > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> --
> > > > > >>>>> *Mike Tutkowski*
> > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > > >>>>> o: 303.746.7302
> > > > > >>>>> Advancing the way the world uses the cloud
> > > > > >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> *Mike Tutkowski*
> > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >>> e: mike.tutkowski@solidfire.com
> > > > > >>> o: 303.746.7302
> > > > > >>> Advancing the way the world uses the cloud
> > > > > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >>>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > e: mike.tutkowski@solidfire.com
> > > > > > o: 303.746.7302
> > > > > > Advancing the way the world uses the cloud
> > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >
> > > > > Find out more about ShapeBlue and our range of CloudStack related
> > > > services:
> > > > > IaaS Cloud Design & Build
> > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > > > > rapid IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > CloudStack Consulting
> > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > CloudStack
> > > > > Software Engineering
> > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > CloudStack Infrastructure Support
> > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > > CloudStack Bootcamp Training Courses
> > > > > <http://shapeblue.com/cloudstack-training/>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> > Find out more about ShapeBlue and our range of CloudStack related
> services:
> > IaaS Cloud Design & Build
> > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > IaaS deployment framework <http://shapeblue.com/csforge/>
> > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack
> > Software Engineering
> > <http://shapeblue.com/cloudstack-software-engineering/>
> > CloudStack Infrastructure Support
> > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
> >
>

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

That is what I was going to suggest Mike. Add a global setting to be
backwards compatible and add the StorageSnapshot API for doing SAN
snapshots (while the VolumeSnapshots uploads to Sec storage)

On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> Now that I re-read your e-mail, it dawned on me: The end-user doesn't care
> where the snapshot is.
>
> If that's true, then we should perhaps control this via Global Settings or
> something.
>
> On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
> > It's not ideal - true, but it does allow us to be backward compatible.
> >
> > If you have other ideas, though, about how to maintain backward
> > compatibility, I'm definitely open to hear them.
> >
> > Thanks!
> >
> > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <sy...@gmail.com>
> > wrote:
> >
> >> Hi Mike,
> >>
> >> Adding a flag to createSnapshot was the first and the most obvious thing
> >> that came to our minds. The problem that I had with this was that:
> >>
> >> 1) I feel it is exposing something to the end user that is internal to
> the
> >> cloud.
> >>
> >> 2) We have to follow two different ways of restore/deletion in the same
> >> code path depending on where the Snapshot resides which I find kind of a
> >> bad design.
> >>
> >> But if exposing a archive flag to the end user is acceptable then we can
> >> definitely use this instead of adding the StorageSnapshot API
> >>
> >> Thanks,
> >> -Syed
> >>
> >>
> >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> >> mike.tutkowski@solidfire.com
> >> > wrote:
> >>
> >> > Hi Pierre-Luc,
> >> >
> >> > My recommendation would be this:
> >> >
> >> > Add an "archive" flag to the current volume-snapshot API. Its default
> >> would
> >> > be "false" because that would be backward compatible with how 4.6 has
> >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6, 4.7,
> and
> >> > 4.8).
> >> >
> >> > If you set archive=true, then we would perform a background migration
> of
> >> > the snapshot from the SAN to the secondary storage (then delete the
> SAN
> >> > snapshot).
> >> >
> >> > That archive parameter would only be valid for managed storage.
> >> >
> >> > Sound reasonable?
> >> >
> >> > Also, a VM snapshot that includes disks provided by managed storage
> >> should
> >> > work.
> >> >
> >> > Talk to you later,
> >> > Mike
> >> >
> >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <pd...@cloudops.com>
> >> > wrote:
> >> >
> >> > > Mike,
> >> > >
> >> > > In terms of API's, would you prefer introducing a parameter to the
> >> > existing
> >> > > VolumeSnapshot, example:   extract={true|false}  with a default
> value
> >> of
> >> > > true  which would extract snapshot into the secondary storage, which
> >> is
> >> > the
> >> > > current default behavior. Then for SAN snapshot that remain on the
> >> SAN we
> >> > > would just set "extract=false" ?  as oppose to create a new
> >> > >  StorageSnapshot API ?
> >> > >
> >> > >
> >> > > Paul,
> >> > >
> >> > > From what I'm seeing so far, we can't do a VM-snapshot when using
> >> managed
> >> > > storage for VM having more than one Volume. For the reason that
> >> snapshot
> >> > > are performed outside of the hypervisor awareness and
> asynchronously.
> >> If
> >> > > someone have a way to address that, it would make thinks much more
> >> > > attractive.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com> wrote:
> >> > >
> >> > > > I think a service provider backup scenario is more likely to take
> >> > > advantage
> >> > > > of SAN snapshot. There are a few reasons for this. Traditional
> >> backups
> >> > > > involve access to the file system, and there is an expectation
> that
> >> > this
> >> > > > can be done with reasonably short time frames without negatively
> >> > > impacting
> >> > > > VM performance, and that the backup orchestrator can apply various
> >> > logic
> >> > > > and or transformations to the data (compress, encrypt, deltas
> >> etc...).
> >> > > > While it is true that one could apply a backup process to a cloud
> >> > > snapshot,
> >> > > > this would be slow and inefficient requiring the data to be moved
> >> > several
> >> > > > times and there are some major bottlenecks with cloud snapshots.
> >> With a
> >> > > > cloud snapshot - there seems to be no reasonable expectation of
> >> being
> >> > > able
> >> > > > to do differential snapshots (I think this depends on the
> >> hypervisor)
> >> > and
> >> > > > if you do differential snapshots this will make file backups from
> >> those
> >> > > > snapshots even more complicated to orchestrate.
> >> > > >
> >> > > > Suspect there needs to be a different thread on how to better
> enable
> >> > > > backups, as a service. As per Paul's suggestion, but it is a
> related
> >> > > > workflow so relevant to this discussion.
> >> > > >
> >> > > > Ian
> >> > > >
> >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> >> > > mike.tutkowski@solidfire.com>
> >> > > > wrote:
> >> > > >
> >> > > > > To me it sounds like number two and number three are different
> >> uses
> >> > for
> >> > > > the
> >> > > > > same "thing"(which is totally fine).
> >> > > > >
> >> > > > > As for taking a fast SAN snapshot and exporting it
> >> asynchronously, do
> >> > > we
> >> > > > > see the SSVM as performing the export?
> >> > > > >
> >> > > > > To be backwards compatible with what we have in 4.6 and later
> for
> >> > > volume
> >> > > > > snapshots for managed storage, I think it might be easier if we
> >> pass
> >> > > in a
> >> > > > > flag that says whether or not to archive the SAN snapshot
> (which,
> >> I
> >> > > > think,
> >> > > > > is something that you suggested, as well, Pierre-Luc).
> >> > > > >
> >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> pdion@cloudops.com
> >> > > > > <javascript:;>> wrote:
> >> > > > >
> >> > > > > > Hi Mike,
> >> > > > > >
> >> > > > > > The reason behind the creation of a SAN snapshot which is
> >> exported
> >> > > into
> >> > > > > > secondary storage, is because creating a copy of the .VHD
> >> directly
> >> > > > would
> >> > > > > > impact uptime of the VM as creating that copy take lots of
> time.
> >> > Has
> >> > > > > oppose
> >> > > > > > to a SAN snapshot that is close to instantaneous which can
> >> > afterward
> >> > > be
> >> > > > > > clone into Secondary Storage asynchronously.
> >> > > > > >
> >> > > > > > I would suspect an extracted VolumeSnapshot taken from a SAN
> >> > snapshot
> >> > > > > could
> >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> >> consumption
> >> > > on
> >> > > > > the
> >> > > > > > Primary Storage side.
> >> > > > > >
> >> > > > > >
> >> > > > > > I see 3 definitions in our current discussion regarding the
> term
> >> > > > snapshot
> >> > > > > > (these are not official terminology but by own interpretation
> of
> >> > > them):
> >> > > > > >
> >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a
> >> > > > snapshot):
> >> > > > > > it's a volume snapshot at the storage level, point in time of
> >> your
> >> > > > data.
> >> > > > > it
> >> > > > > > reside on the primary storage. Useful and efficient for
> software
> >> > side
> >> > > > > > incident.
> >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud
> >> backup
> >> > > > aws-S3
> >> > > > > > style ): Point in time copy of the Virtual Disk that reside
> on a
> >> > > > > different
> >> > > > > > storage array then the original Volume. Facilitate data
> >> migration
> >> > > > between
> >> > > > > > clusters and, in case of primary storage incident, Volume
> >> snapshots
> >> > > are
> >> > > > > not
> >> > > > > > impacted and can be reuse.
> >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that also
> >> > > validate
> >> > > > > data
> >> > > > > > integrity, provide a storage efficient archiving method and an
> >> > > > > independent
> >> > > > > > way to restore your data in case of an major infrastructure
> >> > disaster.
> >> > > > > >
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > >
> >> > > > > > PL
> >> > > > > >
> >> > > > > >
> >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> >> > > > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> >> > > > > > > wrote:
> >> > > > > >
> >> > > > > > > So, let's see if I currently follow the requirements:
> >> > > > > > >
> >> > > > > > > * Augment volume snapshots for managed storage to
> >> conditionally
> >> > > > export
> >> > > > > > data
> >> > > > > > > to NFS. The current process of taking a snapshot on the SAN
> is
> >> > > fine,
> >> > > > > but
> >> > > > > > > we'd like the option to export the data to NFS, as well.
> >> > > > > > >
> >> > > > > > > Questions:
> >> > > > > > >
> >> > > > > > > Once the data has been exported to NFS, do we keep the SAN
> >> > snapshot
> >> > > > or
> >> > > > > > > delete it?
> >> > > > > > >
> >> > > > > > > If we are deleting the SAN snapshot, then why don't we just
> >> copy
> >> > > the
> >> > > > > VHD
> >> > > > > > > from primary to secondary the way we do today for
> non-managed
> >> > (i.e.
> >> > > > > > > traditional) storage? Why create a SAN snapshot in that
> >> scenario?
> >> > > > > Perhaps
> >> > > > > > > to have the SSVM mount and perform the VHD copy to secondary
> >> > > storage
> >> > > > > > > instead of a XenServer host?
> >> > > > > > >
> >> > > > > > > Thanks for the clarification.
> >> > > > > > >
> >> > > > > > > By the way, to me a backup is when you copy data from one
> >> storage
> >> > > > > system
> >> > > > > > to
> >> > > > > > > another (regardless of features, if any, to restore the data
> >> in
> >> > the
> >> > > > > > > future). A snapshot is a point-in-time view of the data of a
> >> > volume
> >> > > > and
> >> > > > > > > it's stored on the same storage system as the volume.
> >> > > > > > >
> >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> >> > > pdion@cloudops.com
> >> > > > > <javascript:;>
> >> > > > > > <javascript:;>>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > That's fun to see that discussion happening. I 100% agree
> >> with
> >> > > > Paul's
> >> > > > > > > > points of view. VolumeSnapshot are not a backup, but I do
> >> > > consider
> >> > > > > them
> >> > > > > > > as
> >> > > > > > > > a safety vest against Primary Storage failure, because
> >> failure
> >> > > > append
> >> > > > > > > :-( .
> >> > > > > > > >
> >> > > > > > > > The current proposal around snapshots that reside on the
> >> > primary
> >> > > > > > storage
> >> > > > > > > or
> >> > > > > > > > snapshots that end in the Secondary Storage  is not to
> >> address
> >> > > any
> >> > > > > kind
> >> > > > > > > of
> >> > > > > > > > backups requirement because a snapshot is not a backup,
> >> event
> >> > an
> >> > > > > > > extracted
> >> > > > > > > > VM snapshot.
> >> > > > > > > >
> >> > > > > > > > The main idea, and again this is for managed storage;
> >> > > > > > > >
> >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> >> capability
> >> > > for
> >> > > > > > fast
> >> > > > > > > > response time that support rollback to previous timestamp,
> >> > create
> >> > > > new
> >> > > > > > > > volume and maybe create template.
> >> > > > > > > >     not required to be a new API if the work is already
> >> done, I
> >> > > > think
> >> > > > > > > this
> >> > > > > > > > is a different behaviors than the user expectation of a
> >> > > > > > volume-snapshot.
> >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior
> >> that
> >> > > > create
> >> > > > > > an
> >> > > > > > > > extraction of a volume into SecondaryStorage which can be
> >> reuse
> >> > > to
> >> > > > > > > create a
> >> > > > > > > > new volume into another Primary Storage. This type of
> >> snapshot
> >> > > is a
> >> > > > > > slow
> >> > > > > > > > job since yes it would have to copy the full volume size
> on
> >> the
> >> > > > > > Secondary
> >> > > > > > > > storage.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > PL
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> >> > > > > > <javascript:;>>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > I think I share you view on the 'Ideal world'. Backup
> (via
> >> > > Volume
> >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is
> >> > > amplified
> >> > > > > > > > especially
> >> > > > > > > > > when you have a object storage as your secondary storage
> >> > > because
> >> > > > it
> >> > > > > > > > > requires two copies (one to an NFS staging area and from
> >> > there
> >> > > to
> >> > > > > > > object
> >> > > > > > > > > storage). And not to mention that all these copies are
> >> > > consuming
> >> > > > > > > > hypervisor
> >> > > > > > > > > resources. Xenserver's Dom0 is also a huge bottleneck as
> >> all
> >> > > the
> >> > > > > > > Network
> >> > > > > > > > > and I/O flow through it. So our intention of proposing
> the
> >> > > > "Storage
> >> > > > > > > > > Snapshots" is to give a better way of achiving snapshots
> >> > while
> >> > > > > still
> >> > > > > > > > > keeping the original definition of volume snpashots (ie
> >> > upload
> >> > > to
> >> > > > > sec
> >> > > > > > > > > storage).
> >> > > > > > > > >
> >> > > > > > > > > But as Erik pointed out volume snapshots are not
> backups.
> >> > They
> >> > > > > don't
> >> > > > > > > work
> >> > > > > > > > > form multi-disk LVM volume groups and dynamic disks. I
> am
> >> all
> >> > > in
> >> > > > > for
> >> > > > > > a
> >> > > > > > > > > better backup solution which handles these use cases and
> >> > > utilizes
> >> > > > > the
> >> > > > > > > > > storage's advanced features.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> >> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > In the beginning... there were CloudStack snapshots
> and
> >> > they
> >> > > > were
> >> > > > > > > > > actually
> >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> snapshots.
> >> > > > > > > > > > Then VM snapshots were created (which are
> point-in-time
> >> > > > > hypervisor
> >> > > > > > > > > > snapshots) and we started referring to the original
> >> > snapshots
> >> > > > as
> >> > > > > > > volume
> >> > > > > > > > > > snapshots.
> >> > > > > > > > > >
> >> > > > > > > > > > CloudStack does not offer 'backups', but many people
> use
> >> > > volume
> >> > > > > > > > snapshots
> >> > > > > > > > > > as backups. However you can't in-place restore volume
> >> > > snapshots
> >> > > > > and
> >> > > > > > > if
> >> > > > > > > > > you
> >> > > > > > > > > > have a VM with multiple volumes, the volume snapshots
> >> must
> >> > be
> >> > > > > done
> >> > > > > > in
> >> > > > > > > > > > series, meaning that the state across all of the
> >> volumes is
> >> > > > > > unlikely
> >> > > > > > > to
> >> > > > > > > > > be
> >> > > > > > > > > > consistent.
> >> > > > > > > > > >
> >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> options
> >> > > which
> >> > > > > > users
> >> > > > > > > > > might
> >> > > > > > > > > > expect as well options as to where they might be
> >> stored. In
> >> > > my
> >> > > > > > ideal
> >> > > > > > > > > world
> >> > > > > > > > > > they would also be able to leverage back-end hardware
> >> (such
> >> > > as
> >> > > > > > > > Solidfire,
> >> > > > > > > > > > NetApp etc :) ) and software such as Veeam, Commvault
> >> etc
> >> > to
> >> > > > > > > accelerate
> >> > > > > > > > > the
> >> > > > > > > > > > process.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> >> > > > > > > > > > Paul Angus
> >> > > > > > > > > > VP Technology ,  ShapeBlue
> >> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> >> > > > > > > > > >
> >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> >> > > > > m:
> >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> >> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> >> > <javascript:;>
> >> > > |
> >> > > > > t: @cloudyangus*
> >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> >> > > > > <ja...@cloudyangus>
> >> > > > > > |  w:
> >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> >> > > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> >> > > > > > > > > > Shape Blue Ltd is a company incorporated in England &
> >> > Wales.
> >> > > > > > > ShapeBlue
> >> > > > > > > > > > Services India LLP is a company incorporated in India
> >> and
> >> > is
> >> > > > > > operated
> >> > > > > > > > > under
> >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> >> Consultoria
> >> > > Ltda
> >> > > > > is
> >> > > > > > a
> >> > > > > > > > > > company incorporated in Brasil and is operated under
> >> > license
> >> > > > from
> >> > > > > > > Shape
> >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> >> by
> >> > The
> >> > > > > > > Republic
> >> > > > > > > > of
> >> > > > > > > > > > South Africa and is traded under license from Shape
> Blue
> >> > Ltd.
> >> > > > > > > ShapeBlue
> >> > > > > > > > > is
> >> > > > > > > > > > a registered trademark.
> >> > > > > > > > > > This email and any attachments to it may be
> confidential
> >> > and
> >> > > > are
> >> > > > > > > > intended
> >> > > > > > > > > > solely for the use of the individual to whom it is
> >> > addressed.
> >> > > > Any
> >> > > > > > > views
> >> > > > > > > > > or
> >> > > > > > > > > > opinions expressed are solely those of the author and
> do
> >> > not
> >> > > > > > > > necessarily
> >> > > > > > > > > > represent those of Shape Blue Ltd or related
> companies.
> >> If
> >> > > you
> >> > > > > are
> >> > > > > > > not
> >> > > > > > > > > the
> >> > > > > > > > > > intended recipient of this email, you must neither
> take
> >> any
> >> > > > > action
> >> > > > > > > > based
> >> > > > > > > > > > upon its contents, nor copy or show it to anyone.
> Please
> >> > > > contact
> >> > > > > > the
> >> > > > > > > > > sender
> >> > > > > > > > > > if you believe you have received this email in error.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > -----Original Message-----
> >> > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> >> > > > > <javascript:;> <javascript:;>]
> >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> >> > <javascript:;>
> >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> >> > > > > > > > > >
> >> > > > > > > > > > Paul,
> >> > > > > > > > > >
> >> > > > > > > > > > When you say actual backups, how would it be different
> >> from
> >> > > the
> >> > > > > > > Volume
> >> > > > > > > > > > Snapshots that exist currently. My understanding is
> that
> >> > > > Backups
> >> > > > > > end
> >> > > > > > > up
> >> > > > > > > > > in
> >> > > > > > > > > > Sec Storage whereas Snapshots are just a point-in-time
> >> > state
> >> > > of
> >> > > > > > your
> >> > > > > > > > > volume
> >> > > > > > > > > > which can be restored back correct?
> >> > > > > > > > > >
> >> > > > > > > > > > -Syed
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> >> > > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Syed,
> >> > > > > > > > > > >
> >> > > > > > > > > > > As I understand it, the SolidFire plugin will export
> >> the
> >> > > > > snapshot
> >> > > > > > > to
> >> > > > > > > > > > > secondary storage if the user requests a template
> from
> >> > the
> >> > > > > > snapshot
> >> > > > > > > > or
> >> > > > > > > > > > > wants to download the snapshot from the cloud. This
> >> is a
> >> > > > good,
> >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> storage
> >> is
> >> > > > super
> >> > > > > > > > > > > reliable and snapshots on SolidFire arrays take up
> >> next
> >> > to
> >> > > no
> >> > > > > > > space.
> >> > > > > > > > > > > BUT I think that we are talking about a more general
> >> > > purpose
> >> > > > > API,
> >> > > > > > > and
> >> > > > > > > > > > > other storage systems may not be as awesome as
> Mike's.
> >> > > That's
> >> > > > > my
> >> > > > > > > > > > > concern. Also, the time to transfer for say 1TB to
> >> move
> >> > > from
> >> > > > > > > primary
> >> > > > > > > > > > > to sec storage and then create a VM template out of
> it
> >> > may
> >> > > be
> >> > > > > too
> >> > > > > > > > long
> >> > > > > > > > > > for users.
> >> > > > > > > > > > >
> >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> snapshot
> >> for
> >> > > > > backup,
> >> > > > > > > > it's
> >> > > > > > > > > > > just that users want to do backups and a volume
> >> snapshot
> >> > is
> >> > > > the
> >> > > > > > > only
> >> > > > > > > > > > > type of snapshot that copies the disk elsewhere and
> >> can
> >> > be
> >> > > > > > > scheduled.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I'm 'pondering' the implications of enabling actual
> >> > backups
> >> > > > > > > (through
> >> > > > > > > > > > > recognised backup providers) and the user
> requirements
> >> > > around
> >> > > > > > them
> >> > > > > > > > > > > (particularly restoration use cases) as a separate
> >> thread
> >> > > of
> >> > > > > > work.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul
> >> Angus
> >> > > VP
> >> > > > > > > > Technology
> >> > > > > > > > > > > , ShapeBlue
> >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> >> > > > > > > > > > >
> >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> >> > > > |
> >> > > > > m:
> >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> >> > > <javascript:;> |
> >> > > > > t: @cloudyangus*
> >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> >> <javascript:;>
> >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS
> UK
> >> > Shape
> >> > > > > Blue
> >> > > > > > > Ltd
> >> > > > > > > > > > > is a company incorporated in England & Wales.
> >> ShapeBlue
> >> > > > > Services
> >> > > > > > > > India
> >> > > > > > > > > > > LLP is a company incorporated in India and is
> operated
> >> > > under
> >> > > > > > > license
> >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria
> >> Ltda
> >> > is
> >> > > a
> >> > > > > > > company
> >> > > > > > > > > > > incorporated in Brasil and is operated under license
> >> from
> >> > > > Shape
> >> > > > > > > Blue
> >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by
> >> The
> >> > > > > Republic
> >> > > > > > > of
> >> > > > > > > > > > > South Africa and is traded under license from Shape
> >> Blue
> >> > > Ltd.
> >> > > > > > > > > > > ShapeBlue is a registered trademark.
> >> > > > > > > > > > > This email and any attachments to it may be
> >> confidential
> >> > > and
> >> > > > > are
> >> > > > > > > > > > > intended solely for the use of the individual to
> whom
> >> it
> >> > is
> >> > > > > > > > addressed.
> >> > > > > > > > > > > Any views or opinions expressed are solely those of
> >> the
> >> > > > author
> >> > > > > > and
> >> > > > > > > do
> >> > > > > > > > > > > not necessarily represent those of Shape Blue Ltd or
> >> > > related
> >> > > > > > > > > > > companies. If you are not the intended recipient of
> >> this
> >> > > > email,
> >> > > > > > you
> >> > > > > > > > > > > must neither take any action based upon its
> contents,
> >> nor
> >> > > > copy
> >> > > > > or
> >> > > > > > > > show
> >> > > > > > > > > > > it to anyone. Please contact the sender if you
> believe
> >> > you
> >> > > > have
> >> > > > > > > > > received
> >> > > > > > > > > > this email in error.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > -----Original Message-----
> >> > > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> >> > > > > <javascript:;>
> >> > > > > > <javascript:;>]
> >> > > > > > > > > > > Sent: 05 February 2016 15:31
> >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> >> > > <javascript:;>
> >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> Snapshots
> >> > > > > > > > > > >
> >> > > > > > > > > > > I think the terminology confusion comes from AWS
> where
> >> > they
> >> > > > do
> >> > > > > > EBS
> >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
> >> followed
> >> > > > that.
> >> > > > > > And
> >> > > > > > > > as
> >> > > > > > > > > > > an end user who is oblivious to the internals of my
> >> > > provider,
> >> > > > > my
> >> > > > > > > > > > > expectation would be something similar to what AWS
> as
> >> > that
> >> > > is
> >> > > > > my
> >> > > > > > > > > > > biggest reference point.
> >> > > > > > > > > > >
> >> > > > > > > > > > > To your point Mike, I agree that a Primary Storage
> >> > failure
> >> > > on
> >> > > > > > > > > > > SolidFire is unlikely, there are other motivations
> >> for us
> >> > > to
> >> > > > > push
> >> > > > > > > > data
> >> > > > > > > > > > > to secondary storage. Primary storage (atleast for
> us)
> >> > > costs
> >> > > > > > > around 3
> >> > > > > > > > > > > times as much as secondary storage and snapshots on
> >> > primary
> >> > > > > > storage
> >> > > > > > > > > > > are rarely used (especially for some of our
> customers
> >> who
> >> > > do
> >> > > > > > daily
> >> > > > > > > > > > backups).
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> >> > > <javascript:;>>
> >> > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Some of the weirdness is around terminology.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > For most systems I've worked on, a snapshot and a
> >> > backup
> >> > > > are
> >> > > > > > two
> >> > > > > > > > > > > > completely different things (but CloudStack has
> >> > > > traditionally
> >> > > > > > > used
> >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
> >> though,
> >> > > that
> >> > > > > if
> >> > > > > > > your
> >> > > > > > > > > > > > primary storage is running on SolidFire that it is
> >> > > unlikely
> >> > > > > > > you'll
> >> > > > > > > > > > > > encounter an issue where your primary storage goes
> >> > > offline
> >> > > > > (and
> >> > > > > > > > > > > > you'll even maintain your performance guarantees
> >> during
> >> > > > > failure
> >> > > > > > > > > > > > scenarios and upgrades, as well). That being the
> >> case,
> >> > it
> >> > > > is
> >> > > > > > less
> >> > > > > > > > > > > > useful to require a backup to Swift (but it's
> >> perfectly
> >> > > OK
> >> > > > if
> >> > > > > > > > that's
> >> > > > > > > > > > > > what we want to do
> >> > > > > > > > > > > here).
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> >> > <javascript:;>>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Paul,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > I believe with the current implementation of
> >> > Snapshots
> >> > > on
> >> > > > > > > managed
> >> > > > > > > > > > > > > storage
> >> > > > > > > > > > > > > (SolidFire) the snapshots are never exported to
> >> the
> >> > > > > secondary
> >> > > > > > > > > > storage.
> >> > > > > > > > > > > > > While this solves the problem of having
> snapshots
> >> > > taking
> >> > > > > > > forever
> >> > > > > > > > > > > > > to get to sec storage, this leaves us with a
> >> > > > > > > > > > > > huge
> >> > > > > > > > > > > > > liability if our primary storage goes down. We
> see
> >> > > > > snapshots
> >> > > > > > as
> >> > > > > > > > > > > > > our recovery path because we store them in Swift
> >> > which
> >> > > is
> >> > > > > > > > reliable
> >> > > > > > > > > > > > > and resilient to failures.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> Volume
> >> > > > snapshots
> >> > > > > > > > always
> >> > > > > > > > > > > > > backed up to secondary storage and Storage
> >> Snapshots
> >> > > stay
> >> > > > > on
> >> > > > > > > the
> >> > > > > > > > > > > > > primary
> >> > > > > > > > > > > > storage.
> >> > > > > > > > > > > > > A provider could potentially mix both these and
> >> solve
> >> > > the
> >> > > > > > > problem
> >> > > > > > > > > > > > > that you mentioned where you want to meet user's
> >> > > > > expectation
> >> > > > > > > of a
> >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while having
> >> an
> >> > > > ability
> >> > > > > > to
> >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the device)
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Hope this clarifies things.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > -Syed
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> >> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> >> > > <javascript:;>>
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > HI guys,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Could someone point me to the Jira bug of FS
> for
> >> > the
> >> > > > > > > > > > > > > > SAN-snapshot
> >> > > > > > > > > > > > feature
> >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > From my discussions with users and operators
> >> around
> >> > > > > > snapshots
> >> > > > > > > > > > > > > > I'd make
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > following observations:
> >> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
> >> long-term
> >> > > and
> >> > > > > > short
> >> > > > > > > > > > > > > > term)
> >> > > > > > > > > > > > with
> >> > > > > > > > > > > > > > the expectation that they can use them for
> >> recovery
> >> > > if
> >> > > > > > > > required.
> >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> something
> >> > has
> >> > > > gone
> >> > > > > > > wrong
> >> > > > > > > > > > > > > > with primary storage.
> >> > > > > > > > > > > > > > c. users sometimes want to be able to export
> >> > > snapshots
> >> > > > as
> >> > > > > > > well
> >> > > > > > > > > > > > > > as
> >> > > > > > > > > > > > create
> >> > > > > > > > > > > > > > new VMs from their snapshots
> >> > > > > > > > > > > > > > d. snapshots are a currently a massive pain
> for
> >> > > > > operators,
> >> > > > > > I
> >> > > > > > > > > > > > > > know at
> >> > > > > > > > > > > > > least
> >> > > > > > > > > > > > > > one public cloud who have snapshots which
> take 2
> >> > days
> >> > > > to
> >> > > > > > > > > complete.
> >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used for
> >> > multiple
> >> > > > LVM
> >> > > > > > > > disks.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I think the process Mike has used in the
> >> SolidFire
> >> > > > plugin
> >> > > > > > > (only
> >> > > > > > > > > > > > > > moving
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > disk image to secondary storage when you
> >> absolutely
> >> > > > have
> >> > > > > > to)
> >> > > > > > > is
> >> > > > > > > > > > > > > > a very
> >> > > > > > > > > > > > > good
> >> > > > > > > > > > > > > > and pragmatic solution. I wonder what problems
> >> an
> >> > > > > operator
> >> > > > > > > > might
> >> > > > > > > > > > > > > experience
> >> > > > > > > > > > > > > > if they have an issue with a given primary
> >> storage
> >> > > pool
> >> > > > > in
> >> > > > > > a
> >> > > > > > > > > > cluster.
> >> > > > > > > > > > > > (I
> >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> >> SolidFire
> >> > > case
> >> > > > > > Mike
> >> > > > > > > :)
> >> > > > > > > > > > > > > > ) And
> >> > > > > > > > > > > > if
> >> > > > > > > > > > > > > > the transfer from primary to secondary is
> slow,
> >> the
> >> > > > time
> >> > > > > to
> >> > > > > > > > > > > > > > being able
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > create a template or export the volume will be
> >> > slow.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > So for me the issue is around making sure that
> >> the
> >> > > end
> >> > > > > > users
> >> > > > > > > > > > > > expectations
> >> > > > > > > > > > > > > > are met (while improving the speed/efficiency
> of
> >> > the
> >> > > > back
> >> > > > > > > end)
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> >> Paul
> >> > > > Angus
> >> > > > > > VP
> >> > > > > > > > > > > > > > Technology , ShapeBlue
> >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> >> > > > > > > > > > > > > >
> >> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> >> > > > > > > |
> >> > > > > > > > m:
> >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> >> > > > > <javascript:;> | t:
> >> > > > > > @cloudyangus*
> >> > > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> >> > > > <javascript:;>
> >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> >> > > > > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com
> >
> >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> >> 4HS
> >> > UK
> >> > > > > Shape
> >> > > > > > > > Blue
> >> > > > > > > > > > > > > > Ltd is a company incorporated in England &
> >> Wales.
> >> > > > > ShapeBlue
> >> > > > > > > > > > > > > > Services India LLP is a company incorporated
> in
> >> > India
> >> > > > and
> >> > > > > > is
> >> > > > > > > > > > > > > > operated
> >> > > > > > > > > > > > > under
> >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> >> > > > > Consultoria
> >> > > > > > > Ltda
> >> > > > > > > > > > > > > > is a company incorporated in Brasil and is
> >> operated
> >> > > > under
> >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty
> >> Ltd
> >> > is
> >> > > a
> >> > > > > > > company
> >> > > > > > > > > > > > > > registered by The Republic
> >> > > > > > > > > > > > of
> >> > > > > > > > > > > > > > South Africa and is traded under license from
> >> Shape
> >> > > > Blue
> >> > > > > > Ltd.
> >> > > > > > > > > > > > > > ShapeBlue
> >> > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > a registered trademark.
> >> > > > > > > > > > > > > > This email and any attachments to it may be
> >> > > > confidential
> >> > > > > > and
> >> > > > > > > > are
> >> > > > > > > > > > > > intended
> >> > > > > > > > > > > > > > solely for the use of the individual to whom
> it
> >> is
> >> > > > > > addressed.
> >> > > > > > > > > > > > > > Any views
> >> > > > > > > > > > > > > or
> >> > > > > > > > > > > > > > opinions expressed are solely those of the
> >> author
> >> > and
> >> > > > do
> >> > > > > > not
> >> > > > > > > > > > > > necessarily
> >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or related
> >> > > companies.
> >> > > > > If
> >> > > > > > > you
> >> > > > > > > > > > > > > > are not
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > intended recipient of this email, you must
> >> neither
> >> > > take
> >> > > > > any
> >> > > > > > > > > > > > > > action
> >> > > > > > > > > > > > based
> >> > > > > > > > > > > > > > upon its contents, nor copy or show it to
> >> anyone.
> >> > > > Please
> >> > > > > > > > contact
> >> > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > sender
> >> > > > > > > > > > > > > > if you believe you have received this email in
> >> > error.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > -----Original Message-----
> >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> >> pdion@cloudops.com
> >> > > > > <javascript:;>
> >> > > > > > <javascript:;>]
> >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> >> > Snapshots
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Hi Mike,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > The idea of introducing a new API:
> >> StorageSnapshot
> >> > > for
> >> > > > > > > managed
> >> > > > > > > > > > > > > > storage
> >> > > > > > > > > > > > is
> >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> expected,
> >> > > > behavior
> >> > > > > > is
> >> > > > > > > to
> >> > > > > > > > > > > > > > archive snapshots into the Secondary Storage.
> >> So a
> >> > > > > > > > > > > > > > StorageSnapshot API would be
> >> > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > snapshot that remain on the managed storage
> >> > > appliance.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Quickly looking at the API doc and I don't
> see a
> >> > > strong
> >> > > > > > > > > > > > > > requirement for volume snapshots to be moved
> >> into
> >> > > > > secondary
> >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is not
> >> > useful,
> >> > > > but
> >> > > > > > > both
> >> > > > > > > > > > > > > > use cases are required. A snapshot that remain
> >> on
> >> > the
> >> > > > > > managed
> >> > > > > > > > > > > > > > storage, and another type of
> >> > > > > > > > > > > > snapshot
> >> > > > > > > > > > > > > > that end up into the secondary storage. Since
> >> > you've
> >> > > > > done a
> >> > > > > > > lot
> >> > > > > > > > > > > > > > of
> >> > > > > > > > > > > > work,
> >> > > > > > > > > > > > > > might easier to just add a parameter to the
> >> current
> >> > > > > > snapshot
> >> > > > > > > > API
> >> > > > > > > > > > > > > > that
> >> > > > > > > > > > > > > would
> >> > > > > > > > > > > > > > trigger an extraction of the storage snapshot
> >> into
> >> > > the
> >> > > > > > > > secondary
> >> > > > > > > > > > > > storage?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > PL
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> Tutkowski <
> >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > I think that all sounds reasonable then -
> >> thanks!
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> Mushtaq <
> >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> >> <javascript:;>>
> >> > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> >> > requirements.
> >> > > > One
> >> > > > > > of
> >> > > > > > > > our
> >> > > > > > > > > > > > earlier
> >> > > > > > > > > > > > > > >> iterations on this was to have an argument
> to
> >> > the
> >> > > > > create
> >> > > > > > > > > > > > > > >> snapshot
> >> > > > > > > > > > > > API
> >> > > > > > > > > > > > > > >> which decides whether to backup the volume
> to
> >> > sec
> >> > > > > > storage
> >> > > > > > > > but
> >> > > > > > > > > > > > > > >> we realized it would make management of
> >> > snapshots
> >> > > > > quite
> >> > > > > > > > messy
> >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> >> > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> <javascript:;>
> >> > > > > <javascript:;>>
> >> > > > > > > > > > > > > > >> wrote:
> >> > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >>> Hi,
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> Just to make sure I understand all the
> >> > > requirements
> >> > > > > > here:
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 1) This relates only to managed storage
> (1:1
> >> > > > mapping
> >> > > > > > > > between
> >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN volume).
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 2) We want to take the current (introduced
> >> in
> >> > > 4.6)
> >> > > > > > > > > > > > > > >>> functionality, which creates a snapshot on
> >> the
> >> > > SAN,
> >> > > > > and
> >> > > > > > > > > > > > > > >>> extend it via a config option (or
> >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> >> snapshot,
> >> > but
> >> > > > to
> >> > > > > > copy
> >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to
> NFS.
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken. It's
> >> the
> >> > > > backup
> >> > > > > to
> >> > > > > > > NFS
> >> > > > > > > > > > > > > > >>> that is optional.
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> >> snapshot
> >> > > > that's
> >> > > > > on
> >> > > > > > > the
> >> > > > > > > > > > > > > > >>> SAN (already works).
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created from
> >> the
> >> > > > > snapshot
> >> > > > > > > > > > > > > > >>> that's on
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > >>> SAN (already works as long as the new
> >> > CloudStack
> >> > > > > volume
> >> > > > > > > > ends
> >> > > > > > > > > > > > > > >>> up on the same primary storage).
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> snapshot
> >> API
> >> > > > then
> >> > > > > or
> >> > > > > > > > > > > > > > >>> would that just be the standard volume
> >> snapshot
> >> > > > > without
> >> > > > > > > the
> >> > > > > > > > > > > > > > >>> backup to
> >> > > > > > > > > > > NFS?
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> Thanks!
> >> > > > > > > > > > > > > > >>> Mike
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> Mushtaq
> >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
> >> > > > > <javascript:;>>
> >> > > > > > > > > > > > > > >>> wrote:
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>>> Is it possible to have both
> functionalities
> >> > > > > (snapshot
> >> > > > > > on
> >> > > > > > > > > > > > > > >>>> SAN & Sec
> >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we
> would
> >> > like
> >> > > > to
> >> > > > > > have
> >> > > > > > > > > both.
> >> > > > > > > > > > > > > > >>>> For example, some of our customers want
> to
> >> > > > implement
> >> > > > > > > their
> >> > > > > > > > > > > > > > >>>> own backup strategies and do encryption
> to
> >> > their
> >> > > > > > backups
> >> > > > > > > > > > > > > > >>>> which is a perfect use case for Storage
> >> > Snapshot
> >> > > > > while
> >> > > > > > > our
> >> > > > > > > > > > > > > > >>>> other customers will still keep using the
> >> > > standard
> >> > > > > > > volume
> >> > > > > > > > > > > snapshot.
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>> To keep things backward compatible, we
> can
> >> > add a
> >> > > > > > setting
> >> > > > > > > > > > > > > > >>>> which
> >> > > > > > > > > > > > says
> >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> >> because,
> >> > > after
> >> > > > > > all,
> >> > > > > > > > you
> >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when
> doing
> >> a
> >> > > > Volume
> >> > > > > > > > > Snapshot.
> >> > > > > > > > > > > > > > >>>> You could stop the process there and not
> do
> >> > the
> >> > > > > > upload.
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>> What do you think about this approach?
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>> Thanks,
> >> > > > > > > > > > > > > > >>>> -Syed
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> >> > Tutkowski <
> >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> >> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out load
> >> here,
> >> > but
> >> > > > if
> >> > > > > a
> >> > > > > > > > given
> >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need
> to
> >> > > provide
> >> > > > > > both
> >> > > > > > > > the
> >> > > > > > > > > > > > ability
> >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it to
> >> NFS
> >> > (if
> >> > > > > just
> >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we
> >> might
> >> > be
> >> > > > able
> >> > > > > > to
> >> > > > > > > > get
> >> > > > > > > > > > > > > > >>>>> away with no new
> >> > > > > > > > > > > > API
> >> > > > > > > > > > > > > > >>>>> calls and simply implement a new custom
> >> > > snapshot
> >> > > > > > > strategy
> >> > > > > > > > > > > > > > >>>>> and
> >> > > > > > > > > > > > data
> >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case where
> >> the
> >> > > > > > CloudStack
> >> > > > > > > > > > > > > > >>>>> cloud
> >> > > > > > > > > > > > does
> >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> >> exported-to-NFS
> >> > > > > backup.
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> In other words, the "default" behavior
> >> would
> >> > be
> >> > > > to
> >> > > > > > use
> >> > > > > > > > the
> >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> strategy
> >> > that
> >> > > > we
> >> > > > > > > > already
> >> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> >> > snapshot).
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however, wants
> >> to
> >> > > take
> >> > > > a
> >> > > > > > SAN
> >> > > > > > > > > > > > > > >>>>> snapshot and have the data exported to
> >> NFS,
> >> > > then
> >> > > > we
> >> > > > > > > could
> >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config file
> to
> >> > make
> >> > > > use
> >> > > > > > of
> >> > > > > > > a
> >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data motion
> >> > strategy
> >> > > > that
> >> > > > > > > > > > > > > > >>>>> performs both of these
> >> > > > > > > > > > > > > > activities.
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> This way, the old behavior is still the
> >> > default
> >> > > > for
> >> > > > > > > > users,
> >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change this
> >> > behavior
> >> > > > via
> >> > > > > > > > > > > configuration.
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> Thoughts?
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike
> >> > > Tutkowski <
> >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> >> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to come up
> >> > with a
> >> > > > > > viable
> >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way for
> >> them
> >> > > to
> >> > > > > move
> >> > > > > > > > from
> >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and some
> >> obvious
> >> > > way
> >> > > > > > that
> >> > > > > > > > > > > > > > >>>>>> they will know they need
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > do this).
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed
> >> > Mushtaq
> >> > > <
> >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the upgrade
> >> path
> >> > > > > however,
> >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and are
> on a
> >> > > > managed
> >> > > > > > > > storage
> >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> >> functionality
> >> > > > with
> >> > > > > > > Volume
> >> > > > > > > > > > > Snapshots.
> >> > > > > > > > > > > > > > >>>>>>>
> >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed
> >> > Mushtaq
> >> > > <
> >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly,
> currently
> >> > > > taking a
> >> > > > > > > > Volume
> >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a managed
> >> storage
> >> > > > keeps
> >> > > > > > it
> >> > > > > > > on
> >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part of this
> >> > > feature,
> >> > > > we
> >> > > > > > can
> >> > > > > > > > > > > > > > >>>>>>>> make sure
> >> > > > > > > > > > > > that
> >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed storage
> are
> >> > > > uploaded
> >> > > > > > to
> >> > > > > > > > the
> >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would make
> the
> >> > > Volume
> >> > > > > > > Snapshot
> >> > > > > > > > > > > > > > >>>>>>>> feature behave the same regardless of
> >> the
> >> > > > > storage
> >> > > > > > > > > > > > > > >>>>>>>> (managed or
> >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing the
> >> > > efficient
> >> > > > > > > backend
> >> > > > > > > > > > > > > > >>>>>>>> storage
> >> > > > > > > > > > > > > > capabilities, we can use the new storage
> >> snapshots
> >> > > API.
> >> > > > > > > > > > > > > > >>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike
> >> > > > Tutkowski <
> >> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> >> > <javascript:;>
> >> > > > > <javascript:;>>
> >> > > > > > wrote:
> >> > > > > > > > > > > > > > >>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need to
> have a
> >> > plan
> >> > > > to
> >> > > > > > deal
> >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we already have a
> >> > > feature
> >> > > > > (in
> >> > > > > > > 4.6
> >> > > > > > > > > > > > > > >>>>>>>>> and
> >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to use the
> >> > existing
> >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a
> >> volume
> >> > > > > snapshot
> >> > > > > > > (for
> >> > > > > > > > > > > > > > >>>>>>>>> managed
> >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a backend
> SAN
> >> > > > (using a
> >> > > > > > > > custom
> >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom data
> >> > motion
> >> > > > > > > strategy).
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then how
> >> should
> >> > > the
> >> > > > > > > original
> >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6 and
> >> later)
> >> > > be
> >> > > > > > > changed?
> >> > > > > > > > > > > > > > >>>>>>>>> If it
> >> > > > > > > > > > > > is
> >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support customers
> >> who
> >> > > are
> >> > > > > > > already
> >> > > > > > > > > > > > > > >>>>>>>>> using
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API to take
> >> > > > snapshots
> >> > > > > > on a
> >> > > > > > > > > > > > > > >>>>>>>>> backend
> >> > > > > > > > > > > > > SAN?
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> >> > > > > > > > > > > > > > >>>>>>>>> Mike
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM,
> Will
> >> > > > Stevens <
> >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create a
> Template
> >> > > from a
> >> > > > > > > > > > > > StorageSnapshot?
> >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template be stored
> >> in
> >> > the
> >> > > > > > > secondary
> >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal templates or
> will
> >> > that
> >> > > > be
> >> > > > > > > > handled
> >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> >> > > > > > > > > > > > > > vendor side?
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions
> >> Experts
> >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec
> >> *|*
> >> > > H3J
> >> > > > > 1S6
> >> > > > > > w
> >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM,
> Syed
> >> > > > Mushtaq <
> >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> >> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> >> > > > > > > > > > > > > > >>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM,
> Will
> >> > > > Stevens
> >> > > > > <
> >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> >> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design
> >> Spec in
> >> > > the
> >> > > > > > Jira
> >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not clear
> in
> >> the
> >> > > > > > 'mention'
> >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows as a
> page
> >> > 'you
> >> > > do
> >> > > > > not
> >> > > > > > > > have
> >> > > > > > > > > > > permission to'.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions
> >> Experts
> >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|*
> Quebec
> >> > *|*
> >> > > > H3J
> >> > > > > > 1S6
> >> > > > > > > w
> >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM,
> >> Syed
> >> > > Ahmed
> >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> >> <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > >
> >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of
> >> APIs
> >> > to
> >> > > > do
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage
> >> backends
> >> > > > like
> >> > > > > > > > > SolidFire.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed
> >> storage
> >> > > stay
> >> > > > > on
> >> > > > > > > the
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary to what
> >> > > > CloudStack
> >> > > > > > > calls
> >> > > > > > > > > > > snpshots.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage
> >> and
> >> > > > keeping
> >> > > > > > it
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> >> > > > > > > > > > > > and
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to have
> both
> >> > ways
> >> > > > of
> >> > > > > > > doing
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4
> >> new
> >> > > APIs
> >> > > > to
> >> > > > > > > > create
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of this
> >> > > feature? I
> >> > > > > > would
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am
> >> > > working
> >> > > > on
> >> > > > > > > > making
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> >> > > > > > > > > > > > spec
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to
> have a
> >> > high
> >> > > > > level
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before starting
> to
> >> > work
> >> > > on
> >> > > > > it.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> --
> >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer,
> >> SolidFire
> >> > > Inc.*
> >> > > > > > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> >> > > > <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world uses the
> >> > cloud
> >> > > > > > > > > > > > > > >>>>>>>>> <
> >> > > > > > > http://solidfire.com/solution/overview/?video=play
> >> > > > > > > > >*™
> >> > > > > > > > > > > > > > >>>>>>>>> *
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>> --
> >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire
> >> > Inc.*
> >> > > > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> >> > > <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> >> > > > > > > > > > > > > > >>>>>> Advancing the way the world uses the
> >> cloud
> >> > > > > > > > > > > > > > >>>>>> <
> >> > > > > http://solidfire.com/solution/overview/?video=play
> >> > > > > > > >*™*
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> --
> >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire
> >> Inc.*
> >> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> >> > <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> >> > > > > > > > > > > > > > >>>>> Advancing the way the world uses the
> cloud
> >> > > > > > > > > > > > > > >>>>> <
> >> > > > > http://solidfire.com/solution/overview/?video=play
> >> > > > > > > >*™*
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> --
> >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer, SolidFire
> >> Inc.*
> >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> >> <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>> o: 303.746.7302
> >> > > > > > > > > > > > > > >>> Advancing the way the world uses the cloud
> >> > > > > > > > > > > > > > >>> <
> >> > > > http://solidfire.com/solution/overview/?video=play
> >> > > > > > >*™*
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > --
> >> > > > > > > > > > > > > > > *Mike Tutkowski*
> >> > > > > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> Inc.*
> >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> >> <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > > o: 303.746.7302
> >> > > > > > > > > > > > > > > Advancing the way the world uses the cloud
> >> > > > > > > > > > > > > > > <
> >> > > http://solidfire.com/solution/overview/?video=play
> >> > > > > >*™*
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Find out more about ShapeBlue and our range of
> >> > > > CloudStack
> >> > > > > > > > > > > > > > related
> >> > > > > > > > > > > > > services:
> >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> >> > > > > > > > > > > > > > <
> >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> >> > > |
> >> > > > > > > > CSForge –
> >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> >> > > > > > > http://shapeblue.com/csforge/
> >> > > > > > > > >
> >> > > > > > > > > > > > > > CloudStack Consulting
> >> > > > > > > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/
> >
> >> |
> >> > > > > > > > > > > > > CloudStack
> >> > > > > > > > > > > > > > Software Engineering
> >> > > > > > > > > > > > > > <
> >> > > http://shapeblue.com/cloudstack-software-engineering/
> >> > > > >
> >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> >> > > > > > > > > > > > > > <
> >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> >> > > > > >
> >> > > > > > |
> >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> >> > > > > > > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > --
> >> > > > > > > > > > > > *Mike Tutkowski*
> >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > o: 303.746.7302
> >> > > > > > > > > > > > Advancing the way the world uses the cloud
> >> > > > > > > > > > > > <
> http://solidfire.com/solution/overview/?video=play
> >> > >*™*
> >> > > > > > > > > > > >
> >> > > > > > > > > > > Find out more about ShapeBlue and our range of
> >> CloudStack
> >> > > > > related
> >> > > > > > > > > > services:
> >> > > > > > > > > > > IaaS Cloud Design & Build
> >> > > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//
> >
> >> |
> >> > > > > CSForge –
> >> > > > > > > > rapid
> >> > > > > > > > > > > IaaS deployment framework <
> >> http://shapeblue.com/csforge/
> >> > >
> >> > > > > > > CloudStack
> >> > > > > > > > > > > Consulting <
> >> http://shapeblue.com/cloudstack-consultancy/
> >> > >
> >> > > |
> >> > > > > > > > CloudStack
> >> > > > > > > > > > > Software Engineering
> >> > > > > > > > > > > <
> >> http://shapeblue.com/cloudstack-software-engineering/>
> >> > > > > > > > > > > CloudStack Infrastructure Support
> >> > > > > > > > > > > <
> >> http://shapeblue.com/cloudstack-infrastructure-support/
> >> > >
> >> > > |
> >> > > > > > > > CloudStack
> >> > > > > > > > > > > Bootcamp Training Courses <
> >> > > > > > > http://shapeblue.com/cloudstack-training/
> >> > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > Find out more about ShapeBlue and our range of
> >> CloudStack
> >> > > > related
> >> > > > > > > > > services:
> >> > > > > > > > > > IaaS Cloud Design & Build
> >> > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//>
> |
> >> > > > CSForge –
> >> > > > > > > rapid
> >> > > > > > > > > > IaaS deployment framework <
> >> http://shapeblue.com/csforge/>
> >> > > > > > > > > > CloudStack Consulting <
> >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> >> > > > > > > |
> >> > > > > > > > > CloudStack
> >> > > > > > > > > > Software Engineering
> >> > > > > > > > > > <
> http://shapeblue.com/cloudstack-software-engineering/>
> >> > > > > > > > > > CloudStack Infrastructure Support
> >> > > > > > > > > > <
> >> http://shapeblue.com/cloudstack-infrastructure-support/>
> >> > |
> >> > > > > > > CloudStack
> >> > > > > > > > > > Bootcamp Training Courses <
> >> > > > > > http://shapeblue.com/cloudstack-training/
> >> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > *Mike Tutkowski*
> >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> >> > > > > > > o: 303.746.7302
> >> > > > > > > Advancing the way the world uses the cloud
> >> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > *Mike Tutkowski*
> >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> >> > > > > o: 303.746.7302
> >> > > > > Advancing the way the world uses the cloud
> >> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Ian Rae
> >> > > > CEO | PDG
> >> > > > c: 514.944.4008
> >> > > >
> >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > *Mike Tutkowski*
> >> > *Senior CloudStack Developer, SolidFire Inc.*
> >> > e: mike.tutkowski@solidfire.com
> >> > o: 303.746.7302
> >> > Advancing the way the world uses the cloud
> >> > <http://solidfire.com/solution/overview/?video=play>*™*
> >> >
> >>
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

OK, then - perfect. I agree with this approach.

Global setting's value for volume snapshots on managed storage:

Default value: Volume snapshot on managed storage = storage snapshot

Non-default value: Take snapshot on primary storage, but move it to NFS.

On Fri, Feb 12, 2016 at 10:04 AM, Syed Mushtaq <sy...@gmail.com>
wrote:

> Correct. Keeping the snapshot on managed storage should be the default
> behaviour as it will not break backward compatibility.
>
> -Syed
>
>
> On Fri, Feb 12, 2016 at 12:02 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
> > I think so.
> >
> > Just to confirm: The default behavior for a volume snapshot with managed
> > storage is equivalent to a storage snapshot.
> >
> > When the global setting value is changed, the data ends up on secondary
> > storage (NFS).
> >
> > Is that accurate?
> >
> > On Friday, February 12, 2016, Syed Mushtaq <sy...@gmail.com>
> > wrote:
> >
> > > > Situation: Global setting value is changed to a non-default value.
> > > >
> > > > 2) Action: Volume snapshot is taken with managed storage. Result: ???
> > I'm
> > > > not sure which scenario we're looking at here now.
> > > >
> > >
> > > In this situation, volume snapshots will end up on secondary storage
> and
> > > the snapshot on the managed storage will be deleted.
> > >
> > >
> > > >
> > > > 3) Action: Storage snapshot is taken with managed storage. Result:
> ???
> > > I'm
> > > > not sure which scenario we're looking at here now.
> > > >
> > >
> > > In this situation, the snapshot stays on the managed storage.
> > >
> > > Does this answer your question Mike?
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Feb 12, 2016 at 11:43 AM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com <javascript:;>> wrote:
> > >
> > > > Hi Syed,
> > > >
> > > > Can you clarify how you see these behaviors (below) working now that
> we
> > > are
> > > > considering a global settings value applicable to managed storage
> with
> > > > volume snapshots?
> > > >
> > > > State: Global setting value defaults to keep snapshot on storage
> system
> > > > (primary storage) for managed storage.
> > > >
> > > > 1) Action: Volume snapshot is taken with managed storage. Result:
> > > Snapshot
> > > > is taken and kept on storage system (primary storage). (This is what
> > you
> > > > and I have been working on for a couple months now.)
> > > >
> > > >
> > > >
> > > > Thanks!
> > > > Mike
> > > >
> > > > On Fri, Feb 12, 2016 at 9:15 AM, Syed Mushtaq <
> syed1.mushtaq@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > I agree with Mike's concern about backward compatibility. We can
> add
> > a
> > > > > global flag which makes sure that the way volume snapshots work
> > > currently
> > > > > on managed storage (stay on the device) is retained after upgrade.
> We
> > > can
> > > > > then safely implement the Storage Snapshot API while making the
> > Volume
> > > > > Snpashot API move the snapshot to Secondary Storage.
> > > > >
> > > > > Sounds good guys?
> > > > >
> > > > > -Syed
> > > > >
> > > > >
> > > > > On Mon, Feb 8, 2016 at 2:56 PM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > Here's what we have for snapshots for managed storage as of 4.6,
> > > Paul:
> > > > > >
> > > > > > 1. VM snapshots (no proposed changes to this).
> > > > > >
> > > > > > 2. Volume snapshots that do not end up on secondary storage, but
> > > rather
> > > > > are
> > > > > > stored on a SAN (effectively storing snapshots on primary
> storage).
> > > > > >
> > > > > > Pierre-Luc is saying he'd like this for snapshots for managed
> > > storage:
> > > > > >
> > > > > > A. VM snapshots (no proposed changes to this).
> > > > > >
> > > > > > B. Volume snapshots that export to secondary storage.
> > > > > >
> > > > > > C. New: Storage snapshots that behave like 2 (above).
> > > > > >
> > > > > > I like Pierre-Luc's ideas there, but the problem is backward
> > > > > compatibility.
> > > > > >
> > > > > > If customers who were using managed storage with volume snapshots
> > in
> > > > 4.6
> > > > > > were getting their snapshots put on a SAN, then in 4.9 - all of a
> > > > sudden
> > > > > -
> > > > > > their new snapshots are put on secondary storage (unless they
> > > > explicitly
> > > > > > change over to using the new Storage snapshots feature).
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 12:32 PM, Paul Angus <
> > > paul.angus@shapeblue.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Just to make sure I'm on the same page, are we talking about;
> > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9278 ?
> > > > > > >
> > > > > > > The FS reads (to me) more like 1a + the possibility to export
> to
> > > > > > secondary
> > > > > > > storage if required?
> > > > > > > Have I understood correctly?
> > > > > > > I have seen [1a] implemented for VMware by NetApp in their beta
> > > > > > CloudStack
> > > > > > > plugin (pleased I can say that without Mike beating me up now).
> > No
> > > > > > changes
> > > > > > > to the CloudStack API were required. (nb it didn't export to
> > > > secondary
> > > > > > > storage).
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 1. VM Snapshot (point-in-time hypervisor based snapshots)
> > > > > > > 1a. SAN assisted VM snapshots (point-in-time hypervisor
> snapshot
> > > > takes
> > > > > > > place on transparently SAN to avoid performance issue in disk
> > > chains)
> > > > > > > 2. SAN Snapshot (Storage Snapshot) - NEW
> > > > > > > 3. Volume Snapshot (current old/slow transfer to secstorage)
> > > > > > > 4. Backup - JUST AN IDEAL.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > > > Paul Angus
> > > > > > > VP Technology ,  ShapeBlue
> > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> > m:
> > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > > > > <pa...@cloudyangus>  |  w:
> > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > > > Shape Blue Ltd is a company incorporated in England & Wales.
> > > > ShapeBlue
> > > > > > > Services India LLP is a company incorporated in India and is
> > > operated
> > > > > > under
> > > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> > is
> > > a
> > > > > > > company incorporated in Brasil and is operated under license
> from
> > > > Shape
> > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > > > Republic
> > > > > of
> > > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > ShapeBlue
> > > > > > is
> > > > > > > a registered trademark.
> > > > > > > This email and any attachments to it may be confidential and
> are
> > > > > intended
> > > > > > > solely for the use of the individual to whom it is addressed.
> Any
> > > > views
> > > > > > or
> > > > > > > opinions expressed are solely those of the author and do not
> > > > > necessarily
> > > > > > > represent those of Shape Blue Ltd or related companies. If you
> > are
> > > > not
> > > > > > the
> > > > > > > intended recipient of this email, you must neither take any
> > action
> > > > > based
> > > > > > > upon its contents, nor copy or show it to anyone. Please
> contact
> > > the
> > > > > > sender
> > > > > > > if you believe you have received this email in error.
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
> > > > > > > Sent: Monday, February 8, 2016 7:16 PM
> > > > > > > To: dev@cloudstack.apache.org
> > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > >
> > > > > > > Correct, Will.
> > > > > > >
> > > > > > > That Global Settings would only be for managed storage.
> > Non-managed
> > > > > > > (traditional) volume snapshots are completely un-impacted by
> this
> > > > > > feature.
> > > > > > >
> > > > > > > If we need to sometimes keep the snapshots on the SAN and
> > sometimes
> > > > > push
> > > > > > > them to secondary storage, we'll need a more robust solution
> than
> > > > > Global
> > > > > > > Settings, though.
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <
> > > wstevens@cloudops.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Sorry. I missed a bit of context when I responded. The global
> > > > setting
> > > > > > > > would be only for the managed storage case, currently being
> > > called
> > > > > > > Storage
> > > > > > > > Snapshots, and is only to determine if a copy is pushed to
> > > > secondary
> > > > > > > > storage right? The global setting would not change the
> behavior
> > > of
> > > > > the
> > > > > > > > Volume Snapshots right?
> > > > > > > >
> > > > > > > > I was referring to the need for Volume Snapshots and Storage
> > > > > Snapshots
> > > > > > to
> > > > > > > > exist together. Disregard my comment. I caught up on context
> > > after
> > > > I
> > > > > > > > posted. My bad...
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > > > > w cloudops.com *|* tw @CloudOps_
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> > > > > > > > mike.tutkowski@solidfire.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Will,
> > > > > > > > >
> > > > > > > > > Who's picking the behavior? Is it the cloud provider or the
> > end
> > > > > user?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <
> > > > > wstevens@cloudops.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I don't think a global setting is a good option because
> we
> > > need
> > > > > > both
> > > > > > > > > > functionality to be available at the same time and for
> > > > different
> > > > > > use
> > > > > > > > > cases
> > > > > > > > > > to be able to pick which they choose.
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > > > > > > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > > > > > > w cloudops.com *|* tw @CloudOps_
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > > > > > > > > mike.tutkowski@solidfire.com
> > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Now that I re-read your e-mail, it dawned on me: The
> > > end-user
> > > > > > > doesn't
> > > > > > > > > > care
> > > > > > > > > > > where the snapshot is.
> > > > > > > > > > >
> > > > > > > > > > > If that's true, then we should perhaps control this via
> > > > Global
> > > > > > > > Settings
> > > > > > > > > > or
> > > > > > > > > > > something.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > > > > > > > > mike.tutkowski@solidfire.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > It's not ideal - true, but it does allow us to be
> > > backward
> > > > > > > > > compatible.
> > > > > > > > > > > >
> > > > > > > > > > > > If you have other ideas, though, about how to
> maintain
> > > > > backward
> > > > > > > > > > > > compatibility, I'm definitely open to hear them.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > > > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi Mike,
> > > > > > > > > > > >>
> > > > > > > > > > > >> Adding a flag to createSnapshot was the first and
> the
> > > most
> > > > > > > obvious
> > > > > > > > > > thing
> > > > > > > > > > > >> that came to our minds. The problem that I had with
> > this
> > > > was
> > > > > > > that:
> > > > > > > > > > > >>
> > > > > > > > > > > >> 1) I feel it is exposing something to the end user
> > that
> > > is
> > > > > > > > internal
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > >> cloud.
> > > > > > > > > > > >>
> > > > > > > > > > > >> 2) We have to follow two different ways of
> > > > restore/deletion
> > > > > in
> > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > > >> code path depending on where the Snapshot resides
> > which
> > > I
> > > > > find
> > > > > > > > kind
> > > > > > > > > > of a
> > > > > > > > > > > >> bad design.
> > > > > > > > > > > >>
> > > > > > > > > > > >> But if exposing a archive flag to the end user is
> > > > acceptable
> > > > > > > then
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > >> definitely use this instead of adding the
> > > StorageSnapshot
> > > > > API
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks,
> > > > > > > > > > > >> -Syed
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > > > > > > > > >> mike.tutkowski@solidfire.com
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Hi Pierre-Luc,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > My recommendation would be this:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Add an "archive" flag to the current
> volume-snapshot
> > > > API.
> > > > > > Its
> > > > > > > > > > default
> > > > > > > > > > > >> would
> > > > > > > > > > > >> > be "false" because that would be backward
> compatible
> > > > with
> > > > > > how
> > > > > > > > 4.6
> > > > > > > > > > has
> > > > > > > > > > > >> > volume snapshots implemented (i.e. they stay on
> the
> > > SAN
> > > > in
> > > > > > > 4.6,
> > > > > > > > > 4.7,
> > > > > > > > > > > and
> > > > > > > > > > > >> > 4.8).
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > If you set archive=true, then we would perform a
> > > > > background
> > > > > > > > > > migration
> > > > > > > > > > > of
> > > > > > > > > > > >> > the snapshot from the SAN to the secondary storage
> > > (then
> > > > > > > delete
> > > > > > > > > the
> > > > > > > > > > > SAN
> > > > > > > > > > > >> > snapshot).
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > That archive parameter would only be valid for
> > managed
> > > > > > > storage.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Sound reasonable?
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Also, a VM snapshot that includes disks provided
> by
> > > > > managed
> > > > > > > > > storage
> > > > > > > > > > > >> should
> > > > > > > > > > > >> > work.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Talk to you later,
> > > > > > > > > > > >> > Mike
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > > > > > > > > pdion@cloudops.com
> > > > > > > > > > >
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > Mike,
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > In terms of API's, would you prefer introducing
> a
> > > > > > parameter
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > >> > existing
> > > > > > > > > > > >> > > VolumeSnapshot, example: extract={true|false}
> > with a
> > > > > > > > default
> > > > > > > > > > > value
> > > > > > > > > > > >> of
> > > > > > > > > > > >> > > true which would extract snapshot into the
> > secondary
> > > > > > > storage,
> > > > > > > > > > which
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > current default behavior. Then for SAN snapshot
> > that
> > > > > > remain
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > >> SAN we
> > > > > > > > > > > >> > > would just set "extract=false" ? as oppose to
> > > create a
> > > > > new
> > > > > > > > > > > >> > > StorageSnapshot API ?
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Paul,
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > From what I'm seeing so far, we can't do a
> > > VM-snapshot
> > > > > > when
> > > > > > > > > using
> > > > > > > > > > > >> managed
> > > > > > > > > > > >> > > storage for VM having more than one Volume. For
> > the
> > > > > reason
> > > > > > > > that
> > > > > > > > > > > >> snapshot
> > > > > > > > > > > >> > > are performed outside of the hypervisor
> awareness
> > > and
> > > > > > > > > > > asynchronously.
> > > > > > > > > > > >> If
> > > > > > > > > > > >> > > someone have a way to address that, it would
> make
> > > > thinks
> > > > > > > much
> > > > > > > > > more
> > > > > > > > > > > >> > > attractive.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <
> > > > > > irae@cloudops.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > I think a service provider backup scenario is
> > more
> > > > > > likely
> > > > > > > to
> > > > > > > > > > take
> > > > > > > > > > > >> > > advantage
> > > > > > > > > > > >> > > > of SAN snapshot. There are a few reasons for
> > this.
> > > > > > > > Traditional
> > > > > > > > > > > >> backups
> > > > > > > > > > > >> > > > involve access to the file system, and there
> is
> > an
> > > > > > > > expectation
> > > > > > > > > > > that
> > > > > > > > > > > >> > this
> > > > > > > > > > > >> > > > can be done with reasonably short time frames
> > > > without
> > > > > > > > > negatively
> > > > > > > > > > > >> > > impacting
> > > > > > > > > > > >> > > > VM performance, and that the backup
> orchestrator
> > > can
> > > > > > apply
> > > > > > > > > > various
> > > > > > > > > > > >> > logic
> > > > > > > > > > > >> > > > and or transformations to the data (compress,
> > > > encrypt,
> > > > > > > > deltas
> > > > > > > > > > > >> etc...).
> > > > > > > > > > > >> > > > While it is true that one could apply a backup
> > > > process
> > > > > > to
> > > > > > > a
> > > > > > > > > > cloud
> > > > > > > > > > > >> > > snapshot,
> > > > > > > > > > > >> > > > this would be slow and inefficient requiring
> the
> > > > data
> > > > > to
> > > > > > > be
> > > > > > > > > > moved
> > > > > > > > > > > >> > several
> > > > > > > > > > > >> > > > times and there are some major bottlenecks
> with
> > > > cloud
> > > > > > > > > snapshots.
> > > > > > > > > > > >> With a
> > > > > > > > > > > >> > > > cloud snapshot - there seems to be no
> reasonable
> > > > > > > expectation
> > > > > > > > > of
> > > > > > > > > > > >> being
> > > > > > > > > > > >> > > able
> > > > > > > > > > > >> > > > to do differential snapshots (I think this
> > depends
> > > > on
> > > > > > the
> > > > > > > > > > > >> hypervisor)
> > > > > > > > > > > >> > and
> > > > > > > > > > > >> > > > if you do differential snapshots this will
> make
> > > file
> > > > > > > backups
> > > > > > > > > > from
> > > > > > > > > > > >> those
> > > > > > > > > > > >> > > > snapshots even more complicated to
> orchestrate.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Suspect there needs to be a different thread
> on
> > > how
> > > > to
> > > > > > > > better
> > > > > > > > > > > enable
> > > > > > > > > > > >> > > > backups, as a service. As per Paul's
> suggestion,
> > > but
> > > > > it
> > > > > > > is a
> > > > > > > > > > > related
> > > > > > > > > > > >> > > > workflow so relevant to this discussion.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Ian
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > > > > > > > > >> > > mike.tutkowski@solidfire.com>
> > > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > To me it sounds like number two and number
> > three
> > > > are
> > > > > > > > > different
> > > > > > > > > > > >> uses
> > > > > > > > > > > >> > for
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > same "thing"(which is totally fine).
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > As for taking a fast SAN snapshot and
> > exporting
> > > it
> > > > > > > > > > > >> asynchronously, do
> > > > > > > > > > > >> > > we
> > > > > > > > > > > >> > > > > see the SSVM as performing the export?
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > To be backwards compatible with what we have
> > in
> > > > 4.6
> > > > > > and
> > > > > > > > > later
> > > > > > > > > > > for
> > > > > > > > > > > >> > > volume
> > > > > > > > > > > >> > > > > snapshots for managed storage, I think it
> > might
> > > be
> > > > > > > easier
> > > > > > > > if
> > > > > > > > > > we
> > > > > > > > > > > >> pass
> > > > > > > > > > > >> > > in a
> > > > > > > > > > > >> > > > > flag that says whether or not to archive the
> > SAN
> > > > > > > snapshot
> > > > > > > > > > > (which,
> > > > > > > > > > > >> I
> > > > > > > > > > > >> > > > think,
> > > > > > > > > > > >> > > > > is something that you suggested, as well,
> > > > > Pierre-Luc).
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc
> Dion <
> > > > > > > > > > > pdion@cloudops.com
> > > > > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > Hi Mike,
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > The reason behind the creation of a SAN
> > > snapshot
> > > > > > which
> > > > > > > > is
> > > > > > > > > > > >> exported
> > > > > > > > > > > >> > > into
> > > > > > > > > > > >> > > > > > secondary storage, is because creating a
> > copy
> > > of
> > > > > the
> > > > > > > > .VHD
> > > > > > > > > > > >> directly
> > > > > > > > > > > >> > > > would
> > > > > > > > > > > >> > > > > > impact uptime of the VM as creating that
> > copy
> > > > take
> > > > > > > lots
> > > > > > > > of
> > > > > > > > > > > time.
> > > > > > > > > > > >> > Has
> > > > > > > > > > > >> > > > > oppose
> > > > > > > > > > > >> > > > > > to a SAN snapshot that is close to
> > > instantaneous
> > > > > > which
> > > > > > > > can
> > > > > > > > > > > >> > afterward
> > > > > > > > > > > >> > > be
> > > > > > > > > > > >> > > > > > clone into Secondary Storage
> asynchronously.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > I would suspect an extracted
> VolumeSnapshot
> > > > taken
> > > > > > > from a
> > > > > > > > > SAN
> > > > > > > > > > > >> > snapshot
> > > > > > > > > > > >> > > > > could
> > > > > > > > > > > >> > > > > > have is SAN snapshot deleted to avoid
> > duplica
> > > > and
> > > > > > > space
> > > > > > > > > > > >> consumption
> > > > > > > > > > > >> > > on
> > > > > > > > > > > >> > > > > the
> > > > > > > > > > > >> > > > > > Primary Storage side.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > I see 3 definitions in our current
> > discussion
> > > > > > > regarding
> > > > > > > > > the
> > > > > > > > > > > term
> > > > > > > > > > > >> > > > snapshot
> > > > > > > > > > > >> > > > > > (these are not official terminology but by
> > own
> > > > > > > > > > interpretation
> > > > > > > > > > > of
> > > > > > > > > > > >> > > them):
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot /
> > Mike's
> > > > > > > definition
> > > > > > > > > of
> > > > > > > > > > a
> > > > > > > > > > > >> > > > snapshot):
> > > > > > > > > > > >> > > > > > it's a volume snapshot at the storage
> level,
> > > > point
> > > > > > in
> > > > > > > > time
> > > > > > > > > > of
> > > > > > > > > > > >> your
> > > > > > > > > > > >> > > > data.
> > > > > > > > > > > >> > > > > it
> > > > > > > > > > > >> > > > > > reside on the primary storage. Useful and
> > > > > efficient
> > > > > > > for
> > > > > > > > > > > software
> > > > > > > > > > > >> > side
> > > > > > > > > > > >> > > > > > incident.
> > > > > > > > > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack
> > > > > > VolumeSnapshot/
> > > > > > > > > cloud
> > > > > > > > > > > >> backup
> > > > > > > > > > > >> > > > aws-S3
> > > > > > > > > > > >> > > > > > style ): Point in time copy of the Virtual
> > > Disk
> > > > > that
> > > > > > > > > reside
> > > > > > > > > > > on a
> > > > > > > > > > > >> > > > > different
> > > > > > > > > > > >> > > > > > storage array then the original Volume.
> > > > Facilitate
> > > > > > > data
> > > > > > > > > > > >> migration
> > > > > > > > > > > >> > > > between
> > > > > > > > > > > >> > > > > > clusters and, in case of primary storage
> > > > incident,
> > > > > > > > Volume
> > > > > > > > > > > >> snapshots
> > > > > > > > > > > >> > > are
> > > > > > > > > > > >> > > > > not
> > > > > > > > > > > >> > > > > > impacted and can be reuse.
> > > > > > > > > > > >> > > > > > 3. *Backup*: Archival of your
> > Virtual-machines
> > > > > data
> > > > > > > that
> > > > > > > > > > also
> > > > > > > > > > > >> > > validate
> > > > > > > > > > > >> > > > > data
> > > > > > > > > > > >> > > > > > integrity, provide a storage efficient
> > > archiving
> > > > > > > method
> > > > > > > > > and
> > > > > > > > > > an
> > > > > > > > > > > >> > > > > independent
> > > > > > > > > > > >> > > > > > way to restore your data in case of an
> major
> > > > > > > > > infrastructure
> > > > > > > > > > > >> > disaster.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > Regards,
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > PL
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike
> > > Tutkowski <
> > > > > > > > > > > >> > > > > > mike.tutkowski@solidfire.com
> <javascript:;>
> > > > > > > > > <javascript:;>
> > > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > So, let's see if I currently follow the
> > > > > > > requirements:
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > * Augment volume snapshots for managed
> > > storage
> > > > > to
> > > > > > > > > > > >> conditionally
> > > > > > > > > > > >> > > > export
> > > > > > > > > > > >> > > > > > data
> > > > > > > > > > > >> > > > > > > to NFS. The current process of taking a
> > > > snapshot
> > > > > > on
> > > > > > > > the
> > > > > > > > > > SAN
> > > > > > > > > > > is
> > > > > > > > > > > >> > > fine,
> > > > > > > > > > > >> > > > > but
> > > > > > > > > > > >> > > > > > > we'd like the option to export the data
> to
> > > > NFS,
> > > > > as
> > > > > > > > well.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Questions:
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Once the data has been exported to NFS,
> do
> > > we
> > > > > keep
> > > > > > > the
> > > > > > > > > SAN
> > > > > > > > > > > >> > snapshot
> > > > > > > > > > > >> > > > or
> > > > > > > > > > > >> > > > > > > delete it?
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > If we are deleting the SAN snapshot,
> then
> > > why
> > > > > > don't
> > > > > > > we
> > > > > > > > > > just
> > > > > > > > > > > >> copy
> > > > > > > > > > > >> > > the
> > > > > > > > > > > >> > > > > VHD
> > > > > > > > > > > >> > > > > > > from primary to secondary the way we do
> > > today
> > > > > for
> > > > > > > > > > > non-managed
> > > > > > > > > > > >> > (i.e.
> > > > > > > > > > > >> > > > > > > traditional) storage? Why create a SAN
> > > > snapshot
> > > > > in
> > > > > > > > that
> > > > > > > > > > > >> scenario?
> > > > > > > > > > > >> > > > > Perhaps
> > > > > > > > > > > >> > > > > > > to have the SSVM mount and perform the
> VHD
> > > > copy
> > > > > to
> > > > > > > > > > secondary
> > > > > > > > > > > >> > > storage
> > > > > > > > > > > >> > > > > > > instead of a XenServer host?
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Thanks for the clarification.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > By the way, to me a backup is when you
> > copy
> > > > data
> > > > > > > from
> > > > > > > > > one
> > > > > > > > > > > >> storage
> > > > > > > > > > > >> > > > > system
> > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > >> > > > > > > another (regardless of features, if any,
> > to
> > > > > > restore
> > > > > > > > the
> > > > > > > > > > data
> > > > > > > > > > > >> in
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > > > future). A snapshot is a point-in-time
> > view
> > > of
> > > > > the
> > > > > > > > data
> > > > > > > > > > of a
> > > > > > > > > > > >> > volume
> > > > > > > > > > > >> > > > and
> > > > > > > > > > > >> > > > > > > it's stored on the same storage system
> as
> > > the
> > > > > > > volume.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM,
> > Pierre-Luc
> > > > > Dion <
> > > > > > > > > > > >> > > pdion@cloudops.com
> > > > > > > > > > > >> > > > > <javascript:;>
> > > > > > > > > > > >> > > > > > <javascript:;>>
> > > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > > That's fun to see that discussion
> > > > happening. I
> > > > > > > 100%
> > > > > > > > > > agree
> > > > > > > > > > > >> with
> > > > > > > > > > > >> > > > Paul's
> > > > > > > > > > > >> > > > > > > > points of view. VolumeSnapshot are
> not a
> > > > > backup,
> > > > > > > > but I
> > > > > > > > > > do
> > > > > > > > > > > >> > > consider
> > > > > > > > > > > >> > > > > them
> > > > > > > > > > > >> > > > > > > as
> > > > > > > > > > > >> > > > > > > > a safety vest against Primary Storage
> > > > failure,
> > > > > > > > because
> > > > > > > > > > > >> failure
> > > > > > > > > > > >> > > > append
> > > > > > > > > > > >> > > > > > > :-( .
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > The current proposal around snapshots
> > that
> > > > > > reside
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > >> > primary
> > > > > > > > > > > >> > > > > > storage
> > > > > > > > > > > >> > > > > > > or
> > > > > > > > > > > >> > > > > > > > snapshots that end in the Secondary
> > > Storage
> > > > is
> > > > > > not
> > > > > > > > to
> > > > > > > > > > > >> address
> > > > > > > > > > > >> > > any
> > > > > > > > > > > >> > > > > kind
> > > > > > > > > > > >> > > > > > > of
> > > > > > > > > > > >> > > > > > > > backups requirement because a snapshot
> > is
> > > > not
> > > > > a
> > > > > > > > > backup,
> > > > > > > > > > > >> event
> > > > > > > > > > > >> > an
> > > > > > > > > > > >> > > > > > > extracted
> > > > > > > > > > > >> > > > > > > > VM snapshot.
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > The main idea, and again this is for
> > > managed
> > > > > > > > storage;
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage
> > > side
> > > > > > > snapshot
> > > > > > > > > > > >> capability
> > > > > > > > > > > >> > > for
> > > > > > > > > > > >> > > > > > fast
> > > > > > > > > > > >> > > > > > > > response time that support rollback to
> > > > > previous
> > > > > > > > > > timestamp,
> > > > > > > > > > > >> > create
> > > > > > > > > > > >> > > > new
> > > > > > > > > > > >> > > > > > > > volume and maybe create template.
> > > > > > > > > > > >> > > > > > > > not required to be a new API if the
> work
> > > is
> > > > > > > > > already
> > > > > > > > > > > >> done, I
> > > > > > > > > > > >> > > > think
> > > > > > > > > > > >> > > > > > > this
> > > > > > > > > > > >> > > > > > > > is a different behaviors than the user
> > > > > > expectation
> > > > > > > > of
> > > > > > > > > a
> > > > > > > > > > > >> > > > > > volume-snapshot.
> > > > > > > > > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current
> > > > > cloudstack
> > > > > > > > > > behavior
> > > > > > > > > > > >> that
> > > > > > > > > > > >> > > > create
> > > > > > > > > > > >> > > > > > an
> > > > > > > > > > > >> > > > > > > > extraction of a volume into
> > > SecondaryStorage
> > > > > > which
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > >> reuse
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > > create a
> > > > > > > > > > > >> > > > > > > > new volume into another Primary
> Storage.
> > > > This
> > > > > > type
> > > > > > > > of
> > > > > > > > > > > >> snapshot
> > > > > > > > > > > >> > > is a
> > > > > > > > > > > >> > > > > > slow
> > > > > > > > > > > >> > > > > > > > job since yes it would have to copy
> the
> > > full
> > > > > > > volume
> > > > > > > > > size
> > > > > > > > > > > on
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > > Secondary
> > > > > > > > > > > >> > > > > > > > storage.
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > PL
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed
> > > > Mushtaq
> > > > > <
> > > > > > > > > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > > > > > > >> > > > > > <javascript:;>>
> > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > > I think I share you view on the
> 'Ideal
> > > > > world'.
> > > > > > > > > Backup
> > > > > > > > > > > (via
> > > > > > > > > > > >> > > Volume
> > > > > > > > > > > >> > > > > > > > > Snapshots) is a huge bottleneck in
> > > > > Cloudstack.
> > > > > > > > This
> > > > > > > > > is
> > > > > > > > > > > >> > > amplified
> > > > > > > > > > > >> > > > > > > > especially
> > > > > > > > > > > >> > > > > > > > > when you have a object storage as
> your
> > > > > > secondary
> > > > > > > > > > storage
> > > > > > > > > > > >> > > because
> > > > > > > > > > > >> > > > it
> > > > > > > > > > > >> > > > > > > > > requires two copies (one to an NFS
> > > staging
> > > > > > area
> > > > > > > > and
> > > > > > > > > > from
> > > > > > > > > > > >> > there
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > > object
> > > > > > > > > > > >> > > > > > > > > storage). And not to mention that
> all
> > > > these
> > > > > > > copies
> > > > > > > > > are
> > > > > > > > > > > >> > > consuming
> > > > > > > > > > > >> > > > > > > > hypervisor
> > > > > > > > > > > >> > > > > > > > > resources. Xenserver's Dom0 is also
> a
> > > huge
> > > > > > > > > bottleneck
> > > > > > > > > > as
> > > > > > > > > > > >> all
> > > > > > > > > > > >> > > the
> > > > > > > > > > > >> > > > > > > Network
> > > > > > > > > > > >> > > > > > > > > and I/O flow through it. So our
> > > intention
> > > > of
> > > > > > > > > proposing
> > > > > > > > > > > the
> > > > > > > > > > > >> > > > "Storage
> > > > > > > > > > > >> > > > > > > > > Snapshots" is to give a better way
> of
> > > > > achiving
> > > > > > > > > > snapshots
> > > > > > > > > > > >> > while
> > > > > > > > > > > >> > > > > still
> > > > > > > > > > > >> > > > > > > > > keeping the original definition of
> > > volume
> > > > > > > > snpashots
> > > > > > > > > > (ie
> > > > > > > > > > > >> > upload
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > sec
> > > > > > > > > > > >> > > > > > > > > storage).
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > But as Erik pointed out volume
> > snapshots
> > > > are
> > > > > > not
> > > > > > > > > > > backups.
> > > > > > > > > > > >> > They
> > > > > > > > > > > >> > > > > don't
> > > > > > > > > > > >> > > > > > > work
> > > > > > > > > > > >> > > > > > > > > form multi-disk LVM volume groups
> and
> > > > > dynamic
> > > > > > > > > disks. I
> > > > > > > > > > > am
> > > > > > > > > > > >> all
> > > > > > > > > > > >> > > in
> > > > > > > > > > > >> > > > > for
> > > > > > > > > > > >> > > > > > a
> > > > > > > > > > > >> > > > > > > > > better backup solution which handles
> > > these
> > > > > use
> > > > > > > > cases
> > > > > > > > > > and
> > > > > > > > > > > >> > > utilizes
> > > > > > > > > > > >> > > > > the
> > > > > > > > > > > >> > > > > > > > > storage's advanced features.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM,
> Paul
> > > > Angus
> > > > > <
> > > > > > > > > > > >> > > > > > paul.angus@shapeblue.com <javascript:;>
> > > > > > > <javascript:;>>
> > > > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > In the beginning... there were
> > > > CloudStack
> > > > > > > > > snapshots
> > > > > > > > > > > and
> > > > > > > > > > > >> > they
> > > > > > > > > > > >> > > > were
> > > > > > > > > > > >> > > > > > > > > actually
> > > > > > > > > > > >> > > > > > > > > > volume snapshots not hypervisor
> > > > > > point-in-time
> > > > > > > > > > > snapshots.
> > > > > > > > > > > >> > > > > > > > > > Then VM snapshots were created
> > (which
> > > > are
> > > > > > > > > > > point-in-time
> > > > > > > > > > > >> > > > > hypervisor
> > > > > > > > > > > >> > > > > > > > > > snapshots) and we started
> referring
> > to
> > > > the
> > > > > > > > > original
> > > > > > > > > > > >> > snapshots
> > > > > > > > > > > >> > > > as
> > > > > > > > > > > >> > > > > > > volume
> > > > > > > > > > > >> > > > > > > > > > snapshots.
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > CloudStack does not offer
> 'backups',
> > > but
> > > > > > many
> > > > > > > > > people
> > > > > > > > > > > use
> > > > > > > > > > > >> > > volume
> > > > > > > > > > > >> > > > > > > > snapshots
> > > > > > > > > > > >> > > > > > > > > > as backups. However you can't
> > in-place
> > > > > > restore
> > > > > > > > > > volume
> > > > > > > > > > > >> > > snapshots
> > > > > > > > > > > >> > > > > and
> > > > > > > > > > > >> > > > > > > if
> > > > > > > > > > > >> > > > > > > > > you
> > > > > > > > > > > >> > > > > > > > > > have a VM with multiple volumes,
> the
> > > > > volume
> > > > > > > > > > snapshots
> > > > > > > > > > > >> must
> > > > > > > > > > > >> > be
> > > > > > > > > > > >> > > > > done
> > > > > > > > > > > >> > > > > > in
> > > > > > > > > > > >> > > > > > > > > > series, meaning that the state
> > across
> > > > all
> > > > > of
> > > > > > > the
> > > > > > > > > > > >> volumes is
> > > > > > > > > > > >> > > > > > unlikely
> > > > > > > > > > > >> > > > > > > to
> > > > > > > > > > > >> > > > > > > > > be
> > > > > > > > > > > >> > > > > > > > > > consistent.
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > 'Actual Backups' would enable all
> of
> > > the
> > > > > > > restore
> > > > > > > > > > > options
> > > > > > > > > > > >> > > which
> > > > > > > > > > > >> > > > > > users
> > > > > > > > > > > >> > > > > > > > > might
> > > > > > > > > > > >> > > > > > > > > > expect as well options as to where
> > > they
> > > > > > might
> > > > > > > be
> > > > > > > > > > > >> stored. In
> > > > > > > > > > > >> > > my
> > > > > > > > > > > >> > > > > > ideal
> > > > > > > > > > > >> > > > > > > > > world
> > > > > > > > > > > >> > > > > > > > > > they would also be able to
> leverage
> > > > > back-end
> > > > > > > > > > hardware
> > > > > > > > > > > >> (such
> > > > > > > > > > > >> > > as
> > > > > > > > > > > >> > > > > > > > Solidfire,
> > > > > > > > > > > >> > > > > > > > > > NetApp etc :) ) and software such
> as
> > > > > Veeam,
> > > > > > > > > > Commvault
> > > > > > > > > > > >> etc
> > > > > > > > > > > >> > to
> > > > > > > > > > > >> > > > > > > accelerate
> > > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > process.
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > [image: ShapeBlue] <
> > > > > > http://www.shapeblue.com>
> > > > > > > > > > > >> > > > > > > > > > Paul Angus
> > > > > > > > > > > >> > > > > > > > > > VP Technology , ShapeBlue
> > > > > > > > > > > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203
> > 603
> > > > > 0540*
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > >
> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > > |
> > > > > > > > > > > >> > > > > m:
> > > > > > > > > > > >> > > > > > > > > > *+44 7711 418784*
> > > <+44%207711%20418784>
> > > > > > > > > > > >> > > > > > > > > > e: *paul.angus@shapeblue.com
> > > > > <javascript:;>
> > > > > > > > > > > >> > <javascript:;>
> > > > > > > > > > > >> > > |
> > > > > > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > > > > > >> > > > > > > > > > <paul.angus@shapeblue.com
> > > > <javascript:;>
> > > > > > > > > > > >> > > > > <ja...@cloudyangus>
> > > > > > > > > > > >> > > > > > | w:
> > > > > > > > > > > >> > > > > > > > > > *www.shapeblue.com* <
> > > > > > http://www.shapeblue.com
> > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > > > London
> > > > > > WC2N
> > > > > > > > > 4HS
> > > > > > > > > > UK
> > > > > > > > > > > >> > > > > > > > > > Shape Blue Ltd is a company
> > > incorporated
> > > > > in
> > > > > > > > > England
> > > > > > > > > > &
> > > > > > > > > > > >> > Wales.
> > > > > > > > > > > >> > > > > > > ShapeBlue
> > > > > > > > > > > >> > > > > > > > > > Services India LLP is a company
> > > > > incorporated
> > > > > > > in
> > > > > > > > > > India
> > > > > > > > > > > >> and
> > > > > > > > > > > >> > is
> > > > > > > > > > > >> > > > > > operated
> > > > > > > > > > > >> > > > > > > > > under
> > > > > > > > > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape
> > > Blue
> > > > > > Brasil
> > > > > > > > > > > >> Consultoria
> > > > > > > > > > > >> > > Ltda
> > > > > > > > > > > >> > > > > is
> > > > > > > > > > > >> > > > > > a
> > > > > > > > > > > >> > > > > > > > > > company incorporated in Brasil and
> > is
> > > > > > operated
> > > > > > > > > under
> > > > > > > > > > > >> > license
> > > > > > > > > > > >> > > > from
> > > > > > > > > > > >> > > > > > > Shape
> > > > > > > > > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is
> a
> > > > > company
> > > > > > > > > > registered
> > > > > > > > > > > >> by
> > > > > > > > > > > >> > The
> > > > > > > > > > > >> > > > > > > Republic
> > > > > > > > > > > >> > > > > > > > of
> > > > > > > > > > > >> > > > > > > > > > South Africa and is traded under
> > > license
> > > > > > from
> > > > > > > > > Shape
> > > > > > > > > > > Blue
> > > > > > > > > > > >> > Ltd.
> > > > > > > > > > > >> > > > > > > ShapeBlue
> > > > > > > > > > > >> > > > > > > > > is
> > > > > > > > > > > >> > > > > > > > > > a registered trademark.
> > > > > > > > > > > >> > > > > > > > > > This email and any attachments to
> it
> > > may
> > > > > be
> > > > > > > > > > > confidential
> > > > > > > > > > > >> > and
> > > > > > > > > > > >> > > > are
> > > > > > > > > > > >> > > > > > > > intended
> > > > > > > > > > > >> > > > > > > > > > solely for the use of the
> individual
> > > to
> > > > > whom
> > > > > > > it
> > > > > > > > is
> > > > > > > > > > > >> > addressed.
> > > > > > > > > > > >> > > > Any
> > > > > > > > > > > >> > > > > > > views
> > > > > > > > > > > >> > > > > > > > > or
> > > > > > > > > > > >> > > > > > > > > > opinions expressed are solely
> those
> > of
> > > > the
> > > > > > > > author
> > > > > > > > > > and
> > > > > > > > > > > do
> > > > > > > > > > > >> > not
> > > > > > > > > > > >> > > > > > > > necessarily
> > > > > > > > > > > >> > > > > > > > > > represent those of Shape Blue Ltd
> or
> > > > > related
> > > > > > > > > > > companies.
> > > > > > > > > > > >> If
> > > > > > > > > > > >> > > you
> > > > > > > > > > > >> > > > > are
> > > > > > > > > > > >> > > > > > > not
> > > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > intended recipient of this email,
> > you
> > > > must
> > > > > > > > neither
> > > > > > > > > > > take
> > > > > > > > > > > >> any
> > > > > > > > > > > >> > > > > action
> > > > > > > > > > > >> > > > > > > > based
> > > > > > > > > > > >> > > > > > > > > > upon its contents, nor copy or
> show
> > it
> > > > to
> > > > > > > > anyone.
> > > > > > > > > > > Please
> > > > > > > > > > > >> > > > contact
> > > > > > > > > > > >> > > > > > the
> > > > > > > > > > > >> > > > > > > > > sender
> > > > > > > > > > > >> > > > > > > > > > if you believe you have received
> > this
> > > > > email
> > > > > > in
> > > > > > > > > > error.
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > -----Original Message-----
> > > > > > > > > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > > > > >> > > > > <javascript:;> <javascript:;>]
> > > > > > > > > > > >> > > > > > > > > > Sent: Friday, February 5, 2016
> 4:58
> > PM
> > > > > > > > > > > >> > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > <javascript:;>
> > > > > > > > > > > >> > <javascript:;>
> > > > > > > > > > > >> > > > > > > > > > Subject: Re: [Propose][New
> Feature]
> > > > > Storage
> > > > > > > > > > Snapshots
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Paul,
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > When you say actual backups, how
> > would
> > > > it
> > > > > be
> > > > > > > > > > different
> > > > > > > > > > > >> from
> > > > > > > > > > > >> > > the
> > > > > > > > > > > >> > > > > > > Volume
> > > > > > > > > > > >> > > > > > > > > > Snapshots that exist currently. My
> > > > > > > understanding
> > > > > > > > > is
> > > > > > > > > > > that
> > > > > > > > > > > >> > > > Backups
> > > > > > > > > > > >> > > > > > end
> > > > > > > > > > > >> > > > > > > up
> > > > > > > > > > > >> > > > > > > > > in
> > > > > > > > > > > >> > > > > > > > > > Sec Storage whereas Snapshots are
> > > just a
> > > > > > > > > > point-in-time
> > > > > > > > > > > >> > state
> > > > > > > > > > > >> > > of
> > > > > > > > > > > >> > > > > > your
> > > > > > > > > > > >> > > > > > > > > volume
> > > > > > > > > > > >> > > > > > > > > > which can be restored back
> correct?
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > -Syed
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM,
> > Paul
> > > > > Angus
> > > > > > <
> > > > > > > > > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> > > > > > > > <javascript:;>>
> > > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Hi Syed,
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > As I understand it, the
> SolidFire
> > > > plugin
> > > > > > > will
> > > > > > > > > > export
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > snapshot
> > > > > > > > > > > >> > > > > > > to
> > > > > > > > > > > >> > > > > > > > > > > secondary storage if the user
> > > > requests a
> > > > > > > > > template
> > > > > > > > > > > from
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > > snapshot
> > > > > > > > > > > >> > > > > > > > or
> > > > > > > > > > > >> > > > > > > > > > > wants to download the snapshot
> > from
> > > > the
> > > > > > > cloud.
> > > > > > > > > > This
> > > > > > > > > > > >> is a
> > > > > > > > > > > >> > > > good,
> > > > > > > > > > > >> > > > > > > > > > > pragmatic approach and yes Mike
> > the
> > > > > > > SolidFire
> > > > > > > > > > > storage
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > > > super
> > > > > > > > > > > >> > > > > > > > > > > reliable and snapshots on
> > SolidFire
> > > > > arrays
> > > > > > > > take
> > > > > > > > > up
> > > > > > > > > > > >> next
> > > > > > > > > > > >> > to
> > > > > > > > > > > >> > > no
> > > > > > > > > > > >> > > > > > > space.
> > > > > > > > > > > >> > > > > > > > > > > BUT I think that we are talking
> > > about
> > > > a
> > > > > > more
> > > > > > > > > > general
> > > > > > > > > > > >> > > purpose
> > > > > > > > > > > >> > > > > API,
> > > > > > > > > > > >> > > > > > > and
> > > > > > > > > > > >> > > > > > > > > > > other storage systems may not be
> > as
> > > > > > awesome
> > > > > > > as
> > > > > > > > > > > Mike's.
> > > > > > > > > > > >> > > That's
> > > > > > > > > > > >> > > > > my
> > > > > > > > > > > >> > > > > > > > > > > concern. Also, the time to
> > transfer
> > > > for
> > > > > > say
> > > > > > > > 1TB
> > > > > > > > > to
> > > > > > > > > > > >> move
> > > > > > > > > > > >> > > from
> > > > > > > > > > > >> > > > > > > primary
> > > > > > > > > > > >> > > > > > > > > > > to sec storage and then create a
> > VM
> > > > > > template
> > > > > > > > out
> > > > > > > > > > of
> > > > > > > > > > > it
> > > > > > > > > > > >> > may
> > > > > > > > > > > >> > > be
> > > > > > > > > > > >> > > > > too
> > > > > > > > > > > >> > > > > > > > long
> > > > > > > > > > > >> > > > > > > > > > for users.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the
> > > term
> > > > > > volume
> > > > > > > > > > > snapshot
> > > > > > > > > > > >> for
> > > > > > > > > > > >> > > > > backup,
> > > > > > > > > > > >> > > > > > > > it's
> > > > > > > > > > > >> > > > > > > > > > > just that users want to do
> backups
> > > > and a
> > > > > > > > volume
> > > > > > > > > > > >> snapshot
> > > > > > > > > > > >> > is
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > > only
> > > > > > > > > > > >> > > > > > > > > > > type of snapshot that copies the
> > > disk
> > > > > > > > elsewhere
> > > > > > > > > > and
> > > > > > > > > > > >> can
> > > > > > > > > > > >> > be
> > > > > > > > > > > >> > > > > > > scheduled.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > I'm 'pondering' the implications
> > of
> > > > > > enabling
> > > > > > > > > > actual
> > > > > > > > > > > >> > backups
> > > > > > > > > > > >> > > > > > > (through
> > > > > > > > > > > >> > > > > > > > > > > recognised backup providers) and
> > the
> > > > > user
> > > > > > > > > > > requirements
> > > > > > > > > > > >> > > around
> > > > > > > > > > > >> > > > > > them
> > > > > > > > > > > >> > > > > > > > > > > (particularly restoration use
> > cases)
> > > > as
> > > > > a
> > > > > > > > > separate
> > > > > > > > > > > >> thread
> > > > > > > > > > > >> > > of
> > > > > > > > > > > >> > > > > > work.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > [image: ShapeBlue] <
> > > > > > > http://www.shapeblue.com>
> > > > > > > > > > Paul
> > > > > > > > > > > >> Angus
> > > > > > > > > > > >> > > VP
> > > > > > > > > > > >> > > > > > > > Technology
> > > > > > > > > > > >> > > > > > > > > > > , ShapeBlue
> > > > > > > > > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44
> 203
> > > 603
> > > > > > 0540*
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > >
> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > > > > > >> > > > |
> > > > > > > > > > > >> > > > > m:
> > > > > > > > > > > >> > > > > > > > > > > *+44 7711 418784*
> > > > <+44%207711%20418784>
> > > > > > > > > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > > > > <javascript:;>
> > > > > > > > > > > >> > > <javascript:;> |
> > > > > > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > > > > > >> > > > > > > > > > > <paul.angus@shapeblue.com
> > > > > <javascript:;>
> > > > > > > > > > > >> <javascript:;>
> > > > > > > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > > > >> > > > > > > > > > > *www.shapeblue.com* <
> > > > > > > http://www.shapeblue.com
> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent
> Garden
> > > > > London
> > > > > > > WC2N
> > > > > > > > > 4HS
> > > > > > > > > > > UK
> > > > > > > > > > > >> > Shape
> > > > > > > > > > > >> > > > > Blue
> > > > > > > > > > > >> > > > > > > Ltd
> > > > > > > > > > > >> > > > > > > > > > > is a company incorporated in
> > > England &
> > > > > > > Wales.
> > > > > > > > > > > >> ShapeBlue
> > > > > > > > > > > >> > > > > Services
> > > > > > > > > > > >> > > > > > > > India
> > > > > > > > > > > >> > > > > > > > > > > LLP is a company incorporated in
> > > India
> > > > > and
> > > > > > > is
> > > > > > > > > > > operated
> > > > > > > > > > > >> > > under
> > > > > > > > > > > >> > > > > > > license
> > > > > > > > > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue
> > > Brasil
> > > > > > > > > Consultoria
> > > > > > > > > > > >> Ltda
> > > > > > > > > > > >> > is
> > > > > > > > > > > >> > > a
> > > > > > > > > > > >> > > > > > > company
> > > > > > > > > > > >> > > > > > > > > > > incorporated in Brasil and is
> > > operated
> > > > > > under
> > > > > > > > > > license
> > > > > > > > > > > >> from
> > > > > > > > > > > >> > > > Shape
> > > > > > > > > > > >> > > > > > > Blue
> > > > > > > > > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a
> > > company
> > > > > > > > > registered
> > > > > > > > > > by
> > > > > > > > > > > >> The
> > > > > > > > > > > >> > > > > Republic
> > > > > > > > > > > >> > > > > > > of
> > > > > > > > > > > >> > > > > > > > > > > South Africa and is traded under
> > > > license
> > > > > > > from
> > > > > > > > > > Shape
> > > > > > > > > > > >> Blue
> > > > > > > > > > > >> > > Ltd.
> > > > > > > > > > > >> > > > > > > > > > > ShapeBlue is a registered
> > trademark.
> > > > > > > > > > > >> > > > > > > > > > > This email and any attachments
> to
> > it
> > > > may
> > > > > > be
> > > > > > > > > > > >> confidential
> > > > > > > > > > > >> > > and
> > > > > > > > > > > >> > > > > are
> > > > > > > > > > > >> > > > > > > > > > > intended solely for the use of
> the
> > > > > > > individual
> > > > > > > > to
> > > > > > > > > > > whom
> > > > > > > > > > > >> it
> > > > > > > > > > > >> > is
> > > > > > > > > > > >> > > > > > > > addressed.
> > > > > > > > > > > >> > > > > > > > > > > Any views or opinions expressed
> > are
> > > > > solely
> > > > > > > > those
> > > > > > > > > > of
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > author
> > > > > > > > > > > >> > > > > > and
> > > > > > > > > > > >> > > > > > > do
> > > > > > > > > > > >> > > > > > > > > > > not necessarily represent those
> of
> > > > Shape
> > > > > > > Blue
> > > > > > > > > Ltd
> > > > > > > > > > or
> > > > > > > > > > > >> > > related
> > > > > > > > > > > >> > > > > > > > > > > companies. If you are not the
> > > intended
> > > > > > > > recipient
> > > > > > > > > > of
> > > > > > > > > > > >> this
> > > > > > > > > > > >> > > > email,
> > > > > > > > > > > >> > > > > > you
> > > > > > > > > > > >> > > > > > > > > > > must neither take any action
> based
> > > > upon
> > > > > > its
> > > > > > > > > > > contents,
> > > > > > > > > > > >> nor
> > > > > > > > > > > >> > > > copy
> > > > > > > > > > > >> > > > > or
> > > > > > > > > > > >> > > > > > > > show
> > > > > > > > > > > >> > > > > > > > > > > it to anyone. Please contact the
> > > > sender
> > > > > if
> > > > > > > you
> > > > > > > > > > > believe
> > > > > > > > > > > >> > you
> > > > > > > > > > > >> > > > have
> > > > > > > > > > > >> > > > > > > > > received
> > > > > > > > > > > >> > > > > > > > > > this email in error.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > > > > >> > > > > <javascript:;>
> > > > > > > > > > > >> > > > > > <javascript:;>]
> > > > > > > > > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > > > > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > <javascript:;>
> > > > > > > > > > > >> > > <javascript:;>
> > > > > > > > > > > >> > > > > > > > > > > Subject: Re: [Propose][New
> > Feature]
> > > > > > Storage
> > > > > > > > > > > Snapshots
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > I think the terminology
> confusion
> > > > comes
> > > > > > from
> > > > > > > > AWS
> > > > > > > > > > > where
> > > > > > > > > > > >> > they
> > > > > > > > > > > >> > > > do
> > > > > > > > > > > >> > > > > > EBS
> > > > > > > > > > > >> > > > > > > > > > > snapshots backed up to S3 and
> > > > CloudStack
> > > > > > > sort
> > > > > > > > of
> > > > > > > > > > > >> followed
> > > > > > > > > > > >> > > > that.
> > > > > > > > > > > >> > > > > > And
> > > > > > > > > > > >> > > > > > > > as
> > > > > > > > > > > >> > > > > > > > > > > an end user who is oblivious to
> > the
> > > > > > > internals
> > > > > > > > of
> > > > > > > > > > my
> > > > > > > > > > > >> > > provider,
> > > > > > > > > > > >> > > > > my
> > > > > > > > > > > >> > > > > > > > > > > expectation would be something
> > > similar
> > > > > to
> > > > > > > what
> > > > > > > > > AWS
> > > > > > > > > > > as
> > > > > > > > > > > >> > that
> > > > > > > > > > > >> > > is
> > > > > > > > > > > >> > > > > my
> > > > > > > > > > > >> > > > > > > > > > > biggest reference point.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > To your point Mike, I agree
> that a
> > > > > Primary
> > > > > > > > > Storage
> > > > > > > > > > > >> > failure
> > > > > > > > > > > >> > > on
> > > > > > > > > > > >> > > > > > > > > > > SolidFire is unlikely, there are
> > > other
> > > > > > > > > motivations
> > > > > > > > > > > >> for us
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > push
> > > > > > > > > > > >> > > > > > > > data
> > > > > > > > > > > >> > > > > > > > > > > to secondary storage. Primary
> > > storage
> > > > > > > (atleast
> > > > > > > > > for
> > > > > > > > > > > us)
> > > > > > > > > > > >> > > costs
> > > > > > > > > > > >> > > > > > > around 3
> > > > > > > > > > > >> > > > > > > > > > > times as much as secondary
> storage
> > > and
> > > > > > > > snapshots
> > > > > > > > > > on
> > > > > > > > > > > >> > primary<> > > > > > > > take 2
> > > > > > > > > > > >> > days
> > > > > > > > > > > >> > > > to
> > > > > > > > > > > >> > > > > > > > > complete.
> > > > > > > > > > > >> > > > > > > > > > > > > > e. snapshots (as they are)
> > > can't
> > > > > be
> > > > > > > used
> > > > > > > > > for
> > > > > > > > > > > >> > multiple
> > > > > > > > > > > >> > > > LVM
> > > > > > > > > > > >> > > > > > > > disks.
> > > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > > > I think the process Mike
> has
> > > > used
> > > > > in
> > > > > > > the
> > > > > > > > > > > >> SolidFire
> > > > > > > > > > > >> > > > plugin
> > > > > > > > > > > >> > > > > > > (only
> > > > > > > > > > > >> > > > > > > > > > > > > > moving
> > > > > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > > > > > disk image to secondary
> > > storage
> > > > > when
> > > > > > > you
> > > > > > > > > > > >> absolutely
> > > > > > > > > > > >> > > > have
> > > > > > > > > > > >> > > > > > to)
> > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > >> > > > > > > > > > > > > > a very
> > > > > > > > > > > >> > > > > > > > > > > > > good
> > > > > > > > > > > >> > > > > > > > > > > > > > and pragmatic solution. I
> > > wonder
> > > > > > what
> > > > > > > > > > problems
> > > > > > > > > > > >> an
> > > > > > > > > > > >> > > > > operator
> > > > > > > > > > > >> > > > > > > > might
> > > > > > > > > > > >> > > > > > > > > > > > > experience
> > > > > > > > > > > >> > > > > > > > > > > > > > if they have an issue
> with a
> > > > given
> > > > > > > > primary
> > > > > > > > > > > >> storage
> > > > > > > > > > > >> > > pool
> > > > > > > > > > > >> > > > > in
> > > > > > > > > > > >> > > > > > a
> > > > > > > > > > > >> > > > > > > > > > cluster.
> > > > > > > > > > > >> > > > > > > > > > > > (I
> > > > > > > > > > > >> > > > > > > > > > > > > > know that that is REALLY
> > > > unlikely
> > > > > in
> > > > > > > the
> > > > > > > > > > > >> SolidFire
> > > > > > > > > > > >> > > case
> > > > > > > > > > > >> > > > > > Mike
> > > > > > > > > > > >> > > > > > > :)
> > > > > > > > > > > >> > > > > > > > > > > > > > ) And
> > > > > > > > > > > >> > > > > > > > > > > > if
> > > > > > > > > > > >> > > > > > > > > > > > > > the transfer from primary
> to
> > > > > > secondary
> > > > > > > > is
> > > > > > > > > > > slow,
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > time
> > > > > > > > > > > >> > > > > to
> > > > > > > > > > > >> > > > > > > > > > > > > > being able
> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > [image: Shape
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

Correct. Keeping the snapshot on managed storage should be the default
behaviour as it will not break backward compatibility.

-Syed


On Fri, Feb 12, 2016 at 12:02 PM, Mike Tutkowski <
mike.tutkowski@solidfire.com> wrote:

> I think so.
>
> Just to confirm: The default behavior for a volume snapshot with managed
> storage is equivalent to a storage snapshot.
>
> When the global setting value is changed, the data ends up on secondary
> storage (NFS).
>
> Is that accurate?
>
> On Friday, February 12, 2016, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
> > > Situation: Global setting value is changed to a non-default value.
> > >
> > > 2) Action: Volume snapshot is taken with managed storage. Result: ???
> I'm
> > > not sure which scenario we're looking at here now.
> > >
> >
> > In this situation, volume snapshots will end up on secondary storage and
> > the snapshot on the managed storage will be deleted.
> >
> >
> > >
> > > 3) Action: Storage snapshot is taken with managed storage. Result: ???
> > I'm
> > > not sure which scenario we're looking at here now.
> > >
> >
> > In this situation, the snapshot stays on the managed storage.
> >
> > Does this answer your question Mike?
> >
> >
> >
> >
> >
> > On Fri, Feb 12, 2016 at 11:43 AM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com <javascript:;>> wrote:
> >
> > > Hi Syed,
> > >
> > > Can you clarify how you see these behaviors (below) working now that we
> > are
> > > considering a global settings value applicable to managed storage with
> > > volume snapshots?
> > >
> > > State: Global setting value defaults to keep snapshot on storage system
> > > (primary storage) for managed storage.
> > >
> > > 1) Action: Volume snapshot is taken with managed storage. Result:
> > Snapshot
> > > is taken and kept on storage system (primary storage). (This is what
> you
> > > and I have been working on for a couple months now.)
> > >
> > >
> > >
> > > Thanks!
> > > Mike
> > >
> > > On Fri, Feb 12, 2016 at 9:15 AM, Syed Mushtaq <syed1.mushtaq@gmail.com
> >
> > > wrote:
> > >
> > > > I agree with Mike's concern about backward compatibility. We can add
> a
> > > > global flag which makes sure that the way volume snapshots work
> > currently
> > > > on managed storage (stay on the device) is retained after upgrade. We
> > can
> > > > then safely implement the Storage Snapshot API while making the
> Volume
> > > > Snpashot API move the snapshot to Secondary Storage.
> > > >
> > > > Sounds good guys?
> > > >
> > > > -Syed
> > > >
> > > >
> > > > On Mon, Feb 8, 2016 at 2:56 PM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com
> > > > > wrote:
> > > >
> > > > > Here's what we have for snapshots for managed storage as of 4.6,
> > Paul:
> > > > >
> > > > > 1. VM snapshots (no proposed changes to this).
> > > > >
> > > > > 2. Volume snapshots that do not end up on secondary storage, but
> > rather
> > > > are
> > > > > stored on a SAN (effectively storing snapshots on primary storage).
> > > > >
> > > > > Pierre-Luc is saying he'd like this for snapshots for managed
> > storage:
> > > > >
> > > > > A. VM snapshots (no proposed changes to this).
> > > > >
> > > > > B. Volume snapshots that export to secondary storage.
> > > > >
> > > > > C. New: Storage snapshots that behave like 2 (above).
> > > > >
> > > > > I like Pierre-Luc's ideas there, but the problem is backward
> > > > compatibility.
> > > > >
> > > > > If customers who were using managed storage with volume snapshots
> in
> > > 4.6
> > > > > were getting their snapshots put on a SAN, then in 4.9 - all of a
> > > sudden
> > > > -
> > > > > their new snapshots are put on secondary storage (unless they
> > > explicitly
> > > > > change over to using the new Storage snapshots feature).
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 8, 2016 at 12:32 PM, Paul Angus <
> > paul.angus@shapeblue.com>
> > > > > wrote:
> > > > >
> > > > > > Just to make sure I'm on the same page, are we talking about;
> > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9278 ?
> > > > > >
> > > > > > The FS reads (to me) more like 1a + the possibility to export to
> > > > > secondary
> > > > > > storage if required?
> > > > > > Have I understood correctly?
> > > > > > I have seen [1a] implemented for VMware by NetApp in their beta
> > > > > CloudStack
> > > > > > plugin (pleased I can say that without Mike beating me up now).
> No
> > > > > changes
> > > > > > to the CloudStack API were required. (nb it didn't export to
> > > secondary
> > > > > > storage).
> > > > > >
> > > > > >
> > > > > >
> > > > > > 1. VM Snapshot (point-in-time hypervisor based snapshots)
> > > > > > 1a. SAN assisted VM snapshots (point-in-time hypervisor snapshot
> > > takes
> > > > > > place on transparently SAN to avoid performance issue in disk
> > chains)
> > > > > > 2. SAN Snapshot (Storage Snapshot) - NEW
> > > > > > 3. Volume Snapshot (current old/slow transfer to secstorage)
> > > > > > 4. Backup - JUST AN IDEAL.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > > Paul Angus
> > > > > > VP Technology ,  ShapeBlue
> > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> m:
> > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > > > <pa...@cloudyangus>  |  w:
> > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > > Shape Blue Ltd is a company incorporated in England & Wales.
> > > ShapeBlue
> > > > > > Services India LLP is a company incorporated in India and is
> > operated
> > > > > under
> > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> is
> > a
> > > > > > company incorporated in Brasil and is operated under license from
> > > Shape
> > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > > Republic
> > > > of
> > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > ShapeBlue
> > > > > is
> > > > > > a registered trademark.
> > > > > > This email and any attachments to it may be confidential and are
> > > > intended
> > > > > > solely for the use of the individual to whom it is addressed. Any
> > > views
> > > > > or
> > > > > > opinions expressed are solely those of the author and do not
> > > > necessarily
> > > > > > represent those of Shape Blue Ltd or related companies. If you
> are
> > > not
> > > > > the
> > > > > > intended recipient of this email, you must neither take any
> action
> > > > based
> > > > > > upon its contents, nor copy or show it to anyone. Please contact
> > the
> > > > > sender
> > > > > > if you believe you have received this email in error.
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
> > > > > > Sent: Monday, February 8, 2016 7:16 PM
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > >
> > > > > > Correct, Will.
> > > > > >
> > > > > > That Global Settings would only be for managed storage.
> Non-managed
> > > > > > (traditional) volume snapshots are completely un-impacted by this
> > > > > feature.
> > > > > >
> > > > > > If we need to sometimes keep the snapshots on the SAN and
> sometimes
> > > > push
> > > > > > them to secondary storage, we'll need a more robust solution than
> > > > Global
> > > > > > Settings, though.
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <
> > wstevens@cloudops.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Sorry. I missed a bit of context when I responded. The global
> > > setting
> > > > > > > would be only for the managed storage case, currently being
> > called
> > > > > > Storage
> > > > > > > Snapshots, and is only to determine if a copy is pushed to
> > > secondary
> > > > > > > storage right? The global setting would not change the behavior
> > of
> > > > the
> > > > > > > Volume Snapshots right?
> > > > > > >
> > > > > > > I was referring to the need for Volume Snapshots and Storage
> > > > Snapshots
> > > > > to
> > > > > > > exist together. Disregard my comment. I caught up on context
> > after
> > > I
> > > > > > > posted. My bad...
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > > > w cloudops.com *|* tw @CloudOps_
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> > > > > > > mike.tutkowski@solidfire.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Will,
> > > > > > > >
> > > > > > > > Who's picking the behavior? Is it the cloud provider or the
> end
> > > > user?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <
> > > > wstevens@cloudops.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I don't think a global setting is a good option because we
> > need
> > > > > both
> > > > > > > > > functionality to be available at the same time and for
> > > different
> > > > > use
> > > > > > > > cases
> > > > > > > > > to be able to pick which they choose.
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > > > > > w cloudops.com *|* tw @CloudOps_
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > > > > > > > mike.tutkowski@solidfire.com
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Now that I re-read your e-mail, it dawned on me: The
> > end-user
> > > > > > doesn't
> > > > > > > > > care
> > > > > > > > > > where the snapshot is.
> > > > > > > > > >
> > > > > > > > > > If that's true, then we should perhaps control this via
> > > Global
> > > > > > > Settings
> > > > > > > > > or
> > > > > > > > > > something.
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > > > > > > > mike.tutkowski@solidfire.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > It's not ideal - true, but it does allow us to be
> > backward
> > > > > > > > compatible.
> > > > > > > > > > >
> > > > > > > > > > > If you have other ideas, though, about how to maintain
> > > > backward
> > > > > > > > > > > compatibility, I'm definitely open to hear them.
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi Mike,
> > > > > > > > > > >>
> > > > > > > > > > >> Adding a flag to createSnapshot was the first and the
> > most
> > > > > > obvious
> > > > > > > > > thing
> > > > > > > > > > >> that came to our minds. The problem that I had with
> this
> > > was
> > > > > > that:
> > > > > > > > > > >>
> > > > > > > > > > >> 1) I feel it is exposing something to the end user
> that
> > is
> > > > > > > internal
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > >> cloud.
> > > > > > > > > > >>
> > > > > > > > > > >> 2) We have to follow two different ways of
> > > restore/deletion
> > > > in
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > >> code path depending on where the Snapshot resides
> which
> > I
> > > > find
> > > > > > > kind
> > > > > > > > > of a
> > > > > > > > > > >> bad design.
> > > > > > > > > > >>
> > > > > > > > > > >> But if exposing a archive flag to the end user is
> > > acceptable
> > > > > > then
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > >> definitely use this instead of adding the
> > StorageSnapshot
> > > > API
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> -Syed
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > > > > > > > >> mike.tutkowski@solidfire.com
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi Pierre-Luc,
> > > > > > > > > > >> >
> > > > > > > > > > >> > My recommendation would be this:
> > > > > > > > > > >> >
> > > > > > > > > > >> > Add an "archive" flag to the current volume-snapshot
> > > API.
> > > > > Its
> > > > > > > > > default
> > > > > > > > > > >> would
> > > > > > > > > > >> > be "false" because that would be backward compatible
> > > with
> > > > > how
> > > > > > > 4.6
> > > > > > > > > has
> > > > > > > > > > >> > volume snapshots implemented (i.e. they stay on the
> > SAN
> > > in
> > > > > > 4.6,
> > > > > > > > 4.7,
> > > > > > > > > > and
> > > > > > > > > > >> > 4.8).
> > > > > > > > > > >> >
> > > > > > > > > > >> > If you set archive=true, then we would perform a
> > > > background
> > > > > > > > > migration
> > > > > > > > > > of
> > > > > > > > > > >> > the snapshot from the SAN to the secondary storage
> > (then
> > > > > > delete
> > > > > > > > the
> > > > > > > > > > SAN
> > > > > > > > > > >> > snapshot).
> > > > > > > > > > >> >
> > > > > > > > > > >> > That archive parameter would only be valid for
> managed
> > > > > > storage.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Sound reasonable?
> > > > > > > > > > >> >
> > > > > > > > > > >> > Also, a VM snapshot that includes disks provided by
> > > > managed
> > > > > > > > storage
> > > > > > > > > > >> should
> > > > > > > > > > >> > work.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Talk to you later,
> > > > > > > > > > >> > Mike
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > > > > > > > pdion@cloudops.com
> > > > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Mike,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > In terms of API's, would you prefer introducing a
> > > > > parameter
> > > > > > to
> > > > > > > > the
> > > > > > > > > > >> > existing
> > > > > > > > > > >> > > VolumeSnapshot, example: extract={true|false}
> with a
> > > > > > > default
> > > > > > > > > > value
> > > > > > > > > > >> of
> > > > > > > > > > >> > > true which would extract snapshot into the
> secondary
> > > > > > storage,
> > > > > > > > > which
> > > > > > > > > > >> is
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > current default behavior. Then for SAN snapshot
> that
> > > > > remain
> > > > > > on
> > > > > > > > the
> > > > > > > > > > >> SAN we
> > > > > > > > > > >> > > would just set "extract=false" ? as oppose to
> > create a
> > > > new
> > > > > > > > > > >> > > StorageSnapshot API ?
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Paul,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > From what I'm seeing so far, we can't do a
> > VM-snapshot
> > > > > when
> > > > > > > > using
> > > > > > > > > > >> managed
> > > > > > > > > > >> > > storage for VM having more than one Volume. For
> the
> > > > reason
> > > > > > > that
> > > > > > > > > > >> snapshot
> > > > > > > > > > >> > > are performed outside of the hypervisor awareness
> > and
> > > > > > > > > > asynchronously.
> > > > > > > > > > >> If
> > > > > > > > > > >> > > someone have a way to address that, it would make
> > > thinks
> > > > > > much
> > > > > > > > more
> > > > > > > > > > >> > > attractive.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <
> > > > > irae@cloudops.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > I think a service provider backup scenario is
> more
> > > > > likely
> > > > > > to
> > > > > > > > > take
> > > > > > > > > > >> > > advantage
> > > > > > > > > > >> > > > of SAN snapshot. There are a few reasons for
> this.
> > > > > > > Traditional
> > > > > > > > > > >> backups
> > > > > > > > > > >> > > > involve access to the file system, and there is
> an
> > > > > > > expectation
> > > > > > > > > > that
> > > > > > > > > > >> > this
> > > > > > > > > > >> > > > can be done with reasonably short time frames
> > > without
> > > > > > > > negatively
> > > > > > > > > > >> > > impacting
> > > > > > > > > > >> > > > VM performance, and that the backup orchestrator
> > can
> > > > > apply
> > > > > > > > > various
> > > > > > > > > > >> > logic
> > > > > > > > > > >> > > > and or transformations to the data (compress,
> > > encrypt,
> > > > > > > deltas
> > > > > > > > > > >> etc...).
> > > > > > > > > > >> > > > While it is true that one could apply a backup
> > > process
> > > > > to
> > > > > > a
> > > > > > > > > cloud
> > > > > > > > > > >> > > snapshot,
> > > > > > > > > > >> > > > this would be slow and inefficient requiring the
> > > data
> > > > to
> > > > > > be
> > > > > > > > > moved
> > > > > > > > > > >> > several
> > > > > > > > > > >> > > > times and there are some major bottlenecks with
> > > cloud
> > > > > > > > snapshots.
> > > > > > > > > > >> With a
> > > > > > > > > > >> > > > cloud snapshot - there seems to be no reasonable
> > > > > > expectation
> > > > > > > > of
> > > > > > > > > > >> being
> > > > > > > > > > >> > > able
> > > > > > > > > > >> > > > to do differential snapshots (I think this
> depends
> > > on
> > > > > the
> > > > > > > > > > >> hypervisor)
> > > > > > > > > > >> > and
> > > > > > > > > > >> > > > if you do differential snapshots this will make
> > file
> > > > > > backups
> > > > > > > > > from
> > > > > > > > > > >> those
> > > > > > > > > > >> > > > snapshots even more complicated to orchestrate.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Suspect there needs to be a different thread on
> > how
> > > to
> > > > > > > better
> > > > > > > > > > enable
> > > > > > > > > > >> > > > backups, as a service. As per Paul's suggestion,
> > but
> > > > it
> > > > > > is a
> > > > > > > > > > related
> > > > > > > > > > >> > > > workflow so relevant to this discussion.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Ian
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > > > > > > > >> > > mike.tutkowski@solidfire.com>
> > > > > > > > > > >> > > > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > To me it sounds like number two and number
> three
> > > are
> > > > > > > > different
> > > > > > > > > > >> uses
> > > > > > > > > > >> > for
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > same "thing"(which is totally fine).
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > As for taking a fast SAN snapshot and
> exporting
> > it
> > > > > > > > > > >> asynchronously, do
> > > > > > > > > > >> > > we
> > > > > > > > > > >> > > > > see the SSVM as performing the export?
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > To be backwards compatible with what we have
> in
> > > 4.6
> > > > > and
> > > > > > > > later
> > > > > > > > > > for
> > > > > > > > > > >> > > volume
> > > > > > > > > > >> > > > > snapshots for managed storage, I think it
> might
> > be
> > > > > > easier
> > > > > > > if
> > > > > > > > > we
> > > > > > > > > > >> pass
> > > > > > > > > > >> > > in a
> > > > > > > > > > >> > > > > flag that says whether or not to archive the
> SAN
> > > > > > snapshot
> > > > > > > > > > (which,
> > > > > > > > > > >> I
> > > > > > > > > > >> > > > think,
> > > > > > > > > > >> > > > > is something that you suggested, as well,
> > > > Pierre-Luc).
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > > > > > > > pdion@cloudops.com
> > > > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > Hi Mike,
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > The reason behind the creation of a SAN
> > snapshot
> > > > > which
> > > > > > > is
> > > > > > > > > > >> exported
> > > > > > > > > > >> > > into
> > > > > > > > > > >> > > > > > secondary storage, is because creating a
> copy
> > of
> > > > the
> > > > > > > .VHD
> > > > > > > > > > >> directly
> > > > > > > > > > >> > > > would
> > > > > > > > > > >> > > > > > impact uptime of the VM as creating that
> copy
> > > take
> > > > > > lots
> > > > > > > of
> > > > > > > > > > time.
> > > > > > > > > > >> > Has
> > > > > > > > > > >> > > > > oppose
> > > > > > > > > > >> > > > > > to a SAN snapshot that is close to
> > instantaneous
> > > > > which
> > > > > > > can
> > > > > > > > > > >> > afterward
> > > > > > > > > > >> > > be
> > > > > > > > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > I would suspect an extracted VolumeSnapshot
> > > taken
> > > > > > from a
> > > > > > > > SAN
> > > > > > > > > > >> > snapshot
> > > > > > > > > > >> > > > > could
> > > > > > > > > > >> > > > > > have is SAN snapshot deleted to avoid
> duplica
> > > and
> > > > > > space
> > > > > > > > > > >> consumption
> > > > > > > > > > >> > > on
> > > > > > > > > > >> > > > > the
> > > > > > > > > > >> > > > > > Primary Storage side.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > I see 3 definitions in our current
> discussion
> > > > > > regarding
> > > > > > > > the
> > > > > > > > > > term
> > > > > > > > > > >> > > > snapshot
> > > > > > > > > > >> > > > > > (these are not official terminology but by
> own
> > > > > > > > > interpretation
> > > > > > > > > > of
> > > > > > > > > > >> > > them):
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot /
> Mike's
> > > > > > definition
> > > > > > > > of
> > > > > > > > > a
> > > > > > > > > > >> > > > snapshot):
> > > > > > > > > > >> > > > > > it's a volume snapshot at the storage level,
> > > point
> > > > > in
> > > > > > > time
> > > > > > > > > of
> > > > > > > > > > >> your
> > > > > > > > > > >> > > > data.
> > > > > > > > > > >> > > > > it
> > > > > > > > > > >> > > > > > reside on the primary storage. Useful and
> > > > efficient
> > > > > > for
> > > > > > > > > > software
> > > > > > > > > > >> > side
> > > > > > > > > > >> > > > > > incident.
> > > > > > > > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack
> > > > > VolumeSnapshot/
> > > > > > > > cloud
> > > > > > > > > > >> backup
> > > > > > > > > > >> > > > aws-S3
> > > > > > > > > > >> > > > > > style ): Point in time copy of the Virtual
> > Disk
> > > > that
> > > > > > > > reside
> > > > > > > > > > on a
> > > > > > > > > > >> > > > > different
> > > > > > > > > > >> > > > > > storage array then the original Volume.
> > > Facilitate
> > > > > > data
> > > > > > > > > > >> migration
> > > > > > > > > > >> > > > between
> > > > > > > > > > >> > > > > > clusters and, in case of primary storage
> > > incident,
> > > > > > > Volume
> > > > > > > > > > >> snapshots
> > > > > > > > > > >> > > are
> > > > > > > > > > >> > > > > not
> > > > > > > > > > >> > > > > > impacted and can be reuse.
> > > > > > > > > > >> > > > > > 3. *Backup*: Archival of your
> Virtual-machines
> > > > data
> > > > > > that
> > > > > > > > > also
> > > > > > > > > > >> > > validate
> > > > > > > > > > >> > > > > data
> > > > > > > > > > >> > > > > > integrity, provide a storage efficient
> > archiving
> > > > > > method
> > > > > > > > and
> > > > > > > > > an
> > > > > > > > > > >> > > > > independent
> > > > > > > > > > >> > > > > > way to restore your data in case of an major
> > > > > > > > infrastructure
> > > > > > > > > > >> > disaster.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Regards,
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > PL
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike
> > Tutkowski <
> > > > > > > > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > <javascript:;>
> > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > So, let's see if I currently follow the
> > > > > > requirements:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > * Augment volume snapshots for managed
> > storage
> > > > to
> > > > > > > > > > >> conditionally
> > > > > > > > > > >> > > > export
> > > > > > > > > > >> > > > > > data
> > > > > > > > > > >> > > > > > > to NFS. The current process of taking a
> > > snapshot
> > > > > on
> > > > > > > the
> > > > > > > > > SAN
> > > > > > > > > > is
> > > > > > > > > > >> > > fine,
> > > > > > > > > > >> > > > > but
> > > > > > > > > > >> > > > > > > we'd like the option to export the data to
> > > NFS,
> > > > as
> > > > > > > well.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Questions:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Once the data has been exported to NFS, do
> > we
> > > > keep
> > > > > > the
> > > > > > > > SAN
> > > > > > > > > > >> > snapshot
> > > > > > > > > > >> > > > or
> > > > > > > > > > >> > > > > > > delete it?
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > If we are deleting the SAN snapshot, then
> > why
> > > > > don't
> > > > > > we
> > > > > > > > > just
> > > > > > > > > > >> copy
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > VHD
> > > > > > > > > > >> > > > > > > from primary to secondary the way we do
> > today
> > > > for
> > > > > > > > > > non-managed
> > > > > > > > > > >> > (i.e.
> > > > > > > > > > >> > > > > > > traditional) storage? Why create a SAN
> > > snapshot
> > > > in
> > > > > > > that
> > > > > > > > > > >> scenario?
> > > > > > > > > > >> > > > > Perhaps
> > > > > > > > > > >> > > > > > > to have the SSVM mount and perform the VHD
> > > copy
> > > > to
> > > > > > > > > secondary
> > > > > > > > > > >> > > storage
> > > > > > > > > > >> > > > > > > instead of a XenServer host?
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Thanks for the clarification.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > By the way, to me a backup is when you
> copy
> > > data
> > > > > > from
> > > > > > > > one
> > > > > > > > > > >> storage
> > > > > > > > > > >> > > > > system
> > > > > > > > > > >> > > > > > to
> > > > > > > > > > >> > > > > > > another (regardless of features, if any,
> to
> > > > > restore
> > > > > > > the
> > > > > > > > > data
> > > > > > > > > > >> in
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > > > future). A snapshot is a point-in-time
> view
> > of
> > > > the
> > > > > > > data
> > > > > > > > > of a
> > > > > > > > > > >> > volume
> > > > > > > > > > >> > > > and
> > > > > > > > > > >> > > > > > > it's stored on the same storage system as
> > the
> > > > > > volume.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM,
> Pierre-Luc
> > > > Dion <
> > > > > > > > > > >> > > pdion@cloudops.com
> > > > > > > > > > >> > > > > <javascript:;>
> > > > > > > > > > >> > > > > > <javascript:;>>
> > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > > That's fun to see that discussion
> > > happening. I
> > > > > > 100%
> > > > > > > > > agree
> > > > > > > > > > >> with
> > > > > > > > > > >> > > > Paul's
> > > > > > > > > > >> > > > > > > > points of view. VolumeSnapshot are not a
> > > > backup,
> > > > > > > but I
> > > > > > > > > do
> > > > > > > > > > >> > > consider
> > > > > > > > > > >> > > > > them
> > > > > > > > > > >> > > > > > > as
> > > > > > > > > > >> > > > > > > > a safety vest against Primary Storage
> > > failure,
> > > > > > > because
> > > > > > > > > > >> failure
> > > > > > > > > > >> > > > append
> > > > > > > > > > >> > > > > > > :-( .
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > The current proposal around snapshots
> that
> > > > > reside
> > > > > > on
> > > > > > > > the
> > > > > > > > > > >> > primary
> > > > > > > > > > >> > > > > > storage
> > > > > > > > > > >> > > > > > > or
> > > > > > > > > > >> > > > > > > > snapshots that end in the Secondary
> > Storage
> > > is
> > > > > not
> > > > > > > to
> > > > > > > > > > >> address
> > > > > > > > > > >> > > any
> > > > > > > > > > >> > > > > kind
> > > > > > > > > > >> > > > > > > of
> > > > > > > > > > >> > > > > > > > backups requirement because a snapshot
> is
> > > not
> > > > a
> > > > > > > > backup,
> > > > > > > > > > >> event
> > > > > > > > > > >> > an
> > > > > > > > > > >> > > > > > > extracted
> > > > > > > > > > >> > > > > > > > VM snapshot.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > The main idea, and again this is for
> > managed
> > > > > > > storage;
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage
> > side
> > > > > > snapshot
> > > > > > > > > > >> capability
> > > > > > > > > > >> > > for
> > > > > > > > > > >> > > > > > fast
> > > > > > > > > > >> > > > > > > > response time that support rollback to
> > > > previous
> > > > > > > > > timestamp,
> > > > > > > > > > >> > create
> > > > > > > > > > >> > > > new
> > > > > > > > > > >> > > > > > > > volume and maybe create template.
> > > > > > > > > > >> > > > > > > > not required to be a new API if the work
> > is
> > > > > > > > already
> > > > > > > > > > >> done, I
> > > > > > > > > > >> > > > think
> > > > > > > > > > >> > > > > > > this
> > > > > > > > > > >> > > > > > > > is a different behaviors than the user
> > > > > expectation
> > > > > > > of
> > > > > > > > a
> > > > > > > > > > >> > > > > > volume-snapshot.
> > > > > > > > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current
> > > > cloudstack
> > > > > > > > > behavior
> > > > > > > > > > >> that
> > > > > > > > > > >> > > > create
> > > > > > > > > > >> > > > > > an
> > > > > > > > > > >> > > > > > > > extraction of a volume into
> > SecondaryStorage
> > > > > which
> > > > > > > can
> > > > > > > > > be
> > > > > > > > > > >> reuse
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > > create a
> > > > > > > > > > >> > > > > > > > new volume into another Primary Storage.
> > > This
> > > > > type
> > > > > > > of
> > > > > > > > > > >> snapshot
> > > > > > > > > > >> > > is a
> > > > > > > > > > >> > > > > > slow
> > > > > > > > > > >> > > > > > > > job since yes it would have to copy the
> > full
> > > > > > volume
> > > > > > > > size
> > > > > > > > > > on
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > > Secondary
> > > > > > > > > > >> > > > > > > > storage.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > PL
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed
> > > Mushtaq
> > > > <
> > > > > > > > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > > > > > >> > > > > > <javascript:;>>
> > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > > I think I share you view on the 'Ideal
> > > > world'.
> > > > > > > > Backup
> > > > > > > > > > (via
> > > > > > > > > > >> > > Volume
> > > > > > > > > > >> > > > > > > > > Snapshots) is a huge bottleneck in
> > > > Cloudstack.
> > > > > > > This
> > > > > > > > is
> > > > > > > > > > >> > > amplified
> > > > > > > > > > >> > > > > > > > especially
> > > > > > > > > > >> > > > > > > > > when you have a object storage as your
> > > > > secondary
> > > > > > > > > storage
> > > > > > > > > > >> > > because
> > > > > > > > > > >> > > > it
> > > > > > > > > > >> > > > > > > > > requires two copies (one to an NFS
> > staging
> > > > > area
> > > > > > > and
> > > > > > > > > from
> > > > > > > > > > >> > there
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > > object
> > > > > > > > > > >> > > > > > > > > storage). And not to mention that all
> > > these
> > > > > > copies
> > > > > > > > are
> > > > > > > > > > >> > > consuming
> > > > > > > > > > >> > > > > > > > hypervisor
> > > > > > > > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a
> > huge
> > > > > > > > bottleneck
> > > > > > > > > as
> > > > > > > > > > >> all
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > > > Network
> > > > > > > > > > >> > > > > > > > > and I/O flow through it. So our
> > intention
> > > of
> > > > > > > > proposing
> > > > > > > > > > the
> > > > > > > > > > >> > > > "Storage
> > > > > > > > > > >> > > > > > > > > Snapshots" is to give a better way of
> > > > achiving
> > > > > > > > > snapshots
> > > > > > > > > > >> > while
> > > > > > > > > > >> > > > > still
> > > > > > > > > > >> > > > > > > > > keeping the original definition of
> > volume
> > > > > > > snpashots
> > > > > > > > > (ie
> > > > > > > > > > >> > upload
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > sec
> > > > > > > > > > >> > > > > > > > > storage).
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > But as Erik pointed out volume
> snapshots
> > > are
> > > > > not
> > > > > > > > > > backups.
> > > > > > > > > > >> > They
> > > > > > > > > > >> > > > > don't
> > > > > > > > > > >> > > > > > > work
> > > > > > > > > > >> > > > > > > > > form multi-disk LVM volume groups and
> > > > dynamic
> > > > > > > > disks. I
> > > > > > > > > > am
> > > > > > > > > > >> all
> > > > > > > > > > >> > > in
> > > > > > > > > > >> > > > > for
> > > > > > > > > > >> > > > > > a
> > > > > > > > > > >> > > > > > > > > better backup solution which handles
> > these
> > > > use
> > > > > > > cases
> > > > > > > > > and
> > > > > > > > > > >> > > utilizes
> > > > > > > > > > >> > > > > the
> > > > > > > > > > >> > > > > > > > > storage's advanced features.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul
> > > Angus
> > > > <
> > > > > > > > > > >> > > > > > paul.angus@shapeblue.com <javascript:;>
> > > > > > <javascript:;>>
> > > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > In the beginning... there were
> > > CloudStack
> > > > > > > > snapshots
> > > > > > > > > > and
> > > > > > > > > > >> > they
> > > > > > > > > > >> > > > were
> > > > > > > > > > >> > > > > > > > > actually
> > > > > > > > > > >> > > > > > > > > > volume snapshots not hypervisor
> > > > > point-in-time
> > > > > > > > > > snapshots.
> > > > > > > > > > >> > > > > > > > > > Then VM snapshots were created
> (which
> > > are
> > > > > > > > > > point-in-time
> > > > > > > > > > >> > > > > hypervisor
> > > > > > > > > > >> > > > > > > > > > snapshots) and we started referring
> to
> > > the
> > > > > > > > original
> > > > > > > > > > >> > snapshots
> > > > > > > > > > >> > > > as
> > > > > > > > > > >> > > > > > > volume
> > > > > > > > > > >> > > > > > > > > > snapshots.
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > CloudStack does not offer 'backups',
> > but
> > > > > many
> > > > > > > > people
> > > > > > > > > > use
> > > > > > > > > > >> > > volume
> > > > > > > > > > >> > > > > > > > snapshots
> > > > > > > > > > >> > > > > > > > > > as backups. However you can't
> in-place
> > > > > restore
> > > > > > > > > volume
> > > > > > > > > > >> > > snapshots
> > > > > > > > > > >> > > > > and
> > > > > > > > > > >> > > > > > > if
> > > > > > > > > > >> > > > > > > > > you
> > > > > > > > > > >> > > > > > > > > > have a VM with multiple volumes, the
> > > > volume
> > > > > > > > > snapshots
> > > > > > > > > > >> must
> > > > > > > > > > >> > be
> > > > > > > > > > >> > > > > done
> > > > > > > > > > >> > > > > > in
> > > > > > > > > > >> > > > > > > > > > series, meaning that the state
> across
> > > all
> > > > of
> > > > > > the
> > > > > > > > > > >> volumes is
> > > > > > > > > > >> > > > > > unlikely
> > > > > > > > > > >> > > > > > > to
> > > > > > > > > > >> > > > > > > > > be
> > > > > > > > > > >> > > > > > > > > > consistent.
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > 'Actual Backups' would enable all of
> > the
> > > > > > restore
> > > > > > > > > > options
> > > > > > > > > > >> > > which
> > > > > > > > > > >> > > > > > users
> > > > > > > > > > >> > > > > > > > > might
> > > > > > > > > > >> > > > > > > > > > expect as well options as to where
> > they
> > > > > might
> > > > > > be
> > > > > > > > > > >> stored. In
> > > > > > > > > > >> > > my
> > > > > > > > > > >> > > > > > ideal
> > > > > > > > > > >> > > > > > > > > world
> > > > > > > > > > >> > > > > > > > > > they would also be able to leverage
> > > > back-end
> > > > > > > > > hardware
> > > > > > > > > > >> (such
> > > > > > > > > > >> > > as
> > > > > > > > > > >> > > > > > > > Solidfire,
> > > > > > > > > > >> > > > > > > > > > NetApp etc :) ) and software such as
> > > > Veeam,
> > > > > > > > > Commvault
> > > > > > > > > > >> etc
> > > > > > > > > > >> > to
> > > > > > > > > > >> > > > > > > accelerate
> > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > process.
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > [image: ShapeBlue] <
> > > > > http://www.shapeblue.com>
> > > > > > > > > > >> > > > > > > > > > Paul Angus
> > > > > > > > > > >> > > > > > > > > > VP Technology , ShapeBlue
> > > > > > > > > > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203
> 603
> > > > 0540*
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > >
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > |
> > > > > > > > > > >> > > > > m:
> > > > > > > > > > >> > > > > > > > > > *+44 7711 418784*
> > <+44%207711%20418784>
> > > > > > > > > > >> > > > > > > > > > e: *paul.angus@shapeblue.com
> > > > <javascript:;>
> > > > > > > > > > >> > <javascript:;>
> > > > > > > > > > >> > > |
> > > > > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > > > > >> > > > > > > > > > <paul.angus@shapeblue.com
> > > <javascript:;>
> > > > > > > > > > >> > > > > <ja...@cloudyangus>
> > > > > > > > > > >> > > > > > | w:
> > > > > > > > > > >> > > > > > > > > > *www.shapeblue.com* <
> > > > > http://www.shapeblue.com
> > > > > > >
> > > > > > > > > > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > > London
> > > > > WC2N
> > > > > > > > 4HS
> > > > > > > > > UK
> > > > > > > > > > >> > > > > > > > > > Shape Blue Ltd is a company
> > incorporated
> > > > in
> > > > > > > > England
> > > > > > > > > &
> > > > > > > > > > >> > Wales.
> > > > > > > > > > >> > > > > > > ShapeBlue
> > > > > > > > > > >> > > > > > > > > > Services India LLP is a company
> > > > incorporated
> > > > > > in
> > > > > > > > > India
> > > > > > > > > > >> and
> > > > > > > > > > >> > is
> > > > > > > > > > >> > > > > > operated
> > > > > > > > > > >> > > > > > > > > under
> > > > > > > > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape
> > Blue
> > > > > Brasil
> > > > > > > > > > >> Consultoria
> > > > > > > > > > >> > > Ltda
> > > > > > > > > > >> > > > > is
> > > > > > > > > > >> > > > > > a
> > > > > > > > > > >> > > > > > > > > > company incorporated in Brasil and
> is
> > > > > operated
> > > > > > > > under
> > > > > > > > > > >> > license
> > > > > > > > > > >> > > > from
> > > > > > > > > > >> > > > > > > Shape
> > > > > > > > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a
> > > > company
> > > > > > > > > registered
> > > > > > > > > > >> by
> > > > > > > > > > >> > The
> > > > > > > > > > >> > > > > > > Republic
> > > > > > > > > > >> > > > > > > > of
> > > > > > > > > > >> > > > > > > > > > South Africa and is traded under
> > license
> > > > > from
> > > > > > > > Shape
> > > > > > > > > > Blue
> > > > > > > > > > >> > Ltd.
> > > > > > > > > > >> > > > > > > ShapeBlue
> > > > > > > > > > >> > > > > > > > > is
> > > > > > > > > > >> > > > > > > > > > a registered trademark.
> > > > > > > > > > >> > > > > > > > > > This email and any attachments to it
> > may
> > > > be
> > > > > > > > > > confidential
> > > > > > > > > > >> > and
> > > > > > > > > > >> > > > are
> > > > > > > > > > >> > > > > > > > intended
> > > > > > > > > > >> > > > > > > > > > solely for the use of the individual
> > to
> > > > whom
> > > > > > it
> > > > > > > is
> > > > > > > > > > >> > addressed.
> > > > > > > > > > >> > > > Any
> > > > > > > > > > >> > > > > > > views
> > > > > > > > > > >> > > > > > > > > or
> > > > > > > > > > >> > > > > > > > > > opinions expressed are solely those
> of
> > > the
> > > > > > > author
> > > > > > > > > and
> > > > > > > > > > do
> > > > > > > > > > >> > not
> > > > > > > > > > >> > > > > > > > necessarily
> > > > > > > > > > >> > > > > > > > > > represent those of Shape Blue Ltd or
> > > > related
> > > > > > > > > > companies.
> > > > > > > > > > >> If
> > > > > > > > > > >> > > you
> > > > > > > > > > >> > > > > are
> > > > > > > > > > >> > > > > > > not
> > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > intended recipient of this email,
> you
> > > must
> > > > > > > neither
> > > > > > > > > > take
> > > > > > > > > > >> any
> > > > > > > > > > >> > > > > action
> > > > > > > > > > >> > > > > > > > based
> > > > > > > > > > >> > > > > > > > > > upon its contents, nor copy or show
> it
> > > to
> > > > > > > anyone.
> > > > > > > > > > Please
> > > > > > > > > > >> > > > contact
> > > > > > > > > > >> > > > > > the
> > > > > > > > > > >> > > > > > > > > sender
> > > > > > > > > > >> > > > > > > > > > if you believe you have received
> this
> > > > email
> > > > > in
> > > > > > > > > error.
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > -----Original Message-----
> > > > > > > > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > > > >> > > > > <javascript:;> <javascript:;>]
> > > > > > > > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58
> PM
> > > > > > > > > > >> > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > <javascript:;>
> > > > > > > > > > >> > <javascript:;>
> > > > > > > > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature]
> > > > Storage
> > > > > > > > > Snapshots
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Paul,
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > When you say actual backups, how
> would
> > > it
> > > > be
> > > > > > > > > different
> > > > > > > > > > >> from
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > > > Volume
> > > > > > > > > > >> > > > > > > > > > Snapshots that exist currently. My
> > > > > > understanding
> > > > > > > > is
> > > > > > > > > > that
> > > > > > > > > > >> > > > Backups
> > > > > > > > > > >> > > > > > end
> > > > > > > > > > >> > > > > > > up
> > > > > > > > > > >> > > > > > > > > in
> > > > > > > > > > >> > > > > > > > > > Sec Storage whereas Snapshots are
> > just a
> > > > > > > > > point-in-time
> > > > > > > > > > >> > state
> > > > > > > > > > >> > > of
> > > > > > > > > > >> > > > > > your
> > > > > > > > > > >> > > > > > > > > volume
> > > > > > > > > > >> > > > > > > > > > which can be restored back correct?
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > -Syed
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM,
> Paul
> > > > Angus
> > > > > <
> > > > > > > > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> > > > > > > <javascript:;>>
> > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Hi Syed,
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > As I understand it, the SolidFire
> > > plugin
> > > > > > will
> > > > > > > > > export
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > snapshot
> > > > > > > > > > >> > > > > > > to
> > > > > > > > > > >> > > > > > > > > > > secondary storage if the user
> > > requests a
> > > > > > > > template
> > > > > > > > > > from
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > > snapshot
> > > > > > > > > > >> > > > > > > > or
> > > > > > > > > > >> > > > > > > > > > > wants to download the snapshot
> from
> > > the
> > > > > > cloud.
> > > > > > > > > This
> > > > > > > > > > >> is a
> > > > > > > > > > >> > > > good,
> > > > > > > > > > >> > > > > > > > > > > pragmatic approach and yes Mike
> the
> > > > > > SolidFire
> > > > > > > > > > storage
> > > > > > > > > > >> is
> > > > > > > > > > >> > > > super
> > > > > > > > > > >> > > > > > > > > > > reliable and snapshots on
> SolidFire
> > > > arrays
> > > > > > > take
> > > > > > > > up
> > > > > > > > > > >> next
> > > > > > > > > > >> > to
> > > > > > > > > > >> > > no
> > > > > > > > > > >> > > > > > > space.
> > > > > > > > > > >> > > > > > > > > > > BUT I think that we are talking
> > about
> > > a
> > > > > more
> > > > > > > > > general
> > > > > > > > > > >> > > purpose
> > > > > > > > > > >> > > > > API,
> > > > > > > > > > >> > > > > > > and
> > > > > > > > > > >> > > > > > > > > > > other storage systems may not be
> as
> > > > > awesome
> > > > > > as
> > > > > > > > > > Mike's.
> > > > > > > > > > >> > > That's
> > > > > > > > > > >> > > > > my
> > > > > > > > > > >> > > > > > > > > > > concern. Also, the time to
> transfer
> > > for
> > > > > say
> > > > > > > 1TB
> > > > > > > > to
> > > > > > > > > > >> move
> > > > > > > > > > >> > > from
> > > > > > > > > > >> > > > > > > primary
> > > > > > > > > > >> > > > > > > > > > > to sec storage and then create a
> VM
> > > > > template
> > > > > > > out
> > > > > > > > > of
> > > > > > > > > > it
> > > > > > > > > > >> > may
> > > > > > > > > > >> > > be
> > > > > > > > > > >> > > > > too
> > > > > > > > > > >> > > > > > > > long
> > > > > > > > > > >> > > > > > > > > > for users.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the
> > term
> > > > > volume
> > > > > > > > > > snapshot
> > > > > > > > > > >> for
> > > > > > > > > > >> > > > > backup,
> > > > > > > > > > >> > > > > > > > it's
> > > > > > > > > > >> > > > > > > > > > > just that users want to do backups
> > > and a
> > > > > > > volume
> > > > > > > > > > >> snapshot
> > > > > > > > > > >> > is
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > only
> > > > > > > > > > >> > > > > > > > > > > type of snapshot that copies the
> > disk
> > > > > > > elsewhere
> > > > > > > > > and
> > > > > > > > > > >> can
> > > > > > > > > > >> > be
> > > > > > > > > > >> > > > > > > scheduled.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > I'm 'pondering' the implications
> of
> > > > > enabling
> > > > > > > > > actual
> > > > > > > > > > >> > backups
> > > > > > > > > > >> > > > > > > (through
> > > > > > > > > > >> > > > > > > > > > > recognised backup providers) and
> the
> > > > user
> > > > > > > > > > requirements
> > > > > > > > > > >> > > around
> > > > > > > > > > >> > > > > > them
> > > > > > > > > > >> > > > > > > > > > > (particularly restoration use
> cases)
> > > as
> > > > a
> > > > > > > > separate
> > > > > > > > > > >> thread
> > > > > > > > > > >> > > of
> > > > > > > > > > >> > > > > > work.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > [image: ShapeBlue] <
> > > > > > http://www.shapeblue.com>
> > > > > > > > > Paul
> > > > > > > > > > >> Angus
> > > > > > > > > > >> > > VP
> > > > > > > > > > >> > > > > > > > Technology
> > > > > > > > > > >> > > > > > > > > > > , ShapeBlue
> > > > > > > > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203
> > 603
> > > > > 0540*
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > >
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > > > > >> > > > |
> > > > > > > > > > >> > > > > m:
> > > > > > > > > > >> > > > > > > > > > > *+44 7711 418784*
> > > <+44%207711%20418784>
> > > > > > > > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > > > <javascript:;>
> > > > > > > > > > >> > > <javascript:;> |
> > > > > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > > > > >> > > > > > > > > > > <paul.angus@shapeblue.com
> > > > <javascript:;>
> > > > > > > > > > >> <javascript:;>
> > > > > > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > > >> > > > > > > > > > > *www.shapeblue.com* <
> > > > > > http://www.shapeblue.com
> > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > > > London
> > > > > > WC2N
> > > > > > > > 4HS
> > > > > > > > > > UK
> > > > > > > > > > >> > Shape
> > > > > > > > > > >> > > > > Blue
> > > > > > > > > > >> > > > > > > Ltd
> > > > > > > > > > >> > > > > > > > > > > is a company incorporated in
> > England &
> > > > > > Wales.
> > > > > > > > > > >> ShapeBlue
> > > > > > > > > > >> > > > > Services
> > > > > > > > > > >> > > > > > > > India
> > > > > > > > > > >> > > > > > > > > > > LLP is a company incorporated in
> > India
> > > > and
> > > > > > is
> > > > > > > > > > operated
> > > > > > > > > > >> > > under
> > > > > > > > > > >> > > > > > > license
> > > > > > > > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue
> > Brasil
> > > > > > > > Consultoria
> > > > > > > > > > >> Ltda
> > > > > > > > > > >> > is
> > > > > > > > > > >> > > a
> > > > > > > > > > >> > > > > > > company
> > > > > > > > > > >> > > > > > > > > > > incorporated in Brasil and is
> > operated
> > > > > under
> > > > > > > > > license
> > > > > > > > > > >> from
> > > > > > > > > > >> > > > Shape
> > > > > > > > > > >> > > > > > > Blue
> > > > > > > > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a
> > company
> > > > > > > > registered
> > > > > > > > > by
> > > > > > > > > > >> The
> > > > > > > > > > >> > > > > Republic
> > > > > > > > > > >> > > > > > > of
> > > > > > > > > > >> > > > > > > > > > > South Africa and is traded under
> > > license
> > > > > > from
> > > > > > > > > Shape
> > > > > > > > > > >> Blue
> > > > > > > > > > >> > > Ltd.
> > > > > > > > > > >> > > > > > > > > > > ShapeBlue is a registered
> trademark.
> > > > > > > > > > >> > > > > > > > > > > This email and any attachments to
> it
> > > may
> > > > > be
> > > > > > > > > > >> confidential
> > > > > > > > > > >> > > and
> > > > > > > > > > >> > > > > are
> > > > > > > > > > >> > > > > > > > > > > intended solely for the use of the
> > > > > > individual
> > > > > > > to
> > > > > > > > > > whom
> > > > > > > > > > >> it
> > > > > > > > > > >> > is
> > > > > > > > > > >> > > > > > > > addressed.
> > > > > > > > > > >> > > > > > > > > > > Any views or opinions expressed
> are
> > > > solely
> > > > > > > those
> > > > > > > > > of
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > author
> > > > > > > > > > >> > > > > > and
> > > > > > > > > > >> > > > > > > do
> > > > > > > > > > >> > > > > > > > > > > not necessarily represent those of
> > > Shape
> > > > > > Blue
> > > > > > > > Ltd
> > > > > > > > > or
> > > > > > > > > > >> > > related
> > > > > > > > > > >> > > > > > > > > > > companies. If you are not the
> > intended
> > > > > > > recipient
> > > > > > > > > of
> > > > > > > > > > >> this
> > > > > > > > > > >> > > > email,
> > > > > > > > > > >> > > > > > you
> > > > > > > > > > >> > > > > > > > > > > must neither take any action based
> > > upon
> > > > > its
> > > > > > > > > > contents,
> > > > > > > > > > >> nor
> > > > > > > > > > >> > > > copy
> > > > > > > > > > >> > > > > or
> > > > > > > > > > >> > > > > > > > show
> > > > > > > > > > >> > > > > > > > > > > it to anyone. Please contact the
> > > sender
> > > > if
> > > > > > you
> > > > > > > > > > believe
> > > > > > > > > > >> > you
> > > > > > > > > > >> > > > have
> > > > > > > > > > >> > > > > > > > > received
> > > > > > > > > > >> > > > > > > > > > this email in error.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > > > >> > > > > <javascript:;>
> > > > > > > > > > >> > > > > > <javascript:;>]
> > > > > > > > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > > > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > <javascript:;>
> > > > > > > > > > >> > > <javascript:;>
> > > > > > > > > > >> > > > > > > > > > > Subject: Re: [Propose][New
> Feature]
> > > > > Storage
> > > > > > > > > > Snapshots
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > I think the terminology confusion
> > > comes
> > > > > from
> > > > > > > AWS
> > > > > > > > > > where
> > > > > > > > > > >> > they
> > > > > > > > > > >> > > > do
> > > > > > > > > > >> > > > > > EBS
> > > > > > > > > > >> > > > > > > > > > > snapshots backed up to S3 and
> > > CloudStack
> > > > > > sort
> > > > > > > of
> > > > > > > > > > >> followed
> > > > > > > > > > >> > > > that.
> > > > > > > > > > >> > > > > > And
> > > > > > > > > > >> > > > > > > > as
> > > > > > > > > > >> > > > > > > > > > > an end user who is oblivious to
> the
> > > > > > internals
> > > > > > > of
> > > > > > > > > my
> > > > > > > > > > >> > > provider,
> > > > > > > > > > >> > > > > my
> > > > > > > > > > >> > > > > > > > > > > expectation would be something
> > similar
> > > > to
> > > > > > what
> > > > > > > > AWS
> > > > > > > > > > as
> > > > > > > > > > >> > that
> > > > > > > > > > >> > > is
> > > > > > > > > > >> > > > > my
> > > > > > > > > > >> > > > > > > > > > > biggest reference point.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > To your point Mike, I agree that a
> > > > Primary
> > > > > > > > Storage
> > > > > > > > > > >> > failure
> > > > > > > > > > >> > > on
> > > > > > > > > > >> > > > > > > > > > > SolidFire is unlikely, there are
> > other
> > > > > > > > motivations
> > > > > > > > > > >> for us
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > push
> > > > > > > > > > >> > > > > > > > data
> > > > > > > > > > >> > > > > > > > > > > to secondary storage. Primary
> > storage
> > > > > > (atleast
> > > > > > > > for
> > > > > > > > > > us)
> > > > > > > > > > >> > > costs
> > > > > > > > > > >> > > > > > > around 3
> > > > > > > > > > >> > > > > > > > > > > times as much as secondary storage
> > and
> > > > > > > snapshots
> > > > > > > > > on
> > > > > > > > > > >> > primary<> > > > > > > > take 2
> > > > > > > > > > >> > days
> > > > > > > > > > >> > > > to
> > > > > > > > > > >> > > > > > > > > complete.
> > > > > > > > > > >> > > > > > > > > > > > > > e. snapshots (as they are)
> > can't
> > > > be
> > > > > > used
> > > > > > > > for
> > > > > > > > > > >> > multiple
> > > > > > > > > > >> > > > LVM
> > > > > > > > > > >> > > > > > > > disks.
> > > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > > > I think the process Mike has
> > > used
> > > > in
> > > > > > the
> > > > > > > > > > >> SolidFire
> > > > > > > > > > >> > > > plugin
> > > > > > > > > > >> > > > > > > (only
> > > > > > > > > > >> > > > > > > > > > > > > > moving
> > > > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > > > > > disk image to secondary
> > storage
> > > > when
> > > > > > you
> > > > > > > > > > >> absolutely
> > > > > > > > > > >> > > > have
> > > > > > > > > > >> > > > > > to)
> > > > > > > > > > >> > > > > > > is
> > > > > > > > > > >> > > > > > > > > > > > > > a very
> > > > > > > > > > >> > > > > > > > > > > > > good
> > > > > > > > > > >> > > > > > > > > > > > > > and pragmatic solution. I
> > wonder
> > > > > what
> > > > > > > > > problems
> > > > > > > > > > >> an
> > > > > > > > > > >> > > > > operator
> > > > > > > > > > >> > > > > > > > might
> > > > > > > > > > >> > > > > > > > > > > > > experience
> > > > > > > > > > >> > > > > > > > > > > > > > if they have an issue with a
> > > given
> > > > > > > primary
> > > > > > > > > > >> storage
> > > > > > > > > > >> > > pool
> > > > > > > > > > >> > > > > in
> > > > > > > > > > >> > > > > > a
> > > > > > > > > > >> > > > > > > > > > cluster.
> > > > > > > > > > >> > > > > > > > > > > > (I
> > > > > > > > > > >> > > > > > > > > > > > > > know that that is REALLY
> > > unlikely
> > > > in
> > > > > > the
> > > > > > > > > > >> SolidFire
> > > > > > > > > > >> > > case
> > > > > > > > > > >> > > > > > Mike
> > > > > > > > > > >> > > > > > > :)
> > > > > > > > > > >> > > > > > > > > > > > > > ) And
> > > > > > > > > > >> > > > > > > > > > > > if
> > > > > > > > > > >> > > > > > > > > > > > > > the transfer from primary to
> > > > > secondary
> > > > > > > is
> > > > > > > > > > slow,
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > time
> > > > > > > > > > >> > > > > to
> > > > > > > > > > >> > > > > > > > > > > > > > being able
> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > [image: Shape
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

I think so.

Just to confirm: The default behavior for a volume snapshot with managed
storage is equivalent to a storage snapshot.

When the global setting value is changed, the data ends up on secondary
storage (NFS).

Is that accurate?

On Friday, February 12, 2016, Syed Mushtaq <sy...@gmail.com> wrote:

> > Situation: Global setting value is changed to a non-default value.
> >
> > 2) Action: Volume snapshot is taken with managed storage. Result: ??? I'm
> > not sure which scenario we're looking at here now.
> >
>
> In this situation, volume snapshots will end up on secondary storage and
> the snapshot on the managed storage will be deleted.
>
>
> >
> > 3) Action: Storage snapshot is taken with managed storage. Result: ???
> I'm
> > not sure which scenario we're looking at here now.
> >
>
> In this situation, the snapshot stays on the managed storage.
>
> Does this answer your question Mike?
>
>
>
>
>
> On Fri, Feb 12, 2016 at 11:43 AM, Mike Tutkowski <
> mike.tutkowski@solidfire.com <javascript:;>> wrote:
>
> > Hi Syed,
> >
> > Can you clarify how you see these behaviors (below) working now that we
> are
> > considering a global settings value applicable to managed storage with
> > volume snapshots?
> >
> > State: Global setting value defaults to keep snapshot on storage system
> > (primary storage) for managed storage.
> >
> > 1) Action: Volume snapshot is taken with managed storage. Result:
> Snapshot
> > is taken and kept on storage system (primary storage). (This is what you
> > and I have been working on for a couple months now.)
> >
> >
> >
> > Thanks!
> > Mike
> >
> > On Fri, Feb 12, 2016 at 9:15 AM, Syed Mushtaq <sy...@gmail.com>
> > wrote:
> >
> > > I agree with Mike's concern about backward compatibility. We can add a
> > > global flag which makes sure that the way volume snapshots work
> currently
> > > on managed storage (stay on the device) is retained after upgrade. We
> can
> > > then safely implement the Storage Snapshot API while making the Volume
> > > Snpashot API move the snapshot to Secondary Storage.
> > >
> > > Sounds good guys?
> > >
> > > -Syed
> > >
> > >
> > > On Mon, Feb 8, 2016 at 2:56 PM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com
> > > > wrote:
> > >
> > > > Here's what we have for snapshots for managed storage as of 4.6,
> Paul:
> > > >
> > > > 1. VM snapshots (no proposed changes to this).
> > > >
> > > > 2. Volume snapshots that do not end up on secondary storage, but
> rather
> > > are
> > > > stored on a SAN (effectively storing snapshots on primary storage).
> > > >
> > > > Pierre-Luc is saying he'd like this for snapshots for managed
> storage:
> > > >
> > > > A. VM snapshots (no proposed changes to this).
> > > >
> > > > B. Volume snapshots that export to secondary storage.
> > > >
> > > > C. New: Storage snapshots that behave like 2 (above).
> > > >
> > > > I like Pierre-Luc's ideas there, but the problem is backward
> > > compatibility.
> > > >
> > > > If customers who were using managed storage with volume snapshots in
> > 4.6
> > > > were getting their snapshots put on a SAN, then in 4.9 - all of a
> > sudden
> > > -
> > > > their new snapshots are put on secondary storage (unless they
> > explicitly
> > > > change over to using the new Storage snapshots feature).
> > > >
> > > >
> > > >
> > > > On Mon, Feb 8, 2016 at 12:32 PM, Paul Angus <
> paul.angus@shapeblue.com>
> > > > wrote:
> > > >
> > > > > Just to make sure I'm on the same page, are we talking about;
> > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9278 ?
> > > > >
> > > > > The FS reads (to me) more like 1a + the possibility to export to
> > > > secondary
> > > > > storage if required?
> > > > > Have I understood correctly?
> > > > > I have seen [1a] implemented for VMware by NetApp in their beta
> > > > CloudStack
> > > > > plugin (pleased I can say that without Mike beating me up now). No
> > > > changes
> > > > > to the CloudStack API were required. (nb it didn't export to
> > secondary
> > > > > storage).
> > > > >
> > > > >
> > > > >
> > > > > 1. VM Snapshot (point-in-time hypervisor based snapshots)
> > > > > 1a. SAN assisted VM snapshots (point-in-time hypervisor snapshot
> > takes
> > > > > place on transparently SAN to avoid performance issue in disk
> chains)
> > > > > 2. SAN Snapshot (Storage Snapshot) - NEW
> > > > > 3. Volume Snapshot (current old/slow transfer to secstorage)
> > > > > 4. Backup - JUST AN IDEAL.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > Paul Angus
> > > > > VP Technology ,  ShapeBlue
> > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > > <pa...@cloudyangus>  |  w:
> > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > Shape Blue Ltd is a company incorporated in England & Wales.
> > ShapeBlue
> > > > > Services India LLP is a company incorporated in India and is
> operated
> > > > under
> > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
> a
> > > > > company incorporated in Brasil and is operated under license from
> > Shape
> > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > Republic
> > > of
> > > > > South Africa and is traded under license from Shape Blue Ltd.
> > ShapeBlue
> > > > is
> > > > > a registered trademark.
> > > > > This email and any attachments to it may be confidential and are
> > > intended
> > > > > solely for the use of the individual to whom it is addressed. Any
> > views
> > > > or
> > > > > opinions expressed are solely those of the author and do not
> > > necessarily
> > > > > represent those of Shape Blue Ltd or related companies. If you are
> > not
> > > > the
> > > > > intended recipient of this email, you must neither take any action
> > > based
> > > > > upon its contents, nor copy or show it to anyone. Please contact
> the
> > > > sender
> > > > > if you believe you have received this email in error.
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
> > > > > Sent: Monday, February 8, 2016 7:16 PM
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > >
> > > > > Correct, Will.
> > > > >
> > > > > That Global Settings would only be for managed storage. Non-managed
> > > > > (traditional) volume snapshots are completely un-impacted by this
> > > > feature.
> > > > >
> > > > > If we need to sometimes keep the snapshots on the SAN and sometimes
> > > push
> > > > > them to secondary storage, we'll need a more robust solution than
> > > Global
> > > > > Settings, though.
> > > > >
> > > > > On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <
> wstevens@cloudops.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Sorry. I missed a bit of context when I responded. The global
> > setting
> > > > > > would be only for the managed storage case, currently being
> called
> > > > > Storage
> > > > > > Snapshots, and is only to determine if a copy is pushed to
> > secondary
> > > > > > storage right? The global setting would not change the behavior
> of
> > > the
> > > > > > Volume Snapshots right?
> > > > > >
> > > > > > I was referring to the need for Volume Snapshots and Storage
> > > Snapshots
> > > > to
> > > > > > exist together. Disregard my comment. I caught up on context
> after
> > I
> > > > > > posted. My bad...
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > > w cloudops.com *|* tw @CloudOps_
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> > > > > > mike.tutkowski@solidfire.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hey Will,
> > > > > > >
> > > > > > > Who's picking the behavior? Is it the cloud provider or the end
> > > user?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <
> > > wstevens@cloudops.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I don't think a global setting is a good option because we
> need
> > > > both
> > > > > > > > functionality to be available at the same time and for
> > different
> > > > use
> > > > > > > cases
> > > > > > > > to be able to pick which they choose.
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > > > > w cloudops.com *|* tw @CloudOps_
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > > > > > > mike.tutkowski@solidfire.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Now that I re-read your e-mail, it dawned on me: The
> end-user
> > > > > doesn't
> > > > > > > > care
> > > > > > > > > where the snapshot is.
> > > > > > > > >
> > > > > > > > > If that's true, then we should perhaps control this via
> > Global
> > > > > > Settings
> > > > > > > > or
> > > > > > > > > something.
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > > > > > > mike.tutkowski@solidfire.com> wrote:
> > > > > > > > >
> > > > > > > > > > It's not ideal - true, but it does allow us to be
> backward
> > > > > > > compatible.
> > > > > > > > > >
> > > > > > > > > > If you have other ideas, though, about how to maintain
> > > backward
> > > > > > > > > > compatibility, I'm definitely open to hear them.
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi Mike,
> > > > > > > > > >>
> > > > > > > > > >> Adding a flag to createSnapshot was the first and the
> most
> > > > > obvious
> > > > > > > > thing
> > > > > > > > > >> that came to our minds. The problem that I had with this
> > was
> > > > > that:
> > > > > > > > > >>
> > > > > > > > > >> 1) I feel it is exposing something to the end user that
> is
> > > > > > internal
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > >> cloud.
> > > > > > > > > >>
> > > > > > > > > >> 2) We have to follow two different ways of
> > restore/deletion
> > > in
> > > > > the
> > > > > > > > same
> > > > > > > > > >> code path depending on where the Snapshot resides which
> I
> > > find
> > > > > > kind
> > > > > > > > of a
> > > > > > > > > >> bad design.
> > > > > > > > > >>
> > > > > > > > > >> But if exposing a archive flag to the end user is
> > acceptable
> > > > > then
> > > > > > we
> > > > > > > > can
> > > > > > > > > >> definitely use this instead of adding the
> StorageSnapshot
> > > API
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> -Syed
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > > > > > > >> mike.tutkowski@solidfire.com
> > > > > > > > > >> > wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi Pierre-Luc,
> > > > > > > > > >> >
> > > > > > > > > >> > My recommendation would be this:
> > > > > > > > > >> >
> > > > > > > > > >> > Add an "archive" flag to the current volume-snapshot
> > API.
> > > > Its
> > > > > > > > default
> > > > > > > > > >> would
> > > > > > > > > >> > be "false" because that would be backward compatible
> > with
> > > > how
> > > > > > 4.6
> > > > > > > > has
> > > > > > > > > >> > volume snapshots implemented (i.e. they stay on the
> SAN
> > in
> > > > > 4.6,
> > > > > > > 4.7,
> > > > > > > > > and
> > > > > > > > > >> > 4.8).
> > > > > > > > > >> >
> > > > > > > > > >> > If you set archive=true, then we would perform a
> > > background
> > > > > > > > migration
> > > > > > > > > of
> > > > > > > > > >> > the snapshot from the SAN to the secondary storage
> (then
> > > > > delete
> > > > > > > the
> > > > > > > > > SAN
> > > > > > > > > >> > snapshot).
> > > > > > > > > >> >
> > > > > > > > > >> > That archive parameter would only be valid for managed
> > > > > storage.
> > > > > > > > > >> >
> > > > > > > > > >> > Sound reasonable?
> > > > > > > > > >> >
> > > > > > > > > >> > Also, a VM snapshot that includes disks provided by
> > > managed
> > > > > > > storage
> > > > > > > > > >> should
> > > > > > > > > >> > work.
> > > > > > > > > >> >
> > > > > > > > > >> > Talk to you later,
> > > > > > > > > >> > Mike
> > > > > > > > > >> >
> > > > > > > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > > > > > > pdion@cloudops.com
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Mike,
> > > > > > > > > >> > >
> > > > > > > > > >> > > In terms of API's, would you prefer introducing a
> > > > parameter
> > > > > to
> > > > > > > the
> > > > > > > > > >> > existing
> > > > > > > > > >> > > VolumeSnapshot, example: extract={true|false} with a
> > > > > > default
> > > > > > > > > value
> > > > > > > > > >> of
> > > > > > > > > >> > > true which would extract snapshot into the secondary
> > > > > storage,
> > > > > > > > which
> > > > > > > > > >> is
> > > > > > > > > >> > the
> > > > > > > > > >> > > current default behavior. Then for SAN snapshot that
> > > > remain
> > > > > on
> > > > > > > the
> > > > > > > > > >> SAN we
> > > > > > > > > >> > > would just set "extract=false" ? as oppose to
> create a
> > > new
> > > > > > > > > >> > > StorageSnapshot API ?
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > Paul,
> > > > > > > > > >> > >
> > > > > > > > > >> > > From what I'm seeing so far, we can't do a
> VM-snapshot
> > > > when
> > > > > > > using
> > > > > > > > > >> managed
> > > > > > > > > >> > > storage for VM having more than one Volume. For the
> > > reason
> > > > > > that
> > > > > > > > > >> snapshot
> > > > > > > > > >> > > are performed outside of the hypervisor awareness
> and
> > > > > > > > > asynchronously.
> > > > > > > > > >> If
> > > > > > > > > >> > > someone have a way to address that, it would make
> > thinks
> > > > > much
> > > > > > > more
> > > > > > > > > >> > > attractive.
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <
> > > > irae@cloudops.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > I think a service provider backup scenario is more
> > > > likely
> > > > > to
> > > > > > > > take
> > > > > > > > > >> > > advantage
> > > > > > > > > >> > > > of SAN snapshot. There are a few reasons for this.
> > > > > > Traditional
> > > > > > > > > >> backups
> > > > > > > > > >> > > > involve access to the file system, and there is an
> > > > > > expectation
> > > > > > > > > that
> > > > > > > > > >> > this
> > > > > > > > > >> > > > can be done with reasonably short time frames
> > without
> > > > > > > negatively
> > > > > > > > > >> > > impacting
> > > > > > > > > >> > > > VM performance, and that the backup orchestrator
> can
> > > > apply
> > > > > > > > various
> > > > > > > > > >> > logic
> > > > > > > > > >> > > > and or transformations to the data (compress,
> > encrypt,
> > > > > > deltas
> > > > > > > > > >> etc...).
> > > > > > > > > >> > > > While it is true that one could apply a backup
> > process
> > > > to
> > > > > a
> > > > > > > > cloud
> > > > > > > > > >> > > snapshot,
> > > > > > > > > >> > > > this would be slow and inefficient requiring the
> > data
> > > to
> > > > > be
> > > > > > > > moved
> > > > > > > > > >> > several
> > > > > > > > > >> > > > times and there are some major bottlenecks with
> > cloud
> > > > > > > snapshots.
> > > > > > > > > >> With a
> > > > > > > > > >> > > > cloud snapshot - there seems to be no reasonable
> > > > > expectation
> > > > > > > of
> > > > > > > > > >> being
> > > > > > > > > >> > > able
> > > > > > > > > >> > > > to do differential snapshots (I think this depends
> > on
> > > > the
> > > > > > > > > >> hypervisor)
> > > > > > > > > >> > and
> > > > > > > > > >> > > > if you do differential snapshots this will make
> file
> > > > > backups
> > > > > > > > from
> > > > > > > > > >> those
> > > > > > > > > >> > > > snapshots even more complicated to orchestrate.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Suspect there needs to be a different thread on
> how
> > to
> > > > > > better
> > > > > > > > > enable
> > > > > > > > > >> > > > backups, as a service. As per Paul's suggestion,
> but
> > > it
> > > > > is a
> > > > > > > > > related
> > > > > > > > > >> > > > workflow so relevant to this discussion.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Ian
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > > > > > > >> > > mike.tutkowski@solidfire.com>
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > To me it sounds like number two and number three
> > are
> > > > > > > different
> > > > > > > > > >> uses
> > > > > > > > > >> > for
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > same "thing"(which is totally fine).
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > As for taking a fast SAN snapshot and exporting
> it
> > > > > > > > > >> asynchronously, do
> > > > > > > > > >> > > we
> > > > > > > > > >> > > > > see the SSVM as performing the export?
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > To be backwards compatible with what we have in
> > 4.6
> > > > and
> > > > > > > later
> > > > > > > > > for
> > > > > > > > > >> > > volume
> > > > > > > > > >> > > > > snapshots for managed storage, I think it might
> be
> > > > > easier
> > > > > > if
> > > > > > > > we
> > > > > > > > > >> pass
> > > > > > > > > >> > > in a
> > > > > > > > > >> > > > > flag that says whether or not to archive the SAN
> > > > > snapshot
> > > > > > > > > (which,
> > > > > > > > > >> I
> > > > > > > > > >> > > > think,
> > > > > > > > > >> > > > > is something that you suggested, as well,
> > > Pierre-Luc).
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > > > > > > pdion@cloudops.com
> > > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > Hi Mike,
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > The reason behind the creation of a SAN
> snapshot
> > > > which
> > > > > > is
> > > > > > > > > >> exported
> > > > > > > > > >> > > into
> > > > > > > > > >> > > > > > secondary storage, is because creating a copy
> of
> > > the
> > > > > > .VHD
> > > > > > > > > >> directly
> > > > > > > > > >> > > > would
> > > > > > > > > >> > > > > > impact uptime of the VM as creating that copy
> > take
> > > > > lots
> > > > > > of
> > > > > > > > > time.
> > > > > > > > > >> > Has
> > > > > > > > > >> > > > > oppose
> > > > > > > > > >> > > > > > to a SAN snapshot that is close to
> instantaneous
> > > > which
> > > > > > can
> > > > > > > > > >> > afterward
> > > > > > > > > >> > > be
> > > > > > > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > I would suspect an extracted VolumeSnapshot
> > taken
> > > > > from a
> > > > > > > SAN
> > > > > > > > > >> > snapshot
> > > > > > > > > >> > > > > could
> > > > > > > > > >> > > > > > have is SAN snapshot deleted to avoid duplica
> > and
> > > > > space
> > > > > > > > > >> consumption
> > > > > > > > > >> > > on
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > Primary Storage side.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > I see 3 definitions in our current discussion
> > > > > regarding
> > > > > > > the
> > > > > > > > > term
> > > > > > > > > >> > > > snapshot
> > > > > > > > > >> > > > > > (these are not official terminology but by own
> > > > > > > > interpretation
> > > > > > > > > of
> > > > > > > > > >> > > them):
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's
> > > > > definition
> > > > > > > of
> > > > > > > > a
> > > > > > > > > >> > > > snapshot):
> > > > > > > > > >> > > > > > it's a volume snapshot at the storage level,
> > point
> > > > in
> > > > > > time
> > > > > > > > of
> > > > > > > > > >> your
> > > > > > > > > >> > > > data.
> > > > > > > > > >> > > > > it
> > > > > > > > > >> > > > > > reside on the primary storage. Useful and
> > > efficient
> > > > > for
> > > > > > > > > software
> > > > > > > > > >> > side
> > > > > > > > > >> > > > > > incident.
> > > > > > > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack
> > > > VolumeSnapshot/
> > > > > > > cloud
> > > > > > > > > >> backup
> > > > > > > > > >> > > > aws-S3
> > > > > > > > > >> > > > > > style ): Point in time copy of the Virtual
> Disk
> > > that
> > > > > > > reside
> > > > > > > > > on a
> > > > > > > > > >> > > > > different
> > > > > > > > > >> > > > > > storage array then the original Volume.
> > Facilitate
> > > > > data
> > > > > > > > > >> migration
> > > > > > > > > >> > > > between
> > > > > > > > > >> > > > > > clusters and, in case of primary storage
> > incident,
> > > > > > Volume
> > > > > > > > > >> snapshots
> > > > > > > > > >> > > are
> > > > > > > > > >> > > > > not
> > > > > > > > > >> > > > > > impacted and can be reuse.
> > > > > > > > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines
> > > data
> > > > > that
> > > > > > > > also
> > > > > > > > > >> > > validate
> > > > > > > > > >> > > > > data
> > > > > > > > > >> > > > > > integrity, provide a storage efficient
> archiving
> > > > > method
> > > > > > > and
> > > > > > > > an
> > > > > > > > > >> > > > > independent
> > > > > > > > > >> > > > > > way to restore your data in case of an major
> > > > > > > infrastructure
> > > > > > > > > >> > disaster.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Regards,
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > PL
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike
> Tutkowski <
> > > > > > > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > <javascript:;>
> > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > So, let's see if I currently follow the
> > > > > requirements:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > * Augment volume snapshots for managed
> storage
> > > to
> > > > > > > > > >> conditionally
> > > > > > > > > >> > > > export
> > > > > > > > > >> > > > > > data
> > > > > > > > > >> > > > > > > to NFS. The current process of taking a
> > snapshot
> > > > on
> > > > > > the
> > > > > > > > SAN
> > > > > > > > > is
> > > > > > > > > >> > > fine,
> > > > > > > > > >> > > > > but
> > > > > > > > > >> > > > > > > we'd like the option to export the data to
> > NFS,
> > > as
> > > > > > well.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Questions:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Once the data has been exported to NFS, do
> we
> > > keep
> > > > > the
> > > > > > > SAN
> > > > > > > > > >> > snapshot
> > > > > > > > > >> > > > or
> > > > > > > > > >> > > > > > > delete it?
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > If we are deleting the SAN snapshot, then
> why
> > > > don't
> > > > > we
> > > > > > > > just
> > > > > > > > > >> copy
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > VHD
> > > > > > > > > >> > > > > > > from primary to secondary the way we do
> today
> > > for
> > > > > > > > > non-managed
> > > > > > > > > >> > (i.e.
> > > > > > > > > >> > > > > > > traditional) storage? Why create a SAN
> > snapshot
> > > in
> > > > > > that
> > > > > > > > > >> scenario?
> > > > > > > > > >> > > > > Perhaps
> > > > > > > > > >> > > > > > > to have the SSVM mount and perform the VHD
> > copy
> > > to
> > > > > > > > secondary
> > > > > > > > > >> > > storage
> > > > > > > > > >> > > > > > > instead of a XenServer host?
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Thanks for the clarification.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > By the way, to me a backup is when you copy
> > data
> > > > > from
> > > > > > > one
> > > > > > > > > >> storage
> > > > > > > > > >> > > > > system
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > another (regardless of features, if any, to
> > > > restore
> > > > > > the
> > > > > > > > data
> > > > > > > > > >> in
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > > future). A snapshot is a point-in-time view
> of
> > > the
> > > > > > data
> > > > > > > > of a
> > > > > > > > > >> > volume
> > > > > > > > > >> > > > and
> > > > > > > > > >> > > > > > > it's stored on the same storage system as
> the
> > > > > volume.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc
> > > Dion <
> > > > > > > > > >> > > pdion@cloudops.com
> > > > > > > > > >> > > > > <javascript:;>
> > > > > > > > > >> > > > > > <javascript:;>>
> > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > > That's fun to see that discussion
> > happening. I
> > > > > 100%
> > > > > > > > agree
> > > > > > > > > >> with
> > > > > > > > > >> > > > Paul's
> > > > > > > > > >> > > > > > > > points of view. VolumeSnapshot are not a
> > > backup,
> > > > > > but I
> > > > > > > > do
> > > > > > > > > >> > > consider
> > > > > > > > > >> > > > > them
> > > > > > > > > >> > > > > > > as
> > > > > > > > > >> > > > > > > > a safety vest against Primary Storage
> > failure,
> > > > > > because
> > > > > > > > > >> failure
> > > > > > > > > >> > > > append
> > > > > > > > > >> > > > > > > :-( .
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > The current proposal around snapshots that
> > > > reside
> > > > > on
> > > > > > > the
> > > > > > > > > >> > primary
> > > > > > > > > >> > > > > > storage
> > > > > > > > > >> > > > > > > or
> > > > > > > > > >> > > > > > > > snapshots that end in the Secondary
> Storage
> > is
> > > > not
> > > > > > to
> > > > > > > > > >> address
> > > > > > > > > >> > > any
> > > > > > > > > >> > > > > kind
> > > > > > > > > >> > > > > > > of
> > > > > > > > > >> > > > > > > > backups requirement because a snapshot is
> > not
> > > a
> > > > > > > backup,
> > > > > > > > > >> event
> > > > > > > > > >> > an
> > > > > > > > > >> > > > > > > extracted
> > > > > > > > > >> > > > > > > > VM snapshot.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > The main idea, and again this is for
> managed
> > > > > > storage;
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage
> side
> > > > > snapshot
> > > > > > > > > >> capability
> > > > > > > > > >> > > for
> > > > > > > > > >> > > > > > fast
> > > > > > > > > >> > > > > > > > response time that support rollback to
> > > previous
> > > > > > > > timestamp,
> > > > > > > > > >> > create
> > > > > > > > > >> > > > new
> > > > > > > > > >> > > > > > > > volume and maybe create template.
> > > > > > > > > >> > > > > > > > not required to be a new API if the work
> is
> > > > > > > already
> > > > > > > > > >> done, I
> > > > > > > > > >> > > > think
> > > > > > > > > >> > > > > > > this
> > > > > > > > > >> > > > > > > > is a different behaviors than the user
> > > > expectation
> > > > > > of
> > > > > > > a
> > > > > > > > > >> > > > > > volume-snapshot.
> > > > > > > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current
> > > cloudstack
> > > > > > > > behavior
> > > > > > > > > >> that
> > > > > > > > > >> > > > create
> > > > > > > > > >> > > > > > an
> > > > > > > > > >> > > > > > > > extraction of a volume into
> SecondaryStorage
> > > > which
> > > > > > can
> > > > > > > > be
> > > > > > > > > >> reuse
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > create a
> > > > > > > > > >> > > > > > > > new volume into another Primary Storage.
> > This
> > > > type
> > > > > > of
> > > > > > > > > >> snapshot
> > > > > > > > > >> > > is a
> > > > > > > > > >> > > > > > slow
> > > > > > > > > >> > > > > > > > job since yes it would have to copy the
> full
> > > > > volume
> > > > > > > size
> > > > > > > > > on
> > > > > > > > > >> the
> > > > > > > > > >> > > > > > Secondary
> > > > > > > > > >> > > > > > > > storage.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > PL
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed
> > Mushtaq
> > > <
> > > > > > > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > > > > >> > > > > > <javascript:;>>
> > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > > I think I share you view on the 'Ideal
> > > world'.
> > > > > > > Backup
> > > > > > > > > (via
> > > > > > > > > >> > > Volume
> > > > > > > > > >> > > > > > > > > Snapshots) is a huge bottleneck in
> > > Cloudstack.
> > > > > > This
> > > > > > > is
> > > > > > > > > >> > > amplified
> > > > > > > > > >> > > > > > > > especially
> > > > > > > > > >> > > > > > > > > when you have a object storage as your
> > > > secondary
> > > > > > > > storage
> > > > > > > > > >> > > because
> > > > > > > > > >> > > > it
> > > > > > > > > >> > > > > > > > > requires two copies (one to an NFS
> staging
> > > > area
> > > > > > and
> > > > > > > > from
> > > > > > > > > >> > there
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > object
> > > > > > > > > >> > > > > > > > > storage). And not to mention that all
> > these
> > > > > copies
> > > > > > > are
> > > > > > > > > >> > > consuming
> > > > > > > > > >> > > > > > > > hypervisor
> > > > > > > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a
> huge
> > > > > > > bottleneck
> > > > > > > > as
> > > > > > > > > >> all
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > > Network
> > > > > > > > > >> > > > > > > > > and I/O flow through it. So our
> intention
> > of
> > > > > > > proposing
> > > > > > > > > the
> > > > > > > > > >> > > > "Storage
> > > > > > > > > >> > > > > > > > > Snapshots" is to give a better way of
> > > achiving
> > > > > > > > snapshots
> > > > > > > > > >> > while
> > > > > > > > > >> > > > > still
> > > > > > > > > >> > > > > > > > > keeping the original definition of
> volume
> > > > > > snpashots
> > > > > > > > (ie
> > > > > > > > > >> > upload
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > sec
> > > > > > > > > >> > > > > > > > > storage).
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > But as Erik pointed out volume snapshots
> > are
> > > > not
> > > > > > > > > backups.
> > > > > > > > > >> > They
> > > > > > > > > >> > > > > don't
> > > > > > > > > >> > > > > > > work
> > > > > > > > > >> > > > > > > > > form multi-disk LVM volume groups and
> > > dynamic
> > > > > > > disks. I
> > > > > > > > > am
> > > > > > > > > >> all
> > > > > > > > > >> > > in
> > > > > > > > > >> > > > > for
> > > > > > > > > >> > > > > > a
> > > > > > > > > >> > > > > > > > > better backup solution which handles
> these
> > > use
> > > > > > cases
> > > > > > > > and
> > > > > > > > > >> > > utilizes
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > > > > storage's advanced features.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul
> > Angus
> > > <
> > > > > > > > > >> > > > > > paul.angus@shapeblue.com <javascript:;>
> > > > > <javascript:;>>
> > > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > > In the beginning... there were
> > CloudStack
> > > > > > > snapshots
> > > > > > > > > and
> > > > > > > > > >> > they
> > > > > > > > > >> > > > were
> > > > > > > > > >> > > > > > > > > actually
> > > > > > > > > >> > > > > > > > > > volume snapshots not hypervisor
> > > > point-in-time
> > > > > > > > > snapshots.
> > > > > > > > > >> > > > > > > > > > Then VM snapshots were created (which
> > are
> > > > > > > > > point-in-time
> > > > > > > > > >> > > > > hypervisor
> > > > > > > > > >> > > > > > > > > > snapshots) and we started referring to
> > the
> > > > > > > original
> > > > > > > > > >> > snapshots
> > > > > > > > > >> > > > as
> > > > > > > > > >> > > > > > > volume
> > > > > > > > > >> > > > > > > > > > snapshots.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > CloudStack does not offer 'backups',
> but
> > > > many
> > > > > > > people
> > > > > > > > > use
> > > > > > > > > >> > > volume
> > > > > > > > > >> > > > > > > > snapshots
> > > > > > > > > >> > > > > > > > > > as backups. However you can't in-place
> > > > restore
> > > > > > > > volume
> > > > > > > > > >> > > snapshots
> > > > > > > > > >> > > > > and
> > > > > > > > > >> > > > > > > if
> > > > > > > > > >> > > > > > > > > you
> > > > > > > > > >> > > > > > > > > > have a VM with multiple volumes, the
> > > volume
> > > > > > > > snapshots
> > > > > > > > > >> must
> > > > > > > > > >> > be
> > > > > > > > > >> > > > > done
> > > > > > > > > >> > > > > > in
> > > > > > > > > >> > > > > > > > > > series, meaning that the state across
> > all
> > > of
> > > > > the
> > > > > > > > > >> volumes is
> > > > > > > > > >> > > > > > unlikely
> > > > > > > > > >> > > > > > > to
> > > > > > > > > >> > > > > > > > > be
> > > > > > > > > >> > > > > > > > > > consistent.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > 'Actual Backups' would enable all of
> the
> > > > > restore
> > > > > > > > > options
> > > > > > > > > >> > > which
> > > > > > > > > >> > > > > > users
> > > > > > > > > >> > > > > > > > > might
> > > > > > > > > >> > > > > > > > > > expect as well options as to where
> they
> > > > might
> > > > > be
> > > > > > > > > >> stored. In
> > > > > > > > > >> > > my
> > > > > > > > > >> > > > > > ideal
> > > > > > > > > >> > > > > > > > > world
> > > > > > > > > >> > > > > > > > > > they would also be able to leverage
> > > back-end
> > > > > > > > hardware
> > > > > > > > > >> (such
> > > > > > > > > >> > > as
> > > > > > > > > >> > > > > > > > Solidfire,
> > > > > > > > > >> > > > > > > > > > NetApp etc :) ) and software such as
> > > Veeam,
> > > > > > > > Commvault
> > > > > > > > > >> etc
> > > > > > > > > >> > to
> > > > > > > > > >> > > > > > > accelerate
> > > > > > > > > >> > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > process.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > [image: ShapeBlue] <
> > > > http://www.shapeblue.com>
> > > > > > > > > >> > > > > > > > > > Paul Angus
> > > > > > > > > >> > > > > > > > > > VP Technology , ShapeBlue
> > > > > > > > > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> > > 0540*
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > >
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > |
> > > > > > > > > >> > > > > m:
> > > > > > > > > >> > > > > > > > > > *+44 7711 418784*
> <+44%207711%20418784>
> > > > > > > > > >> > > > > > > > > > e: *paul.angus@shapeblue.com
> > > <javascript:;>
> > > > > > > > > >> > <javascript:;>
> > > > > > > > > >> > > |
> > > > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > > > >> > > > > > > > > > <paul.angus@shapeblue.com
> > <javascript:;>
> > > > > > > > > >> > > > > <ja...@cloudyangus>
> > > > > > > > > >> > > > > > | w:
> > > > > > > > > >> > > > > > > > > > *www.shapeblue.com* <
> > > > http://www.shapeblue.com
> > > > > >
> > > > > > > > > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > London
> > > > WC2N
> > > > > > > 4HS
> > > > > > > > UK
> > > > > > > > > >> > > > > > > > > > Shape Blue Ltd is a company
> incorporated
> > > in
> > > > > > > England
> > > > > > > > &
> > > > > > > > > >> > Wales.
> > > > > > > > > >> > > > > > > ShapeBlue
> > > > > > > > > >> > > > > > > > > > Services India LLP is a company
> > > incorporated
> > > > > in
> > > > > > > > India
> > > > > > > > > >> and
> > > > > > > > > >> > is
> > > > > > > > > >> > > > > > operated
> > > > > > > > > >> > > > > > > > > under
> > > > > > > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape
> Blue
> > > > Brasil
> > > > > > > > > >> Consultoria
> > > > > > > > > >> > > Ltda
> > > > > > > > > >> > > > > is
> > > > > > > > > >> > > > > > a
> > > > > > > > > >> > > > > > > > > > company incorporated in Brasil and is
> > > > operated
> > > > > > > under
> > > > > > > > > >> > license
> > > > > > > > > >> > > > from
> > > > > > > > > >> > > > > > > Shape
> > > > > > > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a
> > > company
> > > > > > > > registered
> > > > > > > > > >> by
> > > > > > > > > >> > The
> > > > > > > > > >> > > > > > > Republic
> > > > > > > > > >> > > > > > > > of
> > > > > > > > > >> > > > > > > > > > South Africa and is traded under
> license
> > > > from
> > > > > > > Shape
> > > > > > > > > Blue
> > > > > > > > > >> > Ltd.
> > > > > > > > > >> > > > > > > ShapeBlue
> > > > > > > > > >> > > > > > > > > is
> > > > > > > > > >> > > > > > > > > > a registered trademark.
> > > > > > > > > >> > > > > > > > > > This email and any attachments to it
> may
> > > be
> > > > > > > > > confidential
> > > > > > > > > >> > and
> > > > > > > > > >> > > > are
> > > > > > > > > >> > > > > > > > intended
> > > > > > > > > >> > > > > > > > > > solely for the use of the individual
> to
> > > whom
> > > > > it
> > > > > > is
> > > > > > > > > >> > addressed.
> > > > > > > > > >> > > > Any
> > > > > > > > > >> > > > > > > views
> > > > > > > > > >> > > > > > > > > or
> > > > > > > > > >> > > > > > > > > > opinions expressed are solely those of
> > the
> > > > > > author
> > > > > > > > and
> > > > > > > > > do
> > > > > > > > > >> > not
> > > > > > > > > >> > > > > > > > necessarily
> > > > > > > > > >> > > > > > > > > > represent those of Shape Blue Ltd or
> > > related
> > > > > > > > > companies.
> > > > > > > > > >> If
> > > > > > > > > >> > > you
> > > > > > > > > >> > > > > are
> > > > > > > > > >> > > > > > > not
> > > > > > > > > >> > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > intended recipient of this email, you
> > must
> > > > > > neither
> > > > > > > > > take
> > > > > > > > > >> any
> > > > > > > > > >> > > > > action
> > > > > > > > > >> > > > > > > > based
> > > > > > > > > >> > > > > > > > > > upon its contents, nor copy or show it
> > to
> > > > > > anyone.
> > > > > > > > > Please
> > > > > > > > > >> > > > contact
> > > > > > > > > >> > > > > > the
> > > > > > > > > >> > > > > > > > > sender
> > > > > > > > > >> > > > > > > > > > if you believe you have received this
> > > email
> > > > in
> > > > > > > > error.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > -----Original Message-----
> > > > > > > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > > >> > > > > <javascript:;> <javascript:;>]
> > > > > > > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > > > > >> > > > > > > > > > To: dev@cloudstack.apache.org
> > > > <javascript:;>
> > > > > > > > > >> > <javascript:;>
> > > > > > > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature]
> > > Storage
> > > > > > > > Snapshots
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Paul,
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > When you say actual backups, how would
> > it
> > > be
> > > > > > > > different
> > > > > > > > > >> from
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > > Volume
> > > > > > > > > >> > > > > > > > > > Snapshots that exist currently. My
> > > > > understanding
> > > > > > > is
> > > > > > > > > that
> > > > > > > > > >> > > > Backups
> > > > > > > > > >> > > > > > end
> > > > > > > > > >> > > > > > > up
> > > > > > > > > >> > > > > > > > > in
> > > > > > > > > >> > > > > > > > > > Sec Storage whereas Snapshots are
> just a
> > > > > > > > point-in-time
> > > > > > > > > >> > state
> > > > > > > > > >> > > of
> > > > > > > > > >> > > > > > your
> > > > > > > > > >> > > > > > > > > volume
> > > > > > > > > >> > > > > > > > > > which can be restored back correct?
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > -Syed
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul
> > > Angus
> > > > <
> > > > > > > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> > > > > > <javascript:;>>
> > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Hi Syed,
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > As I understand it, the SolidFire
> > plugin
> > > > > will
> > > > > > > > export
> > > > > > > > > >> the
> > > > > > > > > >> > > > > snapshot
> > > > > > > > > >> > > > > > > to
> > > > > > > > > >> > > > > > > > > > > secondary storage if the user
> > requests a
> > > > > > > template
> > > > > > > > > from
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > snapshot
> > > > > > > > > >> > > > > > > > or
> > > > > > > > > >> > > > > > > > > > > wants to download the snapshot from
> > the
> > > > > cloud.
> > > > > > > > This
> > > > > > > > > >> is a
> > > > > > > > > >> > > > good,
> > > > > > > > > >> > > > > > > > > > > pragmatic approach and yes Mike the
> > > > > SolidFire
> > > > > > > > > storage
> > > > > > > > > >> is
> > > > > > > > > >> > > > super
> > > > > > > > > >> > > > > > > > > > > reliable and snapshots on SolidFire
> > > arrays
> > > > > > take
> > > > > > > up
> > > > > > > > > >> next
> > > > > > > > > >> > to
> > > > > > > > > >> > > no
> > > > > > > > > >> > > > > > > space.
> > > > > > > > > >> > > > > > > > > > > BUT I think that we are talking
> about
> > a
> > > > more
> > > > > > > > general
> > > > > > > > > >> > > purpose
> > > > > > > > > >> > > > > API,
> > > > > > > > > >> > > > > > > and
> > > > > > > > > >> > > > > > > > > > > other storage systems may not be as
> > > > awesome
> > > > > as
> > > > > > > > > Mike's.
> > > > > > > > > >> > > That's
> > > > > > > > > >> > > > > my
> > > > > > > > > >> > > > > > > > > > > concern. Also, the time to transfer
> > for
> > > > say
> > > > > > 1TB
> > > > > > > to
> > > > > > > > > >> move
> > > > > > > > > >> > > from
> > > > > > > > > >> > > > > > > primary
> > > > > > > > > >> > > > > > > > > > > to sec storage and then create a VM
> > > > template
> > > > > > out
> > > > > > > > of
> > > > > > > > > it
> > > > > > > > > >> > may
> > > > > > > > > >> > > be
> > > > > > > > > >> > > > > too
> > > > > > > > > >> > > > > > > > long
> > > > > > > > > >> > > > > > > > > > for users.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the
> term
> > > > volume
> > > > > > > > > snapshot
> > > > > > > > > >> for
> > > > > > > > > >> > > > > backup,
> > > > > > > > > >> > > > > > > > it's
> > > > > > > > > >> > > > > > > > > > > just that users want to do backups
> > and a
> > > > > > volume
> > > > > > > > > >> snapshot
> > > > > > > > > >> > is
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > only
> > > > > > > > > >> > > > > > > > > > > type of snapshot that copies the
> disk
> > > > > > elsewhere
> > > > > > > > and
> > > > > > > > > >> can
> > > > > > > > > >> > be
> > > > > > > > > >> > > > > > > scheduled.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > I'm 'pondering' the implications of
> > > > enabling
> > > > > > > > actual
> > > > > > > > > >> > backups
> > > > > > > > > >> > > > > > > (through
> > > > > > > > > >> > > > > > > > > > > recognised backup providers) and the
> > > user
> > > > > > > > > requirements
> > > > > > > > > >> > > around
> > > > > > > > > >> > > > > > them
> > > > > > > > > >> > > > > > > > > > > (particularly restoration use cases)
> > as
> > > a
> > > > > > > separate
> > > > > > > > > >> thread
> > > > > > > > > >> > > of
> > > > > > > > > >> > > > > > work.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > [image: ShapeBlue] <
> > > > > http://www.shapeblue.com>
> > > > > > > > Paul
> > > > > > > > > >> Angus
> > > > > > > > > >> > > VP
> > > > > > > > > >> > > > > > > > Technology
> > > > > > > > > >> > > > > > > > > > > , ShapeBlue
> > > > > > > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203
> 603
> > > > 0540*
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > >
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > > > >> > > > |
> > > > > > > > > >> > > > > m:
> > > > > > > > > >> > > > > > > > > > > *+44 7711 418784*
> > <+44%207711%20418784>
> > > > > > > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > > <javascript:;>
> > > > > > > > > >> > > <javascript:;> |
> > > > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > > > >> > > > > > > > > > > <paul.angus@shapeblue.com
> > > <javascript:;>
> > > > > > > > > >> <javascript:;>
> > > > > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > >> > > > > > > > > > > *www.shapeblue.com* <
> > > > > http://www.shapeblue.com
> > > > > > >
> > > > > > > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > > London
> > > > > WC2N
> > > > > > > 4HS
> > > > > > > > > UK
> > > > > > > > > >> > Shape
> > > > > > > > > >> > > > > Blue
> > > > > > > > > >> > > > > > > Ltd
> > > > > > > > > >> > > > > > > > > > > is a company incorporated in
> England &
> > > > > Wales.
> > > > > > > > > >> ShapeBlue
> > > > > > > > > >> > > > > Services
> > > > > > > > > >> > > > > > > > India
> > > > > > > > > >> > > > > > > > > > > LLP is a company incorporated in
> India
> > > and
> > > > > is
> > > > > > > > > operated
> > > > > > > > > >> > > under
> > > > > > > > > >> > > > > > > license
> > > > > > > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue
> Brasil
> > > > > > > Consultoria
> > > > > > > > > >> Ltda
> > > > > > > > > >> > is
> > > > > > > > > >> > > a
> > > > > > > > > >> > > > > > > company
> > > > > > > > > >> > > > > > > > > > > incorporated in Brasil and is
> operated
> > > > under
> > > > > > > > license
> > > > > > > > > >> from
> > > > > > > > > >> > > > Shape
> > > > > > > > > >> > > > > > > Blue
> > > > > > > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a
> company
> > > > > > > registered
> > > > > > > > by
> > > > > > > > > >> The
> > > > > > > > > >> > > > > Republic
> > > > > > > > > >> > > > > > > of
> > > > > > > > > >> > > > > > > > > > > South Africa and is traded under
> > license
> > > > > from
> > > > > > > > Shape
> > > > > > > > > >> Blue
> > > > > > > > > >> > > Ltd.
> > > > > > > > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > > > > > > >> > > > > > > > > > > This email and any attachments to it
> > may
> > > > be
> > > > > > > > > >> confidential
> > > > > > > > > >> > > and
> > > > > > > > > >> > > > > are
> > > > > > > > > >> > > > > > > > > > > intended solely for the use of the
> > > > > individual
> > > > > > to
> > > > > > > > > whom
> > > > > > > > > >> it
> > > > > > > > > >> > is
> > > > > > > > > >> > > > > > > > addressed.
> > > > > > > > > >> > > > > > > > > > > Any views or opinions expressed are
> > > solely
> > > > > > those
> > > > > > > > of
> > > > > > > > > >> the
> > > > > > > > > >> > > > author
> > > > > > > > > >> > > > > > and
> > > > > > > > > >> > > > > > > do
> > > > > > > > > >> > > > > > > > > > > not necessarily represent those of
> > Shape
> > > > > Blue
> > > > > > > Ltd
> > > > > > > > or
> > > > > > > > > >> > > related
> > > > > > > > > >> > > > > > > > > > > companies. If you are not the
> intended
> > > > > > recipient
> > > > > > > > of
> > > > > > > > > >> this
> > > > > > > > > >> > > > email,
> > > > > > > > > >> > > > > > you
> > > > > > > > > >> > > > > > > > > > > must neither take any action based
> > upon
> > > > its
> > > > > > > > > contents,
> > > > > > > > > >> nor
> > > > > > > > > >> > > > copy
> > > > > > > > > >> > > > > or
> > > > > > > > > >> > > > > > > > show
> > > > > > > > > >> > > > > > > > > > > it to anyone. Please contact the
> > sender
> > > if
> > > > > you
> > > > > > > > > believe
> > > > > > > > > >> > you
> > > > > > > > > >> > > > have
> > > > > > > > > >> > > > > > > > > received
> > > > > > > > > >> > > > > > > > > > this email in error.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > -----Original Message-----
> > > > > > > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > > >> > > > > <javascript:;>
> > > > > > > > > >> > > > > > <javascript:;>]
> > > > > > > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > <javascript:;>
> > > > > > > > > >> > > <javascript:;>
> > > > > > > > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature]
> > > > Storage
> > > > > > > > > Snapshots
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > I think the terminology confusion
> > comes
> > > > from
> > > > > > AWS
> > > > > > > > > where
> > > > > > > > > >> > they
> > > > > > > > > >> > > > do
> > > > > > > > > >> > > > > > EBS
> > > > > > > > > >> > > > > > > > > > > snapshots backed up to S3 and
> > CloudStack
> > > > > sort
> > > > > > of
> > > > > > > > > >> followed
> > > > > > > > > >> > > > that.
> > > > > > > > > >> > > > > > And
> > > > > > > > > >> > > > > > > > as
> > > > > > > > > >> > > > > > > > > > > an end user who is oblivious to the
> > > > > internals
> > > > > > of
> > > > > > > > my
> > > > > > > > > >> > > provider,
> > > > > > > > > >> > > > > my
> > > > > > > > > >> > > > > > > > > > > expectation would be something
> similar
> > > to
> > > > > what
> > > > > > > AWS
> > > > > > > > > as
> > > > > > > > > >> > that
> > > > > > > > > >> > > is
> > > > > > > > > >> > > > > my
> > > > > > > > > >> > > > > > > > > > > biggest reference point.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > To your point Mike, I agree that a
> > > Primary
> > > > > > > Storage
> > > > > > > > > >> > failure
> > > > > > > > > >> > > on
> > > > > > > > > >> > > > > > > > > > > SolidFire is unlikely, there are
> other
> > > > > > > motivations
> > > > > > > > > >> for us
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > push
> > > > > > > > > >> > > > > > > > data
> > > > > > > > > >> > > > > > > > > > > to secondary storage. Primary
> storage
> > > > > (atleast
> > > > > > > for
> > > > > > > > > us)
> > > > > > > > > >> > > costs
> > > > > > > > > >> > > > > > > around 3
> > > > > > > > > >> > > > > > > > > > > times as much as secondary storage
> and
> > > > > > snapshots
> > > > > > > > on
> > > > > > > > > >> > primary<> > > > > > > > take 2
> > > > > > > > > >> > days
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > > > > > complete.
> > > > > > > > > >> > > > > > > > > > > > > > e. snapshots (as they are)
> can't
> > > be
> > > > > used
> > > > > > > for
> > > > > > > > > >> > multiple
> > > > > > > > > >> > > > LVM
> > > > > > > > > >> > > > > > > > disks.
> > > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > > > I think the process Mike has
> > used
> > > in
> > > > > the
> > > > > > > > > >> SolidFire
> > > > > > > > > >> > > > plugin
> > > > > > > > > >> > > > > > > (only
> > > > > > > > > >> > > > > > > > > > > > > > moving
> > > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > > > > > disk image to secondary
> storage
> > > when
> > > > > you
> > > > > > > > > >> absolutely
> > > > > > > > > >> > > > have
> > > > > > > > > >> > > > > > to)
> > > > > > > > > >> > > > > > > is
> > > > > > > > > >> > > > > > > > > > > > > > a very
> > > > > > > > > >> > > > > > > > > > > > > good
> > > > > > > > > >> > > > > > > > > > > > > > and pragmatic solution. I
> wonder
> > > > what
> > > > > > > > problems
> > > > > > > > > >> an
> > > > > > > > > >> > > > > operator
> > > > > > > > > >> > > > > > > > might
> > > > > > > > > >> > > > > > > > > > > > > experience
> > > > > > > > > >> > > > > > > > > > > > > > if they have an issue with a
> > given
> > > > > > primary
> > > > > > > > > >> storage
> > > > > > > > > >> > > pool
> > > > > > > > > >> > > > > in
> > > > > > > > > >> > > > > > a
> > > > > > > > > >> > > > > > > > > > cluster.
> > > > > > > > > >> > > > > > > > > > > > (I
> > > > > > > > > >> > > > > > > > > > > > > > know that that is REALLY
> > unlikely
> > > in
> > > > > the
> > > > > > > > > >> SolidFire
> > > > > > > > > >> > > case
> > > > > > > > > >> > > > > > Mike
> > > > > > > > > >> > > > > > > :)
> > > > > > > > > >> > > > > > > > > > > > > > ) And
> > > > > > > > > >> > > > > > > > > > > > if
> > > > > > > > > >> > > > > > > > > > > > > > the transfer from primary to
> > > > secondary
> > > > > > is
> > > > > > > > > slow,
> > > > > > > > > >> the
> > > > > > > > > >> > > > time
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > > > > > > > > being able
> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > [image: Shape



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

> Situation: Global setting value is changed to a non-default value.
>
> 2) Action: Volume snapshot is taken with managed storage. Result: ??? I'm
> not sure which scenario we're looking at here now.
>

In this situation, volume snapshots will end up on secondary storage and
the snapshot on the managed storage will be deleted.


>
> 3) Action: Storage snapshot is taken with managed storage. Result: ??? I'm
> not sure which scenario we're looking at here now.
>

In this situation, the snapshot stays on the managed storage.

Does this answer your question Mike?





On Fri, Feb 12, 2016 at 11:43 AM, Mike Tutkowski <
mike.tutkowski@solidfire.com> wrote:

> Hi Syed,
>
> Can you clarify how you see these behaviors (below) working now that we are
> considering a global settings value applicable to managed storage with
> volume snapshots?
>
> State: Global setting value defaults to keep snapshot on storage system
> (primary storage) for managed storage.
>
> 1) Action: Volume snapshot is taken with managed storage. Result: Snapshot
> is taken and kept on storage system (primary storage). (This is what you
> and I have been working on for a couple months now.)
>
>
>
> Thanks!
> Mike
>
> On Fri, Feb 12, 2016 at 9:15 AM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
> > I agree with Mike's concern about backward compatibility. We can add a
> > global flag which makes sure that the way volume snapshots work currently
> > on managed storage (stay on the device) is retained after upgrade. We can
> > then safely implement the Storage Snapshot API while making the Volume
> > Snpashot API move the snapshot to Secondary Storage.
> >
> > Sounds good guys?
> >
> > -Syed
> >
> >
> > On Mon, Feb 8, 2016 at 2:56 PM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com
> > > wrote:
> >
> > > Here's what we have for snapshots for managed storage as of 4.6, Paul:
> > >
> > > 1. VM snapshots (no proposed changes to this).
> > >
> > > 2. Volume snapshots that do not end up on secondary storage, but rather
> > are
> > > stored on a SAN (effectively storing snapshots on primary storage).
> > >
> > > Pierre-Luc is saying he'd like this for snapshots for managed storage:
> > >
> > > A. VM snapshots (no proposed changes to this).
> > >
> > > B. Volume snapshots that export to secondary storage.
> > >
> > > C. New: Storage snapshots that behave like 2 (above).
> > >
> > > I like Pierre-Luc's ideas there, but the problem is backward
> > compatibility.
> > >
> > > If customers who were using managed storage with volume snapshots in
> 4.6
> > > were getting their snapshots put on a SAN, then in 4.9 - all of a
> sudden
> > -
> > > their new snapshots are put on secondary storage (unless they
> explicitly
> > > change over to using the new Storage snapshots feature).
> > >
> > >
> > >
> > > On Mon, Feb 8, 2016 at 12:32 PM, Paul Angus <pa...@shapeblue.com>
> > > wrote:
> > >
> > > > Just to make sure I'm on the same page, are we talking about;
> > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9278 ?
> > > >
> > > > The FS reads (to me) more like 1a + the possibility to export to
> > > secondary
> > > > storage if required?
> > > > Have I understood correctly?
> > > > I have seen [1a] implemented for VMware by NetApp in their beta
> > > CloudStack
> > > > plugin (pleased I can say that without Mike beating me up now). No
> > > changes
> > > > to the CloudStack API were required. (nb it didn't export to
> secondary
> > > > storage).
> > > >
> > > >
> > > >
> > > > 1. VM Snapshot (point-in-time hypervisor based snapshots)
> > > > 1a. SAN assisted VM snapshots (point-in-time hypervisor snapshot
> takes
> > > > place on transparently SAN to avoid performance issue in disk chains)
> > > > 2. SAN Snapshot (Storage Snapshot) - NEW
> > > > 3. Volume Snapshot (current old/slow transfer to secstorage)
> > > > 4. Backup - JUST AN IDEAL.
> > > >
> > > >
> > > >
> > > >
> > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > Paul Angus
> > > > VP Technology ,  ShapeBlue
> > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > > > *+44 7711 418784* <+44%207711%20418784>
> > > > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > <pa...@cloudyangus>  |  w:
> > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > Shape Blue Ltd is a company incorporated in England & Wales.
> ShapeBlue
> > > > Services India LLP is a company incorporated in India and is operated
> > > under
> > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > > > company incorporated in Brasil and is operated under license from
> Shape
> > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> Republic
> > of
> > > > South Africa and is traded under license from Shape Blue Ltd.
> ShapeBlue
> > > is
> > > > a registered trademark.
> > > > This email and any attachments to it may be confidential and are
> > intended
> > > > solely for the use of the individual to whom it is addressed. Any
> views
> > > or
> > > > opinions expressed are solely those of the author and do not
> > necessarily
> > > > represent those of Shape Blue Ltd or related companies. If you are
> not
> > > the
> > > > intended recipient of this email, you must neither take any action
> > based
> > > > upon its contents, nor copy or show it to anyone. Please contact the
> > > sender
> > > > if you believe you have received this email in error.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
> > > > Sent: Monday, February 8, 2016 7:16 PM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > >
> > > > Correct, Will.
> > > >
> > > > That Global Settings would only be for managed storage. Non-managed
> > > > (traditional) volume snapshots are completely un-impacted by this
> > > feature.
> > > >
> > > > If we need to sometimes keep the snapshots on the SAN and sometimes
> > push
> > > > them to secondary storage, we'll need a more robust solution than
> > Global
> > > > Settings, though.
> > > >
> > > > On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <wstevens@cloudops.com
> >
> > > > wrote:
> > > >
> > > > > Sorry. I missed a bit of context when I responded. The global
> setting
> > > > > would be only for the managed storage case, currently being called
> > > > Storage
> > > > > Snapshots, and is only to determine if a copy is pushed to
> secondary
> > > > > storage right? The global setting would not change the behavior of
> > the
> > > > > Volume Snapshots right?
> > > > >
> > > > > I was referring to the need for Volume Snapshots and Storage
> > Snapshots
> > > to
> > > > > exist together. Disregard my comment. I caught up on context after
> I
> > > > > posted. My bad...
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > w cloudops.com *|* tw @CloudOps_
> > > > >
> > > > > On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > Hey Will,
> > > > > >
> > > > > > Who's picking the behavior? Is it the cloud provider or the end
> > user?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <
> > wstevens@cloudops.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I don't think a global setting is a good option because we need
> > > both
> > > > > > > functionality to be available at the same time and for
> different
> > > use
> > > > > > cases
> > > > > > > to be able to pick which they choose.
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > > > w cloudops.com *|* tw @CloudOps_
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > > > > > mike.tutkowski@solidfire.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Now that I re-read your e-mail, it dawned on me: The end-user
> > > > doesn't
> > > > > > > care
> > > > > > > > where the snapshot is.
> > > > > > > >
> > > > > > > > If that's true, then we should perhaps control this via
> Global
> > > > > Settings
> > > > > > > or
> > > > > > > > something.
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > > > > > mike.tutkowski@solidfire.com> wrote:
> > > > > > > >
> > > > > > > > > It's not ideal - true, but it does allow us to be backward
> > > > > > compatible.
> > > > > > > > >
> > > > > > > > > If you have other ideas, though, about how to maintain
> > backward
> > > > > > > > > compatibility, I'm definitely open to hear them.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > > > > > syed1.mushtaq@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi Mike,
> > > > > > > > >>
> > > > > > > > >> Adding a flag to createSnapshot was the first and the most
> > > > obvious
> > > > > > > thing
> > > > > > > > >> that came to our minds. The problem that I had with this
> was
> > > > that:
> > > > > > > > >>
> > > > > > > > >> 1) I feel it is exposing something to the end user that is
> > > > > internal
> > > > > > to
> > > > > > > > the
> > > > > > > > >> cloud.
> > > > > > > > >>
> > > > > > > > >> 2) We have to follow two different ways of
> restore/deletion
> > in
> > > > the
> > > > > > > same
> > > > > > > > >> code path depending on where the Snapshot resides which I
> > find
> > > > > kind
> > > > > > > of a
> > > > > > > > >> bad design.
> > > > > > > > >>
> > > > > > > > >> But if exposing a archive flag to the end user is
> acceptable
> > > > then
> > > > > we
> > > > > > > can
> > > > > > > > >> definitely use this instead of adding the StorageSnapshot
> > API
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> -Syed
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > > > > > >> mike.tutkowski@solidfire.com
> > > > > > > > >> > wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Pierre-Luc,
> > > > > > > > >> >
> > > > > > > > >> > My recommendation would be this:
> > > > > > > > >> >
> > > > > > > > >> > Add an "archive" flag to the current volume-snapshot
> API.
> > > Its
> > > > > > > default
> > > > > > > > >> would
> > > > > > > > >> > be "false" because that would be backward compatible
> with
> > > how
> > > > > 4.6
> > > > > > > has
> > > > > > > > >> > volume snapshots implemented (i.e. they stay on the SAN
> in
> > > > 4.6,
> > > > > > 4.7,
> > > > > > > > and
> > > > > > > > >> > 4.8).
> > > > > > > > >> >
> > > > > > > > >> > If you set archive=true, then we would perform a
> > background
> > > > > > > migration
> > > > > > > > of
> > > > > > > > >> > the snapshot from the SAN to the secondary storage (then
> > > > delete
> > > > > > the
> > > > > > > > SAN
> > > > > > > > >> > snapshot).
> > > > > > > > >> >
> > > > > > > > >> > That archive parameter would only be valid for managed
> > > > storage.
> > > > > > > > >> >
> > > > > > > > >> > Sound reasonable?
> > > > > > > > >> >
> > > > > > > > >> > Also, a VM snapshot that includes disks provided by
> > managed
> > > > > > storage
> > > > > > > > >> should
> > > > > > > > >> > work.
> > > > > > > > >> >
> > > > > > > > >> > Talk to you later,
> > > > > > > > >> > Mike
> > > > > > > > >> >
> > > > > > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > > > > > pdion@cloudops.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Mike,
> > > > > > > > >> > >
> > > > > > > > >> > > In terms of API's, would you prefer introducing a
> > > parameter
> > > > to
> > > > > > the
> > > > > > > > >> > existing
> > > > > > > > >> > > VolumeSnapshot, example: extract={true|false} with a
> > > > > default
> > > > > > > > value
> > > > > > > > >> of
> > > > > > > > >> > > true which would extract snapshot into the secondary
> > > > storage,
> > > > > > > which
> > > > > > > > >> is
> > > > > > > > >> > the
> > > > > > > > >> > > current default behavior. Then for SAN snapshot that
> > > remain
> > > > on
> > > > > > the
> > > > > > > > >> SAN we
> > > > > > > > >> > > would just set "extract=false" ? as oppose to create a
> > new
> > > > > > > > >> > > StorageSnapshot API ?
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > Paul,
> > > > > > > > >> > >
> > > > > > > > >> > > From what I'm seeing so far, we can't do a VM-snapshot
> > > when
> > > > > > using
> > > > > > > > >> managed
> > > > > > > > >> > > storage for VM having more than one Volume. For the
> > reason
> > > > > that
> > > > > > > > >> snapshot
> > > > > > > > >> > > are performed outside of the hypervisor awareness and
> > > > > > > > asynchronously.
> > > > > > > > >> If
> > > > > > > > >> > > someone have a way to address that, it would make
> thinks
> > > > much
> > > > > > more
> > > > > > > > >> > > attractive.
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <
> > > irae@cloudops.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > I think a service provider backup scenario is more
> > > likely
> > > > to
> > > > > > > take
> > > > > > > > >> > > advantage
> > > > > > > > >> > > > of SAN snapshot. There are a few reasons for this.
> > > > > Traditional
> > > > > > > > >> backups
> > > > > > > > >> > > > involve access to the file system, and there is an
> > > > > expectation
> > > > > > > > that
> > > > > > > > >> > this
> > > > > > > > >> > > > can be done with reasonably short time frames
> without
> > > > > > negatively
> > > > > > > > >> > > impacting
> > > > > > > > >> > > > VM performance, and that the backup orchestrator can
> > > apply
> > > > > > > various
> > > > > > > > >> > logic
> > > > > > > > >> > > > and or transformations to the data (compress,
> encrypt,
> > > > > deltas
> > > > > > > > >> etc...).
> > > > > > > > >> > > > While it is true that one could apply a backup
> process
> > > to
> > > > a
> > > > > > > cloud
> > > > > > > > >> > > snapshot,
> > > > > > > > >> > > > this would be slow and inefficient requiring the
> data
> > to
> > > > be
> > > > > > > moved
> > > > > > > > >> > several
> > > > > > > > >> > > > times and there are some major bottlenecks with
> cloud
> > > > > > snapshots.
> > > > > > > > >> With a
> > > > > > > > >> > > > cloud snapshot - there seems to be no reasonable
> > > > expectation
> > > > > > of
> > > > > > > > >> being
> > > > > > > > >> > > able
> > > > > > > > >> > > > to do differential snapshots (I think this depends
> on
> > > the
> > > > > > > > >> hypervisor)
> > > > > > > > >> > and
> > > > > > > > >> > > > if you do differential snapshots this will make file
> > > > backups
> > > > > > > from
> > > > > > > > >> those
> > > > > > > > >> > > > snapshots even more complicated to orchestrate.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Suspect there needs to be a different thread on how
> to
> > > > > better
> > > > > > > > enable
> > > > > > > > >> > > > backups, as a service. As per Paul's suggestion, but
> > it
> > > > is a
> > > > > > > > related
> > > > > > > > >> > > > workflow so relevant to this discussion.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Ian
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > > > > > >> > > mike.tutkowski@solidfire.com>
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > To me it sounds like number two and number three
> are
> > > > > > different
> > > > > > > > >> uses
> > > > > > > > >> > for
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > same "thing"(which is totally fine).
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > > > > > > >> asynchronously, do
> > > > > > > > >> > > we
> > > > > > > > >> > > > > see the SSVM as performing the export?
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > To be backwards compatible with what we have in
> 4.6
> > > and
> > > > > > later
> > > > > > > > for
> > > > > > > > >> > > volume
> > > > > > > > >> > > > > snapshots for managed storage, I think it might be
> > > > easier
> > > > > if
> > > > > > > we
> > > > > > > > >> pass
> > > > > > > > >> > > in a
> > > > > > > > >> > > > > flag that says whether or not to archive the SAN
> > > > snapshot
> > > > > > > > (which,
> > > > > > > > >> I
> > > > > > > > >> > > > think,
> > > > > > > > >> > > > > is something that you suggested, as well,
> > Pierre-Luc).
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > > > > > pdion@cloudops.com
> > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Hi Mike,
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > The reason behind the creation of a SAN snapshot
> > > which
> > > > > is
> > > > > > > > >> exported
> > > > > > > > >> > > into
> > > > > > > > >> > > > > > secondary storage, is because creating a copy of
> > the
> > > > > .VHD
> > > > > > > > >> directly
> > > > > > > > >> > > > would
> > > > > > > > >> > > > > > impact uptime of the VM as creating that copy
> take
> > > > lots
> > > > > of
> > > > > > > > time.
> > > > > > > > >> > Has
> > > > > > > > >> > > > > oppose
> > > > > > > > >> > > > > > to a SAN snapshot that is close to instantaneous
> > > which
> > > > > can
> > > > > > > > >> > afterward
> > > > > > > > >> > > be
> > > > > > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > I would suspect an extracted VolumeSnapshot
> taken
> > > > from a
> > > > > > SAN
> > > > > > > > >> > snapshot
> > > > > > > > >> > > > > could
> > > > > > > > >> > > > > > have is SAN snapshot deleted to avoid duplica
> and
> > > > space
> > > > > > > > >> consumption
> > > > > > > > >> > > on
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > Primary Storage side.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > I see 3 definitions in our current discussion
> > > > regarding
> > > > > > the
> > > > > > > > term
> > > > > > > > >> > > > snapshot
> > > > > > > > >> > > > > > (these are not official terminology but by own
> > > > > > > interpretation
> > > > > > > > of
> > > > > > > > >> > > them):
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's
> > > > definition
> > > > > > of
> > > > > > > a
> > > > > > > > >> > > > snapshot):
> > > > > > > > >> > > > > > it's a volume snapshot at the storage level,
> point
> > > in
> > > > > time
> > > > > > > of
> > > > > > > > >> your
> > > > > > > > >> > > > data.
> > > > > > > > >> > > > > it
> > > > > > > > >> > > > > > reside on the primary storage. Useful and
> > efficient
> > > > for
> > > > > > > > software
> > > > > > > > >> > side
> > > > > > > > >> > > > > > incident.
> > > > > > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack
> > > VolumeSnapshot/
> > > > > > cloud
> > > > > > > > >> backup
> > > > > > > > >> > > > aws-S3
> > > > > > > > >> > > > > > style ): Point in time copy of the Virtual Disk
> > that
> > > > > > reside
> > > > > > > > on a
> > > > > > > > >> > > > > different
> > > > > > > > >> > > > > > storage array then the original Volume.
> Facilitate
> > > > data
> > > > > > > > >> migration
> > > > > > > > >> > > > between
> > > > > > > > >> > > > > > clusters and, in case of primary storage
> incident,
> > > > > Volume
> > > > > > > > >> snapshots
> > > > > > > > >> > > are
> > > > > > > > >> > > > > not
> > > > > > > > >> > > > > > impacted and can be reuse.
> > > > > > > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines
> > data
> > > > that
> > > > > > > also
> > > > > > > > >> > > validate
> > > > > > > > >> > > > > data
> > > > > > > > >> > > > > > integrity, provide a storage efficient archiving
> > > > method
> > > > > > and
> > > > > > > an
> > > > > > > > >> > > > > independent
> > > > > > > > >> > > > > > way to restore your data in case of an major
> > > > > > infrastructure
> > > > > > > > >> > disaster.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Regards,
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > PL
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > > > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > > > <javascript:;>
> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > So, let's see if I currently follow the
> > > > requirements:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > * Augment volume snapshots for managed storage
> > to
> > > > > > > > >> conditionally
> > > > > > > > >> > > > export
> > > > > > > > >> > > > > > data
> > > > > > > > >> > > > > > > to NFS. The current process of taking a
> snapshot
> > > on
> > > > > the
> > > > > > > SAN
> > > > > > > > is
> > > > > > > > >> > > fine,
> > > > > > > > >> > > > > but
> > > > > > > > >> > > > > > > we'd like the option to export the data to
> NFS,
> > as
> > > > > well.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Questions:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Once the data has been exported to NFS, do we
> > keep
> > > > the
> > > > > > SAN
> > > > > > > > >> > snapshot
> > > > > > > > >> > > > or
> > > > > > > > >> > > > > > > delete it?
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > If we are deleting the SAN snapshot, then why
> > > don't
> > > > we
> > > > > > > just
> > > > > > > > >> copy
> > > > > > > > >> > > the
> > > > > > > > >> > > > > VHD
> > > > > > > > >> > > > > > > from primary to secondary the way we do today
> > for
> > > > > > > > non-managed
> > > > > > > > >> > (i.e.
> > > > > > > > >> > > > > > > traditional) storage? Why create a SAN
> snapshot
> > in
> > > > > that
> > > > > > > > >> scenario?
> > > > > > > > >> > > > > Perhaps
> > > > > > > > >> > > > > > > to have the SSVM mount and perform the VHD
> copy
> > to
> > > > > > > secondary
> > > > > > > > >> > > storage
> > > > > > > > >> > > > > > > instead of a XenServer host?
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Thanks for the clarification.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > By the way, to me a backup is when you copy
> data
> > > > from
> > > > > > one
> > > > > > > > >> storage
> > > > > > > > >> > > > > system
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > another (regardless of features, if any, to
> > > restore
> > > > > the
> > > > > > > data
> > > > > > > > >> in
> > > > > > > > >> > the
> > > > > > > > >> > > > > > > future). A snapshot is a point-in-time view of
> > the
> > > > > data
> > > > > > > of a
> > > > > > > > >> > volume
> > > > > > > > >> > > > and
> > > > > > > > >> > > > > > > it's stored on the same storage system as the
> > > > volume.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc
> > Dion <
> > > > > > > > >> > > pdion@cloudops.com
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > <javascript:;>>
> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > That's fun to see that discussion
> happening. I
> > > > 100%
> > > > > > > agree
> > > > > > > > >> with
> > > > > > > > >> > > > Paul's
> > > > > > > > >> > > > > > > > points of view. VolumeSnapshot are not a
> > backup,
> > > > > but I
> > > > > > > do
> > > > > > > > >> > > consider
> > > > > > > > >> > > > > them
> > > > > > > > >> > > > > > > as
> > > > > > > > >> > > > > > > > a safety vest against Primary Storage
> failure,
> > > > > because
> > > > > > > > >> failure
> > > > > > > > >> > > > append
> > > > > > > > >> > > > > > > :-( .
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > The current proposal around snapshots that
> > > reside
> > > > on
> > > > > > the
> > > > > > > > >> > primary
> > > > > > > > >> > > > > > storage
> > > > > > > > >> > > > > > > or
> > > > > > > > >> > > > > > > > snapshots that end in the Secondary Storage
> is
> > > not
> > > > > to
> > > > > > > > >> address
> > > > > > > > >> > > any
> > > > > > > > >> > > > > kind
> > > > > > > > >> > > > > > > of
> > > > > > > > >> > > > > > > > backups requirement because a snapshot is
> not
> > a
> > > > > > backup,
> > > > > > > > >> event
> > > > > > > > >> > an
> > > > > > > > >> > > > > > > extracted
> > > > > > > > >> > > > > > > > VM snapshot.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > The main idea, and again this is for managed
> > > > > storage;
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side
> > > > snapshot
> > > > > > > > >> capability
> > > > > > > > >> > > for
> > > > > > > > >> > > > > > fast
> > > > > > > > >> > > > > > > > response time that support rollback to
> > previous
> > > > > > > timestamp,
> > > > > > > > >> > create
> > > > > > > > >> > > > new
> > > > > > > > >> > > > > > > > volume and maybe create template.
> > > > > > > > >> > > > > > > > not required to be a new API if the work is
> > > > > > already
> > > > > > > > >> done, I
> > > > > > > > >> > > > think
> > > > > > > > >> > > > > > > this
> > > > > > > > >> > > > > > > > is a different behaviors than the user
> > > expectation
> > > > > of
> > > > > > a
> > > > > > > > >> > > > > > volume-snapshot.
> > > > > > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current
> > cloudstack
> > > > > > > behavior
> > > > > > > > >> that
> > > > > > > > >> > > > create
> > > > > > > > >> > > > > > an
> > > > > > > > >> > > > > > > > extraction of a volume into SecondaryStorage
> > > which
> > > > > can
> > > > > > > be
> > > > > > > > >> reuse
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > create a
> > > > > > > > >> > > > > > > > new volume into another Primary Storage.
> This
> > > type
> > > > > of
> > > > > > > > >> snapshot
> > > > > > > > >> > > is a
> > > > > > > > >> > > > > > slow
> > > > > > > > >> > > > > > > > job since yes it would have to copy the full
> > > > volume
> > > > > > size
> > > > > > > > on
> > > > > > > > >> the
> > > > > > > > >> > > > > > Secondary
> > > > > > > > >> > > > > > > > storage.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > PL
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed
> Mushtaq
> > <
> > > > > > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > > > >> > > > > > <javascript:;>>
> > > > > > > > >> > > > > > > > wrote:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > I think I share you view on the 'Ideal
> > world'.
> > > > > > Backup
> > > > > > > > (via
> > > > > > > > >> > > Volume
> > > > > > > > >> > > > > > > > > Snapshots) is a huge bottleneck in
> > Cloudstack.
> > > > > This
> > > > > > is
> > > > > > > > >> > > amplified
> > > > > > > > >> > > > > > > > especially
> > > > > > > > >> > > > > > > > > when you have a object storage as your
> > > secondary
> > > > > > > storage
> > > > > > > > >> > > because
> > > > > > > > >> > > > it
> > > > > > > > >> > > > > > > > > requires two copies (one to an NFS staging
> > > area
> > > > > and
> > > > > > > from
> > > > > > > > >> > there
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > object
> > > > > > > > >> > > > > > > > > storage). And not to mention that all
> these
> > > > copies
> > > > > > are
> > > > > > > > >> > > consuming
> > > > > > > > >> > > > > > > > hypervisor
> > > > > > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> > > > > > bottleneck
> > > > > > > as
> > > > > > > > >> all
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > Network
> > > > > > > > >> > > > > > > > > and I/O flow through it. So our intention
> of
> > > > > > proposing
> > > > > > > > the
> > > > > > > > >> > > > "Storage
> > > > > > > > >> > > > > > > > > Snapshots" is to give a better way of
> > achiving
> > > > > > > snapshots
> > > > > > > > >> > while
> > > > > > > > >> > > > > still
> > > > > > > > >> > > > > > > > > keeping the original definition of volume
> > > > > snpashots
> > > > > > > (ie
> > > > > > > > >> > upload
> > > > > > > > >> > > to
> > > > > > > > >> > > > > sec
> > > > > > > > >> > > > > > > > > storage).
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > But as Erik pointed out volume snapshots
> are
> > > not
> > > > > > > > backups.
> > > > > > > > >> > They
> > > > > > > > >> > > > > don't
> > > > > > > > >> > > > > > > work
> > > > > > > > >> > > > > > > > > form multi-disk LVM volume groups and
> > dynamic
> > > > > > disks. I
> > > > > > > > am
> > > > > > > > >> all
> > > > > > > > >> > > in
> > > > > > > > >> > > > > for
> > > > > > > > >> > > > > > a
> > > > > > > > >> > > > > > > > > better backup solution which handles these
> > use
> > > > > cases
> > > > > > > and
> > > > > > > > >> > > utilizes
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > > > storage's advanced features.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul
> Angus
> > <
> > > > > > > > >> > > > > > paul.angus@shapeblue.com <javascript:;>
> > > > <javascript:;>>
> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > In the beginning... there were
> CloudStack
> > > > > > snapshots
> > > > > > > > and
> > > > > > > > >> > they
> > > > > > > > >> > > > were
> > > > > > > > >> > > > > > > > > actually
> > > > > > > > >> > > > > > > > > > volume snapshots not hypervisor
> > > point-in-time
> > > > > > > > snapshots.
> > > > > > > > >> > > > > > > > > > Then VM snapshots were created (which
> are
> > > > > > > > point-in-time
> > > > > > > > >> > > > > hypervisor
> > > > > > > > >> > > > > > > > > > snapshots) and we started referring to
> the
> > > > > > original
> > > > > > > > >> > snapshots
> > > > > > > > >> > > > as
> > > > > > > > >> > > > > > > volume
> > > > > > > > >> > > > > > > > > > snapshots.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > CloudStack does not offer 'backups', but
> > > many
> > > > > > people
> > > > > > > > use
> > > > > > > > >> > > volume
> > > > > > > > >> > > > > > > > snapshots
> > > > > > > > >> > > > > > > > > > as backups. However you can't in-place
> > > restore
> > > > > > > volume
> > > > > > > > >> > > snapshots
> > > > > > > > >> > > > > and
> > > > > > > > >> > > > > > > if
> > > > > > > > >> > > > > > > > > you
> > > > > > > > >> > > > > > > > > > have a VM with multiple volumes, the
> > volume
> > > > > > > snapshots
> > > > > > > > >> must
> > > > > > > > >> > be
> > > > > > > > >> > > > > done
> > > > > > > > >> > > > > > in
> > > > > > > > >> > > > > > > > > > series, meaning that the state across
> all
> > of
> > > > the
> > > > > > > > >> volumes is
> > > > > > > > >> > > > > > unlikely
> > > > > > > > >> > > > > > > to
> > > > > > > > >> > > > > > > > > be
> > > > > > > > >> > > > > > > > > > consistent.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > 'Actual Backups' would enable all of the
> > > > restore
> > > > > > > > options
> > > > > > > > >> > > which
> > > > > > > > >> > > > > > users
> > > > > > > > >> > > > > > > > > might
> > > > > > > > >> > > > > > > > > > expect as well options as to where they
> > > might
> > > > be
> > > > > > > > >> stored. In
> > > > > > > > >> > > my
> > > > > > > > >> > > > > > ideal
> > > > > > > > >> > > > > > > > > world
> > > > > > > > >> > > > > > > > > > they would also be able to leverage
> > back-end
> > > > > > > hardware
> > > > > > > > >> (such
> > > > > > > > >> > > as
> > > > > > > > >> > > > > > > > Solidfire,
> > > > > > > > >> > > > > > > > > > NetApp etc :) ) and software such as
> > Veeam,
> > > > > > > Commvault
> > > > > > > > >> etc
> > > > > > > > >> > to
> > > > > > > > >> > > > > > > accelerate
> > > > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > > > > > process.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > [image: ShapeBlue] <
> > > http://www.shapeblue.com>
> > > > > > > > >> > > > > > > > > > Paul Angus
> > > > > > > > >> > > > > > > > > > VP Technology , ShapeBlue
> > > > > > > > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> > 0540*
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > >
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > |
> > > > > > > > >> > > > > m:
> > > > > > > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > >> > > > > > > > > > e: *paul.angus@shapeblue.com
> > <javascript:;>
> > > > > > > > >> > <javascript:;>
> > > > > > > > >> > > |
> > > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > > >> > > > > > > > > > <paul.angus@shapeblue.com
> <javascript:;>
> > > > > > > > >> > > > > <ja...@cloudyangus>
> > > > > > > > >> > > > > > | w:
> > > > > > > > >> > > > > > > > > > *www.shapeblue.com* <
> > > http://www.shapeblue.com
> > > > >
> > > > > > > > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden
> London
> > > WC2N
> > > > > > 4HS
> > > > > > > UK
> > > > > > > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated
> > in
> > > > > > England
> > > > > > > &
> > > > > > > > >> > Wales.
> > > > > > > > >> > > > > > > ShapeBlue
> > > > > > > > >> > > > > > > > > > Services India LLP is a company
> > incorporated
> > > > in
> > > > > > > India
> > > > > > > > >> and
> > > > > > > > >> > is
> > > > > > > > >> > > > > > operated
> > > > > > > > >> > > > > > > > > under
> > > > > > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> > > Brasil
> > > > > > > > >> Consultoria
> > > > > > > > >> > > Ltda
> > > > > > > > >> > > > > is
> > > > > > > > >> > > > > > a
> > > > > > > > >> > > > > > > > > > company incorporated in Brasil and is
> > > operated
> > > > > > under
> > > > > > > > >> > license
> > > > > > > > >> > > > from
> > > > > > > > >> > > > > > > Shape
> > > > > > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a
> > company
> > > > > > > registered
> > > > > > > > >> by
> > > > > > > > >> > The
> > > > > > > > >> > > > > > > Republic
> > > > > > > > >> > > > > > > > of
> > > > > > > > >> > > > > > > > > > South Africa and is traded under license
> > > from
> > > > > > Shape
> > > > > > > > Blue
> > > > > > > > >> > Ltd.
> > > > > > > > >> > > > > > > ShapeBlue
> > > > > > > > >> > > > > > > > > is
> > > > > > > > >> > > > > > > > > > a registered trademark.
> > > > > > > > >> > > > > > > > > > This email and any attachments to it may
> > be
> > > > > > > > confidential
> > > > > > > > >> > and
> > > > > > > > >> > > > are
> > > > > > > > >> > > > > > > > intended
> > > > > > > > >> > > > > > > > > > solely for the use of the individual to
> > whom
> > > > it
> > > > > is
> > > > > > > > >> > addressed.
> > > > > > > > >> > > > Any
> > > > > > > > >> > > > > > > views
> > > > > > > > >> > > > > > > > > or
> > > > > > > > >> > > > > > > > > > opinions expressed are solely those of
> the
> > > > > author
> > > > > > > and
> > > > > > > > do
> > > > > > > > >> > not
> > > > > > > > >> > > > > > > > necessarily
> > > > > > > > >> > > > > > > > > > represent those of Shape Blue Ltd or
> > related
> > > > > > > > companies.
> > > > > > > > >> If
> > > > > > > > >> > > you
> > > > > > > > >> > > > > are
> > > > > > > > >> > > > > > > not
> > > > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > > > > > intended recipient of this email, you
> must
> > > > > neither
> > > > > > > > take
> > > > > > > > >> any
> > > > > > > > >> > > > > action
> > > > > > > > >> > > > > > > > based
> > > > > > > > >> > > > > > > > > > upon its contents, nor copy or show it
> to
> > > > > anyone.
> > > > > > > > Please
> > > > > > > > >> > > > contact
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > > > > sender
> > > > > > > > >> > > > > > > > > > if you believe you have received this
> > email
> > > in
> > > > > > > error.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > -----Original Message-----
> > > > > > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > > syed1.mushtaq@gmail.com
> > > > > > > > >> > > > > <javascript:;> <javascript:;>]
> > > > > > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > > > >> > > > > > > > > > To: dev@cloudstack.apache.org
> > > <javascript:;>
> > > > > > > > >> > <javascript:;>
> > > > > > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature]
> > Storage
> > > > > > > Snapshots
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Paul,
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > When you say actual backups, how would
> it
> > be
> > > > > > > different
> > > > > > > > >> from
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > Volume
> > > > > > > > >> > > > > > > > > > Snapshots that exist currently. My
> > > > understanding
> > > > > > is
> > > > > > > > that
> > > > > > > > >> > > > Backups
> > > > > > > > >> > > > > > end
> > > > > > > > >> > > > > > > up
> > > > > > > > >> > > > > > > > > in
> > > > > > > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > > > > > > point-in-time
> > > > > > > > >> > state
> > > > > > > > >> > > of
> > > > > > > > >> > > > > > your
> > > > > > > > >> > > > > > > > > volume
> > > > > > > > >> > > > > > > > > > which can be restored back correct?
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > -Syed
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul
> > Angus
> > > <
> > > > > > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> > > > > <javascript:;>>
> > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Hi Syed,
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > As I understand it, the SolidFire
> plugin
> > > > will
> > > > > > > export
> > > > > > > > >> the
> > > > > > > > >> > > > > snapshot
> > > > > > > > >> > > > > > > to
> > > > > > > > >> > > > > > > > > > > secondary storage if the user
> requests a
> > > > > > template
> > > > > > > > from
> > > > > > > > >> > the
> > > > > > > > >> > > > > > snapshot
> > > > > > > > >> > > > > > > > or
> > > > > > > > >> > > > > > > > > > > wants to download the snapshot from
> the
> > > > cloud.
> > > > > > > This
> > > > > > > > >> is a
> > > > > > > > >> > > > good,
> > > > > > > > >> > > > > > > > > > > pragmatic approach and yes Mike the
> > > > SolidFire
> > > > > > > > storage
> > > > > > > > >> is
> > > > > > > > >> > > > super
> > > > > > > > >> > > > > > > > > > > reliable and snapshots on SolidFire
> > arrays
> > > > > take
> > > > > > up
> > > > > > > > >> next
> > > > > > > > >> > to
> > > > > > > > >> > > no
> > > > > > > > >> > > > > > > space.
> > > > > > > > >> > > > > > > > > > > BUT I think that we are talking about
> a
> > > more
> > > > > > > general
> > > > > > > > >> > > purpose
> > > > > > > > >> > > > > API,
> > > > > > > > >> > > > > > > and
> > > > > > > > >> > > > > > > > > > > other storage systems may not be as
> > > awesome
> > > > as
> > > > > > > > Mike's.
> > > > > > > > >> > > That's
> > > > > > > > >> > > > > my
> > > > > > > > >> > > > > > > > > > > concern. Also, the time to transfer
> for
> > > say
> > > > > 1TB
> > > > > > to
> > > > > > > > >> move
> > > > > > > > >> > > from
> > > > > > > > >> > > > > > > primary
> > > > > > > > >> > > > > > > > > > > to sec storage and then create a VM
> > > template
> > > > > out
> > > > > > > of
> > > > > > > > it
> > > > > > > > >> > may
> > > > > > > > >> > > be
> > > > > > > > >> > > > > too
> > > > > > > > >> > > > > > > > long
> > > > > > > > >> > > > > > > > > > for users.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term
> > > volume
> > > > > > > > snapshot
> > > > > > > > >> for
> > > > > > > > >> > > > > backup,
> > > > > > > > >> > > > > > > > it's
> > > > > > > > >> > > > > > > > > > > just that users want to do backups
> and a
> > > > > volume
> > > > > > > > >> snapshot
> > > > > > > > >> > is
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > only
> > > > > > > > >> > > > > > > > > > > type of snapshot that copies the disk
> > > > > elsewhere
> > > > > > > and
> > > > > > > > >> can
> > > > > > > > >> > be
> > > > > > > > >> > > > > > > scheduled.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > I'm 'pondering' the implications of
> > > enabling
> > > > > > > actual
> > > > > > > > >> > backups
> > > > > > > > >> > > > > > > (through
> > > > > > > > >> > > > > > > > > > > recognised backup providers) and the
> > user
> > > > > > > > requirements
> > > > > > > > >> > > around
> > > > > > > > >> > > > > > them
> > > > > > > > >> > > > > > > > > > > (particularly restoration use cases)
> as
> > a
> > > > > > separate
> > > > > > > > >> thread
> > > > > > > > >> > > of
> > > > > > > > >> > > > > > work.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > [image: ShapeBlue] <
> > > > http://www.shapeblue.com>
> > > > > > > Paul
> > > > > > > > >> Angus
> > > > > > > > >> > > VP
> > > > > > > > >> > > > > > > > Technology
> > > > > > > > >> > > > > > > > > > > , ShapeBlue
> > > > > > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> > > 0540*
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > >
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > > >> > > > |
> > > > > > > > >> > > > > m:
> > > > > > > > >> > > > > > > > > > > *+44 7711 418784*
> <+44%207711%20418784>
> > > > > > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > <javascript:;>
> > > > > > > > >> > > <javascript:;> |
> > > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > > >> > > > > > > > > > > <paul.angus@shapeblue.com
> > <javascript:;>
> > > > > > > > >> <javascript:;>
> > > > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > >> > > > > > > > > > > *www.shapeblue.com* <
> > > > http://www.shapeblue.com
> > > > > >
> > > > > > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > London
> > > > WC2N
> > > > > > 4HS
> > > > > > > > UK
> > > > > > > > >> > Shape
> > > > > > > > >> > > > > Blue
> > > > > > > > >> > > > > > > Ltd
> > > > > > > > >> > > > > > > > > > > is a company incorporated in England &
> > > > Wales.
> > > > > > > > >> ShapeBlue
> > > > > > > > >> > > > > Services
> > > > > > > > >> > > > > > > > India
> > > > > > > > >> > > > > > > > > > > LLP is a company incorporated in India
> > and
> > > > is
> > > > > > > > operated
> > > > > > > > >> > > under
> > > > > > > > >> > > > > > > license
> > > > > > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> > > > > > Consultoria
> > > > > > > > >> Ltda
> > > > > > > > >> > is
> > > > > > > > >> > > a
> > > > > > > > >> > > > > > > company
> > > > > > > > >> > > > > > > > > > > incorporated in Brasil and is operated
> > > under
> > > > > > > license
> > > > > > > > >> from
> > > > > > > > >> > > > Shape
> > > > > > > > >> > > > > > > Blue
> > > > > > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> > > > > > registered
> > > > > > > by
> > > > > > > > >> The
> > > > > > > > >> > > > > Republic
> > > > > > > > >> > > > > > > of
> > > > > > > > >> > > > > > > > > > > South Africa and is traded under
> license
> > > > from
> > > > > > > Shape
> > > > > > > > >> Blue
> > > > > > > > >> > > Ltd.
> > > > > > > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > > > > > >> > > > > > > > > > > This email and any attachments to it
> may
> > > be
> > > > > > > > >> confidential
> > > > > > > > >> > > and
> > > > > > > > >> > > > > are
> > > > > > > > >> > > > > > > > > > > intended solely for the use of the
> > > > individual
> > > > > to
> > > > > > > > whom
> > > > > > > > >> it
> > > > > > > > >> > is
> > > > > > > > >> > > > > > > > addressed.
> > > > > > > > >> > > > > > > > > > > Any views or opinions expressed are
> > solely
> > > > > those
> > > > > > > of
> > > > > > > > >> the
> > > > > > > > >> > > > author
> > > > > > > > >> > > > > > and
> > > > > > > > >> > > > > > > do
> > > > > > > > >> > > > > > > > > > > not necessarily represent those of
> Shape
> > > > Blue
> > > > > > Ltd
> > > > > > > or
> > > > > > > > >> > > related
> > > > > > > > >> > > > > > > > > > > companies. If you are not the intended
> > > > > recipient
> > > > > > > of
> > > > > > > > >> this
> > > > > > > > >> > > > email,
> > > > > > > > >> > > > > > you
> > > > > > > > >> > > > > > > > > > > must neither take any action based
> upon
> > > its
> > > > > > > > contents,
> > > > > > > > >> nor
> > > > > > > > >> > > > copy
> > > > > > > > >> > > > > or
> > > > > > > > >> > > > > > > > show
> > > > > > > > >> > > > > > > > > > > it to anyone. Please contact the
> sender
> > if
> > > > you
> > > > > > > > believe
> > > > > > > > >> > you
> > > > > > > > >> > > > have
> > > > > > > > >> > > > > > > > > received
> > > > > > > > >> > > > > > > > > > this email in error.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > -----Original Message-----
> > > > > > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > > > syed1.mushtaq@gmail.com
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > <javascript:;>]
> > > > > > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > <javascript:;>
> > > > > > > > >> > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature]
> > > Storage
> > > > > > > > Snapshots
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > I think the terminology confusion
> comes
> > > from
> > > > > AWS
> > > > > > > > where
> > > > > > > > >> > they
> > > > > > > > >> > > > do
> > > > > > > > >> > > > > > EBS
> > > > > > > > >> > > > > > > > > > > snapshots backed up to S3 and
> CloudStack
> > > > sort
> > > > > of
> > > > > > > > >> followed
> > > > > > > > >> > > > that.
> > > > > > > > >> > > > > > And
> > > > > > > > >> > > > > > > > as
> > > > > > > > >> > > > > > > > > > > an end user who is oblivious to the
> > > > internals
> > > > > of
> > > > > > > my
> > > > > > > > >> > > provider,
> > > > > > > > >> > > > > my
> > > > > > > > >> > > > > > > > > > > expectation would be something similar
> > to
> > > > what
> > > > > > AWS
> > > > > > > > as
> > > > > > > > >> > that
> > > > > > > > >> > > is
> > > > > > > > >> > > > > my
> > > > > > > > >> > > > > > > > > > > biggest reference point.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > To your point Mike, I agree that a
> > Primary
> > > > > > Storage
> > > > > > > > >> > failure
> > > > > > > > >> > > on
> > > > > > > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> > > > > > motivations
> > > > > > > > >> for us
> > > > > > > > >> > > to
> > > > > > > > >> > > > > push
> > > > > > > > >> > > > > > > > data
> > > > > > > > >> > > > > > > > > > > to secondary storage. Primary storage
> > > > (atleast
> > > > > > for
> > > > > > > > us)
> > > > > > > > >> > > costs
> > > > > > > > >> > > > > > > around 3
> > > > > > > > >> > > > > > > > > > > times as much as secondary storage and
> > > > > snapshots
> > > > > > > on
> > > > > > > > >> > primary
> > > > > > > > >> > > > > > storage
> > > > > > > > >> > > > > > > > > > > are rarely used (especially for some
> of
> > > our
> > > > > > > > customers
> > > > > > > > >> who
> > > > > > > > >> > > do
> > > > > > > > >> > > > > > daily
> > > > > > > > >> > > > > > > > > > backups).
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike
> > > > > Tutkowski
> > > > > > <
> > > > > > > > >> > > > > > > > > > > mike.tutkowski@solidfire.com
> > > <javascript:;>
> > > > > > > > >> > > <javascript:;>>
> > > > > > > > >> > > > > wrote:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Some of the weirdness is around
> > > > terminology.
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > For most systems I've worked on, a
> > > > snapshot
> > > > > > and
> > > > > > > a
> > > > > > > > >> > backup
> > > > > > > > >> > > > are
> > > > > > > > >> > > > > > two
> > > > > > > > >> > > > > > > > > > > > completely different things (but
> > > > CloudStack
> > > > > > has
> > > > > > > > >> > > > traditionally
> > > > > > > > >> > > > > > > used
> > > > > > > > >> > > > > > > > > > > > the term "volume snapshot" to mean
> > > > backup).
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > I will put in a SolidFire "plug"
> here
> > > and
> > > > > say,
> > > > > > > > >> though,
> > > > > > > > >> > > that
> > > > > > > > >> > > > > if
> > > > > > > > >> > > > > > > your
> > > > > > > > >> > > > > > > > > > > > primary storage is running on
> > SolidFire
> > > > that
> > > > > > it
> > > > > > > is
> > > > > > > > >> > > unlikely
> > > > > > > > >> > > > > > > you'll
> > > > > > > > >> > > > > > > > > > > > encounter an issue where your
> primary
> > > > > storage
> > > > > > > goes
> > > > > > > > >> > > offline
> > > > > > > > >> > > > > (and
> > > > > > > > >> > > > > > > > > > > > you'll even maintain your
> performance
> > > > > > guarantees
> > > > > > > > >> during
> > > > > > > > >> > > > > failure
> > > > > > > > >> > > > > > > > > > > > scenarios and upgrades, as well).
> That
> > > > being
> > > > > > the
> > > > > > > > >> case,
> > > > > > > > >> > it
> > > > > > > > >> > > > is
> > > > > > > > >> > > > > > less
> > > > > > > > >> > > > > > > > > > > > useful to require a backup to Swift
> > (but
> > > > > it's
> > > > > > > > >> perfectly
> > > > > > > > >> > > OK
> > > > > > > > >> > > > if
> > > > > > > > >> > > > > > > > that's
> > > > > > > > >> > > > > > > > > > > > what we want to do
> > > > > > > > >> > > > > > > > > > > here).
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed
> > > > Mushtaq
> > > > > > > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com
> > <javascript:;>
> > > > > > > > >> > <javascript:;>>
> > > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Paul,
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > I believe with the current
> > > > implementation
> > > > > of
> > > > > > > > >> > Snapshots
> > > > > > > > >> > > on
> > > > > > > > >> > > > > > > managed
> > > > > > > > >> > > > > > > > > > > > > storage
> > > > > > > > >> > > > > > > > > > > > > (SolidFire) the snapshots are
> never
> > > > > exported
> > > > > > > to
> > > > > > > > >> the
> > > > > > > > >> > > > > secondary
> > > > > > > > >> > > > > > > > > > storage.
> > > > > > > > >> > > > > > > > > > > > > While this solves the problem of
> > > having
> > > > > > > > snapshots
> > > > > > > > >> > > taking
> > > > > > > > >> > > > > > > forever
> > > > > > > > >> > > > > > > > > > > > > to get to sec storage, this leaves
> > us
> > > > > with a
> > > > > > > > >> > > > > > > > > > > > huge
> > > > > > > > >> > > > > > > > > > > > > liability if our primary storage
> > goes
> > > > > down.
> > > > > > We
> > > > > > > > see
> > > > > > > > >> > > > > snapshots
> > > > > > > > >> > > > > > as
> > > > > > > > >> > > > > > > > > > > > > our recovery path because we store
> > > them
> > > > in
> > > > > > > Swift
> > > > > > > > >> > which
> > > > > > > > >> > > is
> > > > > > > > >> > > > > > > > reliable
> > > > > > > > >> > > > > > > > > > > > > and resilient to failures.
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > With Storage snpashots our goal is
> > to
> > > > have
> > > > > > > > Volume
> > > > > > > > >> > > > snapshots
> > > > > > > > >> > > > > > > > always
> > > > > > > > >> > > > > > > > > > > > > backed up to secondary storage and
> > > > Storage
> > > > > > > > >> Snapshots
> > > > > > > > >> > > stay
> > > > > > > > >> > > > > on
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > primary
> > > > > > > > >> > > > > > > > > > > > storage.
> > > > > > > > >> > > > > > > > > > > > > A provider could potentially mix
> > both
> > > > > these
> > > > > > > and
> > > > > > > > >> solve
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > problem
> > > > > > > > >> > > > > > > > > > > > > that you mentioned where you want
> to
> > > > meet
> > > > > > > user's
> > > > > > > > >> > > > > expectation
> > > > > > > > >> > > > > > > of a
> > > > > > > > >> > > > > > > > > > > > > snapshot (ie backup to sec
> storage)
> > > > while
> > > > > > > having
> > > > > > > > >> an
> > > > > > > > >> > > > ability
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on
> > the
> > > > > > device)
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > Thanks,
> > > > > > > > >> > > > > > > > > > > > > -Syed
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM,
> Paul
> > > > Angus
> > > > > > > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com
> > > > <javascript:;>
> > > > > > > > >> > > <javascript:;>>
> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > HI guys,
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Could someone point me to the
> Jira
> > > bug
> > > > > of
> > > > > > FS
> > > > > > > > for
> > > > > > > > >> > the
> > > > > > > > >> > > > > > > > > > > > > > SAN-snapshot
> > > > > > > > >> > > > > > > > > > > > feature
> > > > > > > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > From my discussions with users
> and
> > > > > > operators
> > > > > > > > >> around
> > > > > > > > >> > > > > > snapshots
> > > > > > > > >> > > > > > > > > > > > > > I'd make
> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > following observations:
> > > > > > > > >> > > > > > > > > > > > > > a. 'users' use snapshots as
> > backups
> > > > > (both
> > > > > > > > >> long-term
> > > > > > > > >> > > and
> > > > > > > > >> > > > > > short
> > > > > > > > >> > > > > > > > > > > > > > term)
> > > > > > > > >> > > > > > > > > > > > with
> > > > > > > > >> > > > > > > > > > > > > > the expectation that they can
> use
> > > them
> > > > > for
> > > > > > > > >> recovery
> > > > > > > > >> > > if
> > > > > > > > >> > > > > > > > required.
> > > > > > > > >> > > > > > > > > > > > > > b. operators fall back to
> > snapshots
> > > if
> > > > > > > > something
> > > > > > > > >> > has
> > > > > > > > >> > > > gone
> > > > > > > > >> > > > > > > wrong
> > > > > > > > >> > > > > > > > > > > > > > with primary storage.
> > > > > > > > >> > > > > > > > > > > > > > c. users sometimes want to be
> able
> > > to
> > > > > > export
> > > > > > > > >> > > snapshots
> > > > > > > > >> > > > as
> > > > > > > > >> > > > > > > well
> > > > > > > > >> > > > > > > > > > > > > > as
> > > > > > > > >> > > > > > > > > > > > create
> > > > > > > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > > > > > > >> > > > > > > > > > > > > > d. snapshots are a currently a
> > > massive
> > > > > > pain
> > > > > > > > for
> > > > > > > > >> > > > > operators,
> > > > > > > > >> > > > > > I
> > > > > > > > >> > > > > > > > > > > > > > know at
> > > > > > > > >> > > > > > > > > > > > > least
> > > > > > > > >> > > > > > > > > > > > > > one public cloud who have
> > snapshots
> > > > > which
> > > > > > > > take 2
> > > > > > > > >> > days
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > > > > complete.
> > > > > > > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't
> > be
> > > > used
> > > > > > for
> > > > > > > > >> > multiple
> > > > > > > > >> > > > LVM
> > > > > > > > >> > > > > > > > disks.
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > I think the process Mike has
> used
> > in
> > > > the
> > > > > > > > >> SolidFire
> > > > > > > > >> > > > plugin
> > > > > > > > >> > > > > > > (only
> > > > > > > > >> > > > > > > > > > > > > > moving
> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > disk image to secondary storage
> > when
> > > > you
> > > > > > > > >> absolutely
> > > > > > > > >> > > > have
> > > > > > > > >> > > > > > to)
> > > > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > a very
> > > > > > > > >> > > > > > > > > > > > > good
> > > > > > > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder
> > > what
> > > > > > > problems
> > > > > > > > >> an
> > > > > > > > >> > > > > operator
> > > > > > > > >> > > > > > > > might
> > > > > > > > >> > > > > > > > > > > > > experience
> > > > > > > > >> > > > > > > > > > > > > > if they have an issue with a
> given
> > > > > primary
> > > > > > > > >> storage
> > > > > > > > >> > > pool
> > > > > > > > >> > > > > in
> > > > > > > > >> > > > > > a
> > > > > > > > >> > > > > > > > > > cluster.
> > > > > > > > >> > > > > > > > > > > > (I
> > > > > > > > >> > > > > > > > > > > > > > know that that is REALLY
> unlikely
> > in
> > > > the
> > > > > > > > >> SolidFire
> > > > > > > > >> > > case
> > > > > > > > >> > > > > > Mike
> > > > > > > > >> > > > > > > :)
> > > > > > > > >> > > > > > > > > > > > > > ) And
> > > > > > > > >> > > > > > > > > > > > if
> > > > > > > > >> > > > > > > > > > > > > > the transfer from primary to
> > > secondary
> > > > > is
> > > > > > > > slow,
> > > > > > > > >> the
> > > > > > > > >> > > > time
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > > > > > > > > being able
> > > > > > > > >> > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > create a template or export the
> > > volume
> > > > > > will
> > > > > > > be
> > > > > > > > >> > slow.
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > So for me the issue is around
> > making
> > > > > sure
> > > > > > > that
> > > > > > > > >> the
> > > > > > > > >> > > end
> > > > > > > > >> > > > > > users
> > > > > > > > >> > > > > > > > > > > > expectations
> > > > > > > > >> > > > > > > > > > > > > > are met (while improving the
> > > > > > > speed/efficiency
> > > > > > > > of
> > > > > > > > >> > the
> > > > > > > > >> > > > back
> > > > > > > > >> > > > > > > end)
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > > > > > > http://www.shapeblue.com>
> > > > > > > > >> Paul
> > > > > > > > >> > > > Angus
> > > > > > > > >> > > > > > VP
> > > > > > > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > > > > > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44
> 203
> > > 603
> > > > > > 0540*
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > >
> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > > >> > > > > > > |
> > > > > > > > >> > > > > > > > m:
> > > > > > > > >> > > > > > > > > > > > > > *+44 7711 418784*
> > > > <+44%207711%20418784>
> > > > > > > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;> | t:
> > > > > > > > >> > > > > > @cloudyangus*
> > > > > > > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com
> > > > > <javascript:;>
> > > > > > > > >> > > > <javascript:;>
> > > > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > > > > > > http://www.shapeblue.com
> > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent
> Garden
> > > > > London
> > > > > > > WC2N
> > > > > > > > >> 4HS
> > > > > > > > >> > UK
> > > > > > > > >> > > > > Shape
> > > > > > > > >> > > > > > > > Blue
> > > > > > > > >> > > > > > > > > > > > > > Ltd is a company incorporated in
> > > > > England &
> > > > > > > > >> Wales.
> > > > > > > > >> > > > > ShapeBlue
> > > > > > > > >> > > > > > > > > > > > > > Services India LLP is a company
> > > > > > incorporated
> > > > > > > > in
> > > > > > > > >> > India
> > > > > > > > >> > > > and
> > > > > > > > >> > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > operated
> > > > > > > > >> > > > > > > > > > > > > under
> > > > > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd.
> Shape
> > > > Blue
> > > > > > > Brasil
> > > > > > > > >> > > > > Consultoria
> > > > > > > > >> > > > > > > Ltda
> > > > > > > > >> > > > > > > > > > > > > > is a company incorporated in
> > Brasil
> > > > and
> > > > > is
> > > > > > > > >> operated
> > > > > > > > >> > > > under
> > > > > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd.
> > > ShapeBlue
> > > > > SA
> > > > > > > Pty
> > > > > > > > >> Ltd
> > > > > > > > >> > is
> > > > > > > > >> > > a
> > > > > > > > >> > > > > > > company
> > > > > > > > >> > > > > > > > > > > > > > registered by The Republic
> > > > > > > > >> > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > > > South Africa and is traded under
> > > > license
> > > > > > > from
> > > > > > > > >> Shape
> > > > > > > > >> > > > Blue
> > > > > > > > >> > > > > > Ltd.
> > > > > > > > >> > > > > > > > > > > > > > ShapeBlue
> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > a registered trademark.
> > > > > > > > >> > > > > > > > > > > > > > This email and any attachments
> to
> > it
> > > > may
> > > > > > be
> > > > > > > > >> > > > confidential
> > > > > > > > >> > > > > > and
> > > > > > > > >> > > > > > > > are
> > > > > > > > >> > > > > > > > > > > > intended
> > > > > > > > >> > > > > > > > > > > > > > solely for the use of the
> > individual
> > > > to
> > > > > > whom
> > > > > > > > it
> > > > > > > > >> is
> > > > > > > > >> > > > > > addressed.
> > > > > > > > >> > > > > > > > > > > > > > Any views
> > > > > > > > >> > > > > > > > > > > > > or
> > > > > > > > >> > > > > > > > > > > > > > opinions expressed are solely
> > those
> > > of
> > > > > the
> > > > > > > > >> author
> > > > > > > > >> > and
> > > > > > > > >> > > > do
> > > > > > > > >> > > > > > not
> > > > > > > > >> > > > > > > > > > > > necessarily
> > > > > > > > >> > > > > > > > > > > > > > represent those of Shape Blue
> Ltd
> > or
> > > > > > related
> > > > > > > > >> > > companies.
> > > > > > > > >> > > > > If
> > > > > > > > >> > > > > > > you
> > > > > > > > >> > > > > > > > > > > > > > are not
> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > intended recipient of this
> email,
> > > you
> > > > > must
> > > > > > > > >> neither
> > > > > > > > >> > > take
> > > > > > > > >> > > > > any
> > > > > > > > >> > > > > > > > > > > > > > action
> > > > > > > > >> > > > > > > > > > > > based
> > > > > > > > >> > > > > > > > > > > > > > upon its contents, nor copy or
> > show
> > > it
> > > > > to
> > > > > > > > >> anyone.
> > > > > > > > >> > > > Please
> > > > > > > > >> > > > > > > > contact
> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > sender
> > > > > > > > >> > > > > > > > > > > > > > if you believe you have received
> > > this
> > > > > > email
> > > > > > > in
> > > > > > > > >> > error.
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > -----Original Message-----
> > > > > > > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > > > > > > >> pdion@cloudops.com
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > <javascript:;>]
> > > > > > > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016
> > 12:56
> > > > PM
> > > > > > > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New
> > Feature]
> > > > > > Storage
> > > > > > > > >> > Snapshots
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Hi Mike,
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > The idea of introducing a new
> API:
> > > > > > > > >> StorageSnapshot
> > > > > > > > >> > > for
> > > > > > > > >> > > > > > > managed
> > > > > > > > >> > > > > > > > > > > > > > storage
> > > > > > > > >> > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > because the VolumeSnapshot
> > default,
> > > or
> > > > > > > > expected,
> > > > > > > > >> > > > behavior
> > > > > > > > >> > > > > > is
> > > > > > > > >> > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > archive snapshots into the
> > Secondary
> > > > > > > Storage.
> > > > > > > > >> So a
> > > > > > > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > >> > > > > > > > > > > > > > snapshot that remain on the
> > managed
> > > > > > storage
> > > > > > > > >> > > appliance.
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Quickly looking at the API doc
> > and I
> > > > > don't
> > > > > > > > see a
> > > > > > > > >> > > strong
> > > > > > > > >> > > > > > > > > > > > > > requirement for volume snapshots
> > to
> > > be
> > > > > > moved
> > > > > > > > >> into
> > > > > > > > >> > > > > secondary
> > > > > > > > >> > > > > > > > > > > > > > storage. So, maybe
> StorageSnapshot
> > > API
> > > > > is
> > > > > > > not
> > > > > > > > >> > useful,
> > > > > > > > >> > > > but
> > > > > > > > >> > > > > > > both
> > > > > > > > >> > > > > > > > > > > > > > use cases are required. A
> snapshot
> > > > that
> > > > > > > remain
> > > > > > > > >> on
> > > > > > > > >> > the
> > > > > > > > >> > > > > > managed
> > > > > > > > >> > > > > > > > > > > > > > storage, and another type of
> > > > > > > > >> > > > > > > > > > > > snapshot
> > > > > > > > >> > > > > > > > > > > > > > that end up into the secondary
> > > > storage.
> > > > > > > Since
> > > > > > > > >> > you've
> > > > > > > > >> > > > > done a
> > > > > > > > >> > > > > > > lot
> > > > > > > > >> > > > > > > > > > > > > > of
> > > > > > > > >> > > > > > > > > > > > work,
> > > > > > > > >> > > > > > > > > > > > > > might easier to just add a
> > parameter
> > > > to
> > > > > > the
> > > > > > > > >> current
> > > > > > > > >> > > > > > snapshot
> > > > > > > > >> > > > > > > > API
> > > > > > > > >> > > > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > would
> > > > > > > > >> > > > > > > > > > > > > > trigger an extraction of the
> > storage
> > > > > > > snapshot
> > > > > > > > >> into
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > > secondary
> > > > > > > > >> > > > > > > > > > > > storage?
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > PL
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM,
> > Mike
> > > > > > > > Tutkowski <
> > > > > > > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> > > > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > I think that all sounds
> > reasonable
> > > > > then
> > > > > > -
> > > > > > > > >> thanks!
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52
> PM,
> > > Syed
> > > > > > > > Mushtaq <
> > > > > > > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com
> > <javascript:;>
> > > > > > > > >> <javascript:;>>
> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >> You are correct Mike in terms
> > of
> > > > the
> > > > > > > > >> > requirements.
> > > > > > > > >> > > > One
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > > our
> > > > > > > > >> > > > > > > > > > > > earlier
> > > > > > > > >> > > > > > > > > > > > > > >> iterations on this was to
> have
> > an
> > > > > > > argument
> > > > > > > > to
> > > > > > > > >> > the
> > > > > > > > >> > > > > create
> > > > > > > > >> > > > > > > > > > > > > > >> snapshot
> > > > > > > > >> > > > > > > > > > > > API
> > > > > > > > >> > > > > > > > > > > > > > >> which decides whether to
> backup
> > > the
> > > > > > > volume
> > > > > > > > to
> > > > > > > > >> > sec
> > > > > > > > >> > > > > > storage
> > > > > > > > >> > > > > > > > but
> > > > > > > > >> > > > > > > > > > > > > > >> we realized it would make
> > > > management
> > > > > of
> > > > > > > > >> > snapshots
> > > > > > > > >> > > > > quite
> > > > > > > > >> > > > > > > > messy
> > > > > > > > >> > > > > > > > > > > > > > >> so we proposed a new api
> > instead.
> > > > > > > > >> > > > > > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM
> > Mike
> > > > > > > Tutkowski
> > > > > > > > >> > > > > > > > > > > > > > >> <
> mike.tutkowski@solidfire.com
> > > > > > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>>
> > > > > > > > >> > > > > > > > > > > > > > >> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > > > > > >>> Hi,
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> Just to make sure I
> understand
> > > all
> > > > > the
> > > > > > > > >> > > requirements
> > > > > > > > >> > > > > > here:
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> 1) This relates only to
> > managed
> > > > > > storage
> > > > > > > > (1:1
> > > > > > > > >> > > > mapping
> > > > > > > > >> > > > > > > > between
> > > > > > > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend
> > SAN
> > > > > > > volume).
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> 2) We want to take the
> current
> > > > > > > (introduced
> > > > > > > > >> in
> > > > > > > > >> > > 4.6)
> > > > > > > > >> > > > > > > > > > > > > > >>> functionality, which
> creates a
> > > > > > snapshot
> > > > > > > on
> > > > > > > > >> the
> > > > > > > > >> > > SAN,
> > > > > > > > >> > > > > and
> > > > > > > > >> > > > > > > > > > > > > > >>> extend it via a config
> option
> > > (or
> > > > > > > > >> > > > > > > > > > > > > > >>> something) to not only take
> > the
> > > > SAN
> > > > > > > > >> snapshot,
> > > > > > > > >> > but
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > copy
> > > > > > > > >> > > > > > > > > > > > > > >>> the underlying VHD
> (XenServer
> > > > only)
> > > > > to
> > > > > > > > NFS.
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is
> always
> > > > taken.
> > > > > > > It's
> > > > > > > > >> the
> > > > > > > > >> > > > backup
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > NFS
> > > > > > > > >> > > > > > > > > > > > > > >>> that is optional.
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> 4) Templates can be created
> > from
> > > > the
> > > > > > > > >> snapshot
> > > > > > > > >> > > > that's
> > > > > > > > >> > > > > on
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be
> > > > created
> > > > > > > from
> > > > > > > > >> the
> > > > > > > > >> > > > > snapshot
> > > > > > > > >> > > > > > > > > > > > > > >>> that's on
> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > >>> SAN (already works as long
> as
> > > the
> > > > > new
> > > > > > > > >> > CloudStack
> > > > > > > > >> > > > > volume
> > > > > > > > >> > > > > > > > ends
> > > > > > > > >> > > > > > > > > > > > > > >>> up on the same primary
> > storage).
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> Would we have a need for a
> > > storage
> > > > > > > > snapshot
> > > > > > > > >> API
> > > > > > > > >> > > > then
> > > > > > > > >> > > > > or
> > > > > > > > >> > > > > > > > > > > > > > >>> would that just be the
> > standard
> > > > > volume
> > > > > > > > >> snapshot
> > > > > > > > >> > > > > without
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > >>> backup to
> > > > > > > > >> > > > > > > > > > > NFS?
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> Thanks!
> > > > > > > > >> > > > > > > > > > > > > > >>> Mike
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24
> > PM,
> > > > Syed
> > > > > > > > Mushtaq
> > > > > > > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> > > > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>>
> > > > > > > > >> > > > > > > > > > > > > > >>> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > > > > > > functionalities
> > > > > > > > >> > > > > (snapshot
> > > > > > > > >> > > > > > on
> > > > > > > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > > > > > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because
> > > > Ideally,
> > > > > we
> > > > > > > > would
> > > > > > > > >> > like
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > have
> > > > > > > > >> > > > > > > > > both.
> > > > > > > > >> > > > > > > > > > > > > > >>>> For example, some of our
> > > > customers
> > > > > > want
> > > > > > > > to
> > > > > > > > >> > > > implement
> > > > > > > > >> > > > > > > their
> > > > > > > > >> > > > > > > > > > > > > > >>>> own backup strategies and
> do
> > > > > > encryption
> > > > > > > > to
> > > > > > > > >> > their
> > > > > > > > >> > > > > > backups
> > > > > > > > >> > > > > > > > > > > > > > >>>> which is a perfect use case
> > for
> > > > > > Storage
> > > > > > > > >> > Snapshot
> > > > > > > > >> > > > > while
> > > > > > > > >> > > > > > > our
> > > > > > > > >> > > > > > > > > > > > > > >>>> other customers will still
> > keep
> > > > > using
> > > > > > > the
> > > > > > > > >> > > standard
> > > > > > > > >> > > > > > > volume
> > > > > > > > >> > > > > > > > > > > snapshot.
> > > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>> To keep things backward
> > > > compatible,
> > > > > > we
> > > > > > > > can
> > > > > > > > >> > add a
> > > > > > > > >> > > > > > setting
> > > > > > > > >> > > > > > > > > > > > > > >>>> which
> > > > > > > > >> > > > > > > > > > > > says
> > > > > > > > >> > > > > > > > > > > > > > >>>> to not upload on secondary
> > > > storage,
> > > > > > > > >> because,
> > > > > > > > >> > > after
> > > > > > > > >> > > > > > all,
> > > > > > > > >> > > > > > > > you
> > > > > > > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot
> > first
> > > > > when
> > > > > > > > doing
> > > > > > > > >> a
> > > > > > > > >> > > > Volume
> > > > > > > > >> > > > > > > > > Snapshot.
> > > > > > > > >> > > > > > > > > > > > > > >>>> You could stop the process
> > > there
> > > > > and
> > > > > > > not
> > > > > > > > do
> > > > > > > > >> > the
> > > > > > > > >> > > > > > upload.
> > > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>> What do you think about
> this
> > > > > > approach?
> > > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > > > > > > >> > > > > > > > > > > > > > >>>> -Syed
> > > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25
> > PM,
> > > > > Mike
> > > > > > > > >> > Tutkowski <
> > > > > > > > >> > > > > > > > > > > > > > >>>>
> mike.tutkowski@solidfire.com
> > > > > > > > >> <javascript:;>
> > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>> So, this is just me
> thinking
> > > out
> > > > > > load
> > > > > > > > >> here,
> > > > > > > > >> > but
> > > > > > > > >> > > > if
> > > > > > > > >> > > > > a
> > > > > > > > >> > > > > > > > given
> > > > > > > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't
> > > > actually
> > > > > > need
> > > > > > > > to
> > > > > > > > >> > > provide
> > > > > > > > >> > > > > > both
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > ability
> > > > > > > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and
> > > > export
> > > > > it
> > > > > > > to
> > > > > > > > >> NFS
> > > > > > > > >> > (if
> > > > > > > > >> > > > > just
> > > > > > > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is
> > OK),
> > > > then
> > > > > > we
> > > > > > > > >> might
> > > > > > > > >> > be
> > > > > > > > >> > > > able
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > get
> > > > > > > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > > > > > > >> > > > > > > > > > > > API
> > > > > > > > >> > > > > > > > > > > > > > >>>>> calls and simply
> implement a
> > > new
> > > > > > > custom
> > > > > > > > >> > > snapshot
> > > > > > > > >> > > > > > > strategy
> > > > > > > > >> > > > > > > > > > > > > > >>>>> and
> > > > > > > > >> > > > > > > > > > > > data
> > > > > > > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle
> > the
> > > > case
> > > > > > > where
> > > > > > > > >> the
> > > > > > > > >> > > > > > CloudStack
> > > > > > > > >> > > > > > > > > > > > > > >>>>> cloud
> > > > > > > > >> > > > > > > > > > > > does
> > > > > > > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot
> and
> > > > > > > > >> exported-to-NFS
> > > > > > > > >> > > > > backup.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>> In other words, the
> > "default"
> > > > > > behavior
> > > > > > > > >> would
> > > > > > > > >> > be
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > use
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data
> > > > motion
> > > > > > > > strategy
> > > > > > > > >> > that
> > > > > > > > >> > > > we
> > > > > > > > >> > > > > > > > already
> > > > > > > > >> > > > > > > > > > > > > > >>>>> have (the one that only
> > takes
> > > a
> > > > > SAN
> > > > > > > > >> > snapshot).
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud,
> > > > however,
> > > > > > > wants
> > > > > > > > >> to
> > > > > > > > >> > > take
> > > > > > > > >> > > > a
> > > > > > > > >> > > > > > SAN
> > > > > > > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data
> > > > > exported
> > > > > > to
> > > > > > > > >> NFS,
> > > > > > > > >> > > then
> > > > > > > > >> > > > we
> > > > > > > > >> > > > > > > could
> > > > > > > > >> > > > > > > > > > > > > > >>>>> have you manipulate a
> Swing
> > > > config
> > > > > > > file
> > > > > > > > to
> > > > > > > > >> > make
> > > > > > > > >> > > > use
> > > > > > > > >> > > > > > of
> > > > > > > > >> > > > > > > a
> > > > > > > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and
> > data
> > > > > > motion
> > > > > > > > >> > strategy
> > > > > > > > >> > > > that
> > > > > > > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > > > > > > >> > > > > > > > > > > > > > activities.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior
> > is
> > > > > still
> > > > > > > the
> > > > > > > > >> > default
> > > > > > > > >> > > > for
> > > > > > > > >> > > > > > > > users,
> > > > > > > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can
> > > change
> > > > > > this
> > > > > > > > >> > behavior
> > > > > > > > >> > > > via
> > > > > > > > >> > > > > > > > > > > configuration.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at
> 11:55
> > > AM,
> > > > > > Mike
> > > > > > > > >> > > Tutkowski <
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > mike.tutkowski@solidfire.com
> > > > > > > > >> <javascript:;>
> > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will
> > need
> > > to
> > > > > > come
> > > > > > > up
> > > > > > > > >> > with a
> > > > > > > > >> > > > > > viable
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some
> > > reasonable
> > > > > way
> > > > > > > for
> > > > > > > > >> them
> > > > > > > > >> > > to
> > > > > > > > >> > > > > move
> > > > > > > > >> > > > > > > > from
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> the old way to the new
> way
> > > (and
> > > > > > some
> > > > > > > > >> obvious
> > > > > > > > >> > > way
> > > > > > > > >> > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > > > > > > >> > > > > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > > > > do this).
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at
> > 11:45
> > > > AM,
> > > > > > Syed
> > > > > > > > >> > Mushtaq
> > > > > > > > >> > > <
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > > > > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure
> about
> > > the
> > > > > > > upgrade
> > > > > > > > >> path
> > > > > > > > >> > > > > however,
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>> customers who are using
> > 4.6
> > > > and
> > > > > > are
> > > > > > > > on a
> > > > > > > > >> > > > managed
> > > > > > > > >> > > > > > > > storage
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the
> > > same
> > > > > > > > >> functionality
> > > > > > > > >> > > > with
> > > > > > > > >> > > > > > > Volume
> > > > > > > > >> > > > > > > > > > > Snapshots.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at
> > 1:43
> > > > PM,
> > > > > > Syed
> > > > > > > > >> > Mushtaq
> > > > > > > > >> > > <
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > > > > > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> So if I understand
> > > correctly,
> > > > > > > > currently
> > > > > > > > >> > > > taking a
> > > > > > > > >> > > > > > > > Volume
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume
> on
> > a
> > > > > > managed
> > > > > > > > >> storage
> > > > > > > > >> > > > keeps
> > > > > > > > >> > > > > > it
> > > > > > > > >> > > > > > > on
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a
> > > part
> > > > of
> > > > > > > this
> > > > > > > > >> > > feature,
> > > > > > > > >> > > > we
> > > > > > > > >> > > > > > can
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > > > > > > > >> > > > > > > > > > > > that
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on
> > managed
> > > > > > storage
> > > > > > > > are
> > > > > > > > >> > > > uploaded
> > > > > > > > >> > > > > > to
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This
> > > would
> > > > > > make
> > > > > > > > the
> > > > > > > > >> > > Volume
> > > > > > > > >> > > > > > > Snapshot
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same
> > > > > > regardless
> > > > > > > of
> > > > > > > > >> the
> > > > > > > > >> > > > > storage
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for
> > > > utilizing
> > > > > > the
> > > > > > > > >> > > efficient
> > > > > > > > >> > > > > > > backend
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> storage
> > > > > > > > >> > > > > > > > > > > > > > capabilities, we can use the new
> > > > storage
> > > > > > > > >> snapshots
> > > > > > > > >> > > API.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at
> > 1:36
> > > > PM,
> > > > > > > Mike
> > > > > > > > >> > > > Tutkowski <
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > mike.tutkowski@solidfire.com
> > > > > > > > >> > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>>
> > > > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here,
> we
> > > need
> > > > > to
> > > > > > > > have a
> > > > > > > > >> > plan
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > deal
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we
> > > > already
> > > > > > > have a
> > > > > > > > >> > > feature
> > > > > > > > >> > > > > (in
> > > > > > > > >> > > > > > > 4.6
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> and
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you
> > to
> > > > use
> > > > > > the
> > > > > > > > >> > existing
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs
> to
> > > > > create a
> > > > > > > > >> volume
> > > > > > > > >> > > > > snapshot
> > > > > > > > >> > > > > > > (for
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides
> > on a
> > > > > > backend
> > > > > > > > SAN
> > > > > > > > >> > > > (using a
> > > > > > > > >> > > > > > > > custom
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and
> a
> > > > custom
> > > > > > > data
> > > > > > > > >> > motion
> > > > > > > > >> > > > > > > strategy).
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go
> in,
> > > > then
> > > > > > how
> > > > > > > > >> should
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > original
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> implementation
> (present
> > in
> > > > 4.6
> > > > > > and
> > > > > > > > >> later)
> > > > > > > > >> > > be
> > > > > > > > >> > > > > > > changed?
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > > > > > > > >> > > > > > > > > > > > is
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we
> > support
> > > > > > > customers
> > > > > > > > >> who
> > > > > > > > >> > > are
> > > > > > > > >> > > > > > > already
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> using
> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> original
> volume-snapshot
> > > API
> > > > > to
> > > > > > > take
> > > > > > > > >> > > > snapshots
> > > > > > > > >> > > > > > on a
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > > > > > > > >> > > > > > > > > > > > > SAN?
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at
> > > 11:27
> > > > > AM,
> > > > > > > > Will
> > > > > > > > >> > > > Stevens <
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > > > > > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to
> > > create
> > > > a
> > > > > > > > Template
> > > > > > > > >> > > from a
> > > > > > > > >> > > > > > > > > > > > StorageSnapshot?
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the
> > template
> > > > be
> > > > > > > stored
> > > > > > > > >> in
> > > > > > > > >> > the
> > > > > > > > >> > > > > > > secondary
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal
> > > > templates
> > > > > > or
> > > > > > > > will
> > > > > > > > >> > that
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > > > > handled
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > > >> > > > > > > > > > > > > > vendor side?
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud
> > > > > Solutions
> > > > > > > > >> Experts
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|*
> > Montreal
> > > > *|*
> > > > > > > Quebec
> > > > > > > > >> *|*
> > > > > > > > >> > > H3J
> > > > > > > > >> > > > > 1S6
> > > > > > > > >> > > > > > w
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw
> > > > > @CloudOps_
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016
> at
> > > 1:22
> > > > > PM,
> > > > > > > > Syed
> > > > > > > > >> > > > Mushtaq <
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > syed1.mushtaq@gmail.com
> > > > > > > > >> <javascript:;>
> > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016
> at
> > > > 1:19
> > > > > > PM,
> > > > > > > > Will
> > > > > > > > >> > > > Stevens
> > > > > > > > >> > > > > <
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> wstevens@cloudops.com
> > > > > > > > >> <javascript:;>
> > > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked
> > the
> > > > > > Design
> > > > > > > > >> Spec in
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > Jira
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it
> was
> > > not
> > > > > > clear
> > > > > > > > in
> > > > > > > > >> the
> > > > > > > > >> > > > > > 'mention'
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it
> > > shows
> > > > > as a
> > > > > > > > page
> > > > > > > > >> > 'you
> > > > > > > > >> > > do
> > > > > > > > >> > > > > not
> > > > > > > > >> > > > > > > > have
> > > > > > > > >> > > > > > > > > > > permission to'.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *|
> *Cloud
> > > > > > Solutions
> > > > > > > > >> Experts
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|*
> > > Montreal
> > > > > *|*
> > > > > > > > Quebec
> > > > > > > > >> > *|*
> > > > > > > > >> > > > H3J
> > > > > > > > >> > > > > > 1S6
> > > > > > > > >> > > > > > > w
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|*
> tw
> > > > > > @CloudOps_
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016
> > at
> > > > 1:02
> > > > > > PM,
> > > > > > > > >> Syed
> > > > > > > > >> > > Ahmed
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> <
> sahmed@cloudops.com
> > > > > > > > >> <javascript:;>
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > >
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > >
> > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to
> propose a
> > > new
> > > > > set
> > > > > > > of
> > > > > > > > >> APIs
> > > > > > > > >> > to
> > > > > > > > >> > > > do
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on
> managed
> > > > > storage
> > > > > > > > >> backends
> > > > > > > > >> > > > like
> > > > > > > > >> > > > > > > > > SolidFire.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on
> current
> > > > > managed
> > > > > > > > >> storage
> > > > > > > > >> > > stay
> > > > > > > > >> > > > > on
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is
> > > contrary
> > > > > to
> > > > > > > what
> > > > > > > > >> > > > CloudStack
> > > > > > > > >> > > > > > > calls
> > > > > > > > >> > > > > > > > > > > snpshots.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking
> snapshots
> > > on
> > > > > > > storage
> > > > > > > > >> and
> > > > > > > > >> > > > keeping
> > > > > > > > >> > > > > > it
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own
> > > > > advantages
> > > > > > > > >> > > > > > > > > > > > and
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally
> > like
> > > to
> > > > > > have
> > > > > > > > both
> > > > > > > > >> > ways
> > > > > > > > >> > > > of
> > > > > > > > >> > > > > > > doing
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This
> > > proposal
> > > > > > adds
> > > > > > > 4
> > > > > > > > >> new
> > > > > > > > >> > > APIs
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > > > > create
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on
> backend
> > > > > > storage.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys
> > think
> > > > of
> > > > > > this
> > > > > > > > >> > > feature? I
> > > > > > > > >> > > > > > would
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some
> > > > > feedback.
> > > > > > I
> > > > > > > am
> > > > > > > > >> > > working
> > > > > > > > >> > > > on
> > > > > > > > >> > > > > > > > making
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > > > >> > > > > > > > > > > > spec
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but
> > > wanted
> > > > > to
> > > > > > > > have a
> > > > > > > > >> > high
> > > > > > > > >> > > > > level
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first
> > before
> > > > > > starting
> > > > > > > > to
> > > > > > > > >> > work
> > > > > > > > >> > > on
> > > > > > > > >> > > > > it.
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> --
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack
> > > > Developer,
> > > > > > > > >> SolidFire
> > > > > > > > >> > > Inc.*
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> e:
> > > > > mike.tutkowski@solidfire.com
> > > > > > > > >> > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the
> > > world
> > > > > uses
> > > > > > > the
> > > > > > > > >> > cloud
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> <
> > > > > > > > >> > > > > > >
> > > http://solidfire.com/solution/overview/?video=play
> > > > > > > > >> > > > > > > > >*™
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> --
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack
> > Developer,
> > > > > > > SolidFire
> > > > > > > > >> > Inc.*
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> e:
> > > > mike.tutkowski@solidfire.com
> > > > > > > > >> > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> Advancing the way the
> world
> > > > uses
> > > > > > the
> > > > > > > > >> cloud
> > > > > > > > >> > > > > > > > > > > > > > >>>>>> <
> > > > > > > > >> > > > >
> http://solidfire.com/solution/overview/?video=play
> > > > > > > > >> > > > > > > >*™*
> > > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>> --
> > > > > > > > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack
> > Developer,
> > > > > > > SolidFire
> > > > > > > > >> Inc.*
> > > > > > > > >> > > > > > > > > > > > > > >>>>> e:
> > > mike.tutkowski@solidfire.com
> > > > > > > > >> > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > > > > >> > > > > > > > > > > > > > >>>>> Advancing the way the
> world
> > > uses
> > > > > the
> > > > > > > > cloud
> > > > > > > > >> > > > > > > > > > > > > > >>>>> <
> > > > > > > > >> > > > >
> http://solidfire.com/solution/overview/?video=play
> > > > > > > > >> > > > > > > >*™*
> > > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>> --
> > > > > > > > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > > > > >> > > > > > > > > > > > > > >>> *Senior CloudStack
> Developer,
> > > > > > SolidFire
> > > > > > > > >> Inc.*
> > > > > > > > >> > > > > > > > > > > > > > >>> e:
> > mike.tutkowski@solidfire.com
> > > > > > > > >> <javascript:;>
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > > > > > > >> > > > > > > > > > > > > > >>> Advancing the way the world
> > uses
> > > > the
> > > > > > > cloud
> > > > > > > > >> > > > > > > > > > > > > > >>> <
> > > > > > > > >> > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > > >> > > > > > >*™*
> > > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > > >> > > > > > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > > --
> > > > > > > > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > > > > > > >> > > > > > > > > > > > > > > *Senior CloudStack Developer,
> > > > > SolidFire
> > > > > > > > Inc.*
> > > > > > > > >> > > > > > > > > > > > > > > e:
> mike.tutkowski@solidfire.com
> > > > > > > > >> <javascript:;>
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > > > > > o: 303.746.7302
> > > > > > > > >> > > > > > > > > > > > > > > Advancing the way the world
> uses
> > > the
> > > > > > cloud
> > > > > > > > >> > > > > > > > > > > > > > > <
> > > > > > > > >> > > http://solidfire.com/solution/overview/?video=play
> > > > > > > > >> > > > > >*™*
> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > Find out more about ShapeBlue
> and
> > > our
> > > > > > range
> > > > > > > of
> > > > > > > > >> > > > CloudStack
> > > > > > > > >> > > > > > > > > > > > > > related
> > > > > > > > >> > > > > > > > > > > > > services:
> > > > > > > > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > > > > > > >> > > |
> > > > > > > > >> > > > > > > > CSForge –
> > > > > > > > >> > > > > > > > > > > > > > rapid IaaS deployment framework
> <
> > > > > > > > >> > > > > > > http://shapeblue.com/csforge/
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > > > CloudStack Consulting
> > > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > http://shapeblue.com/cloudstack-consultancy/
> > > > > > > > >
> > > > > > > > >> |
> > > > > > > > >> > > > > > > > > > > > > CloudStack
> > > > > > > > >> > > > > > > > > > > > > > Software Engineering
> > > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > > > > > CloudStack Infrastructure
> Support
> > > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > > >> > > >
> > http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > |
> > > > > > > > >> > > > > > > > > > > > > > CloudStack Bootcamp Training
> > Courses
> > > > > > > > >> > > > > > > > > > > > > > <
> > > > > > http://shapeblue.com/cloudstack-training/>
> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > --
> > > > > > > > >> > > > > > > > > > > > *Mike Tutkowski*
> > > > > > > > >> > > > > > > > > > > > *Senior CloudStack Developer,
> > SolidFire
> > > > > Inc.*
> > > > > > > > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > <javascript:;>
> > > > > > > > >> > > > > <javascript:;>
> > > > > > > > >> > > > > > > > > > > > o: 303.746.7302
> > > > > > > > >> > > > > > > > > > > > Advancing the way the world uses the
> > > cloud
> > > > > > > > >> > > > > > > > > > > > <
> > > > > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > > >> > >*™*
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Find out more about ShapeBlue and our
> > > range
> > > > of
> > > > > > > > >> CloudStack
> > > > > > > > >> > > > > related
> > > > > > > > >> > > > > > > > > > services:
> > > > > > > > >> > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > >> > > > > > > > > > > <
> > > > > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > > > > >
> > > > > > > > >> |
> > > > > > > > >> > > > > CSForge –
> > > > > > > > >> > > > > > > > rapid
> > > > > > > > >> > > > > > > > > > > IaaS deployment framework <
> > > > > > > > >> http://shapeblue.com/csforge/
> > > > > > > > >> > >
> > > > > > > > >> > > > > > > CloudStack
> > > > > > > > >> > > > > > > > > > > Consulting <
> > > > > > > > >> http://shapeblue.com/cloudstack-consultancy/
> > > > > > > > >> > >
> > > > > > > > >> > > |
> > > > > > > > >> > > > > > > > CloudStack
> > > > > > > > >> > > > > > > > > > > Software Engineering
> > > > > > > > >> > > > > > > > > > > <
> > > > > > > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > >> > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > >> > > > > > > > > > > <
> > > > > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > > > > >> > >
> > > > > > > > >> > > |
> > > > > > > > >> > > > > > > > CloudStack
> > > > > > > > >> > > > > > > > > > > Bootcamp Training Courses <
> > > > > > > > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Find out more about ShapeBlue and our
> > range
> > > of
> > > > > > > > >> CloudStack
> > > > > > > > >> > > > related
> > > > > > > > >> > > > > > > > > services:
> > > > > > > > >> > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > >> > > > > > > > > > <
> > > > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > > > >
> > > > > > > > |
> > > > > > > > >> > > > CSForge –
> > > > > > > > >> > > > > > > rapid
> > > > > > > > >> > > > > > > > > > IaaS deployment framework <
> > > > > > > > >> http://shapeblue.com/csforge/>
> > > > > > > > >> > > > > > > > > > CloudStack Consulting <
> > > > > > > > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > > > > > >> > > > > > > |
> > > > > > > > >> > > > > > > > > CloudStack
> > > > > > > > >> > > > > > > > > > Software Engineering
> > > > > > > > >> > > > > > > > > > <
> > > > > > > > http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > >> > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > >> > > > > > > > > > <
> > > > > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > > > > > > >> > |
> > > > > > > > >> > > > > > > CloudStack
> > > > > > > > >> > > > > > > > > > Bootcamp Training Courses <
> > > > > > > > >> > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > --
> > > > > > > > >> > > > > > > *Mike Tutkowski*
> > > > > > > > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > >> > > > > > > e: mike.tutkowski@solidfire.com
> <javascript:;>
> > > > > > > > <javascript:;>
> > > > > > > > >> > > > > > > o: 303.746.7302
> > > > > > > > >> > > > > > > Advancing the way the world uses the cloud
> > > > > > > > >> > > > > > > <
> > > http://solidfire.com/solution/overview/?video=play
> > > > > >*™*
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > --
> > > > > > > > >> > > > > *Mike Tutkowski*
> > > > > > > > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > >> > > > > o: 303.746.7302
> > > > > > > > >> > > > > Advancing the way the world uses the cloud
> > > > > > > > >> > > > > <
> http://solidfire.com/solution/overview/?video=play
> > > >*™*
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > --
> > > > > > > > >> > > > Ian Rae
> > > > > > > > >> > > > CEO | PDG
> > > > > > > > >> > > > c: 514.944.4008
> > > > > > > > >> > > >
> > > > > > > > >> > > > CloudOps | Cloud Infrastructure and Networking
> > Solutions
> > > > > > > > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada
> |
> > > H3J
> > > > > 1S6
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > --
> > > > > > > > >> > *Mike Tutkowski*
> > > > > > > > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > >> > e: mike.tutkowski@solidfire.com
> > > > > > > > >> > o: 303.746.7302
> > > > > > > > >> > Advancing the way the world uses the cloud
> > > > > > > > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Mike Tutkowski*
> > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > > > o: 303.746.7302
> > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Mike Tutkowski*
> > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > > o: 303.746.7302
> > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > e: mike.tutkowski@solidfire.com
> > > > > > o: 303.746.7302
> > > > > > Advancing the way the world uses the cloud
> > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > Find out more about ShapeBlue and our range of CloudStack related
> > > services:
> > > > IaaS Cloud Design & Build
> > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> rapid
> > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
> |
> > > CloudStack
> > > > Software Engineering
> > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > CloudStack Infrastructure Support
> > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> CloudStack
> > > > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/
> >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Hi Syed,

Can you clarify how you see these behaviors (below) working now that we are
considering a global settings value applicable to managed storage with
volume snapshots?

State: Global setting value defaults to keep snapshot on storage system
(primary storage) for managed storage.

1) Action: Volume snapshot is taken with managed storage. Result: Snapshot
is taken and kept on storage system (primary storage). (This is what you
and I have been working on for a couple months now.)

Situation: Global setting value is changed to a non-default value.

2) Action: Volume snapshot is taken with managed storage. Result: ??? I'm
not sure which scenario we're looking at here now.

3) Action: Storage snapshot is taken with managed storage. Result: ??? I'm
not sure which scenario we're looking at here now.

Thanks!
Mike

On Fri, Feb 12, 2016 at 9:15 AM, Syed Mushtaq <sy...@gmail.com>
wrote:

> I agree with Mike's concern about backward compatibility. We can add a
> global flag which makes sure that the way volume snapshots work currently
> on managed storage (stay on the device) is retained after upgrade. We can
> then safely implement the Storage Snapshot API while making the Volume
> Snpashot API move the snapshot to Secondary Storage.
>
> Sounds good guys?
>
> -Syed
>
>
> On Mon, Feb 8, 2016 at 2:56 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com
> > wrote:
>
> > Here's what we have for snapshots for managed storage as of 4.6, Paul:
> >
> > 1. VM snapshots (no proposed changes to this).
> >
> > 2. Volume snapshots that do not end up on secondary storage, but rather
> are
> > stored on a SAN (effectively storing snapshots on primary storage).
> >
> > Pierre-Luc is saying he'd like this for snapshots for managed storage:
> >
> > A. VM snapshots (no proposed changes to this).
> >
> > B. Volume snapshots that export to secondary storage.
> >
> > C. New: Storage snapshots that behave like 2 (above).
> >
> > I like Pierre-Luc's ideas there, but the problem is backward
> compatibility.
> >
> > If customers who were using managed storage with volume snapshots in 4.6
> > were getting their snapshots put on a SAN, then in 4.9 - all of a sudden
> -
> > their new snapshots are put on secondary storage (unless they explicitly
> > change over to using the new Storage snapshots feature).
> >
> >
> >
> > On Mon, Feb 8, 2016 at 12:32 PM, Paul Angus <pa...@shapeblue.com>
> > wrote:
> >
> > > Just to make sure I'm on the same page, are we talking about;
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-9278 ?
> > >
> > > The FS reads (to me) more like 1a + the possibility to export to
> > secondary
> > > storage if required?
> > > Have I understood correctly?
> > > I have seen [1a] implemented for VMware by NetApp in their beta
> > CloudStack
> > > plugin (pleased I can say that without Mike beating me up now). No
> > changes
> > > to the CloudStack API were required. (nb it didn't export to secondary
> > > storage).
> > >
> > >
> > >
> > > 1. VM Snapshot (point-in-time hypervisor based snapshots)
> > > 1a. SAN assisted VM snapshots (point-in-time hypervisor snapshot takes
> > > place on transparently SAN to avoid performance issue in disk chains)
> > > 2. SAN Snapshot (Storage Snapshot) - NEW
> > > 3. Volume Snapshot (current old/slow transfer to secstorage)
> > > 4. Backup - JUST AN IDEAL.
> > >
> > >
> > >
> > >
> > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > Paul Angus
> > > VP Technology ,  ShapeBlue
> > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > > *+44 7711 418784* <+44%207711%20418784>
> > > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > > <pa...@cloudyangus>  |  w:
> > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> > > Services India LLP is a company incorporated in India and is operated
> > under
> > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > > company incorporated in Brasil and is operated under license from Shape
> > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic
> of
> > > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
> > is
> > > a registered trademark.
> > > This email and any attachments to it may be confidential and are
> intended
> > > solely for the use of the individual to whom it is addressed. Any views
> > or
> > > opinions expressed are solely those of the author and do not
> necessarily
> > > represent those of Shape Blue Ltd or related companies. If you are not
> > the
> > > intended recipient of this email, you must neither take any action
> based
> > > upon its contents, nor copy or show it to anyone. Please contact the
> > sender
> > > if you believe you have received this email in error.
> > >
> > >
> > > -----Original Message-----
> > > From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
> > > Sent: Monday, February 8, 2016 7:16 PM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > >
> > > Correct, Will.
> > >
> > > That Global Settings would only be for managed storage. Non-managed
> > > (traditional) volume snapshots are completely un-impacted by this
> > feature.
> > >
> > > If we need to sometimes keep the snapshots on the SAN and sometimes
> push
> > > them to secondary storage, we'll need a more robust solution than
> Global
> > > Settings, though.
> > >
> > > On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > Sorry. I missed a bit of context when I responded. The global setting
> > > > would be only for the managed storage case, currently being called
> > > Storage
> > > > Snapshots, and is only to determine if a copy is pushed to secondary
> > > > storage right? The global setting would not change the behavior of
> the
> > > > Volume Snapshots right?
> > > >
> > > > I was referring to the need for Volume Snapshots and Storage
> Snapshots
> > to
> > > > exist together. Disregard my comment. I caught up on context after I
> > > > posted. My bad...
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > *CloudOps* *| *Cloud Solutions Experts
> > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > w cloudops.com *|* tw @CloudOps_
> > > >
> > > > On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com
> > > > > wrote:
> > > >
> > > > > Hey Will,
> > > > >
> > > > > Who's picking the behavior? Is it the cloud provider or the end
> user?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <
> wstevens@cloudops.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I don't think a global setting is a good option because we need
> > both
> > > > > > functionality to be available at the same time and for different
> > use
> > > > > cases
> > > > > > to be able to pick which they choose.
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > > w cloudops.com *|* tw @CloudOps_
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > > > > mike.tutkowski@solidfire.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Now that I re-read your e-mail, it dawned on me: The end-user
> > > doesn't
> > > > > > care
> > > > > > > where the snapshot is.
> > > > > > >
> > > > > > > If that's true, then we should perhaps control this via Global
> > > > Settings
> > > > > > or
> > > > > > > something.
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > > > > mike.tutkowski@solidfire.com> wrote:
> > > > > > >
> > > > > > > > It's not ideal - true, but it does allow us to be backward
> > > > > compatible.
> > > > > > > >
> > > > > > > > If you have other ideas, though, about how to maintain
> backward
> > > > > > > > compatibility, I'm definitely open to hear them.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > > > > syed1.mushtaq@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Mike,
> > > > > > > >>
> > > > > > > >> Adding a flag to createSnapshot was the first and the most
> > > obvious
> > > > > > thing
> > > > > > > >> that came to our minds. The problem that I had with this was
> > > that:
> > > > > > > >>
> > > > > > > >> 1) I feel it is exposing something to the end user that is
> > > > internal
> > > > > to
> > > > > > > the
> > > > > > > >> cloud.
> > > > > > > >>
> > > > > > > >> 2) We have to follow two different ways of restore/deletion
> in
> > > the
> > > > > > same
> > > > > > > >> code path depending on where the Snapshot resides which I
> find
> > > > kind
> > > > > > of a
> > > > > > > >> bad design.
> > > > > > > >>
> > > > > > > >> But if exposing a archive flag to the end user is acceptable
> > > then
> > > > we
> > > > > > can
> > > > > > > >> definitely use this instead of adding the StorageSnapshot
> API
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> -Syed
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > > > > >> mike.tutkowski@solidfire.com
> > > > > > > >> > wrote:
> > > > > > > >>
> > > > > > > >> > Hi Pierre-Luc,
> > > > > > > >> >
> > > > > > > >> > My recommendation would be this:
> > > > > > > >> >
> > > > > > > >> > Add an "archive" flag to the current volume-snapshot API.
> > Its
> > > > > > default
> > > > > > > >> would
> > > > > > > >> > be "false" because that would be backward compatible with
> > how
> > > > 4.6
> > > > > > has
> > > > > > > >> > volume snapshots implemented (i.e. they stay on the SAN in
> > > 4.6,
> > > > > 4.7,
> > > > > > > and
> > > > > > > >> > 4.8).
> > > > > > > >> >
> > > > > > > >> > If you set archive=true, then we would perform a
> background
> > > > > > migration
> > > > > > > of
> > > > > > > >> > the snapshot from the SAN to the secondary storage (then
> > > delete
> > > > > the
> > > > > > > SAN
> > > > > > > >> > snapshot).
> > > > > > > >> >
> > > > > > > >> > That archive parameter would only be valid for managed
> > > storage.
> > > > > > > >> >
> > > > > > > >> > Sound reasonable?
> > > > > > > >> >
> > > > > > > >> > Also, a VM snapshot that includes disks provided by
> managed
> > > > > storage
> > > > > > > >> should
> > > > > > > >> > work.
> > > > > > > >> >
> > > > > > > >> > Talk to you later,
> > > > > > > >> > Mike
> > > > > > > >> >
> > > > > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > > > > pdion@cloudops.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Mike,
> > > > > > > >> > >
> > > > > > > >> > > In terms of API's, would you prefer introducing a
> > parameter
> > > to
> > > > > the
> > > > > > > >> > existing
> > > > > > > >> > > VolumeSnapshot, example: extract={true|false} with a
> > > > default
> > > > > > > value
> > > > > > > >> of
> > > > > > > >> > > true which would extract snapshot into the secondary
> > > storage,
> > > > > > which
> > > > > > > >> is
> > > > > > > >> > the
> > > > > > > >> > > current default behavior. Then for SAN snapshot that
> > remain
> > > on
> > > > > the
> > > > > > > >> SAN we
> > > > > > > >> > > would just set "extract=false" ? as oppose to create a
> new
> > > > > > > >> > > StorageSnapshot API ?
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Paul,
> > > > > > > >> > >
> > > > > > > >> > > From what I'm seeing so far, we can't do a VM-snapshot
> > when
> > > > > using
> > > > > > > >> managed
> > > > > > > >> > > storage for VM having more than one Volume. For the
> reason
> > > > that
> > > > > > > >> snapshot
> > > > > > > >> > > are performed outside of the hypervisor awareness and
> > > > > > > asynchronously.
> > > > > > > >> If
> > > > > > > >> > > someone have a way to address that, it would make thinks
> > > much
> > > > > more
> > > > > > > >> > > attractive.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <
> > irae@cloudops.com
> > > >
> > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > I think a service provider backup scenario is more
> > likely
> > > to
> > > > > > take
> > > > > > > >> > > advantage
> > > > > > > >> > > > of SAN snapshot. There are a few reasons for this.
> > > > Traditional
> > > > > > > >> backups
> > > > > > > >> > > > involve access to the file system, and there is an
> > > > expectation
> > > > > > > that
> > > > > > > >> > this
> > > > > > > >> > > > can be done with reasonably short time frames without
> > > > > negatively
> > > > > > > >> > > impacting
> > > > > > > >> > > > VM performance, and that the backup orchestrator can
> > apply
> > > > > > various
> > > > > > > >> > logic
> > > > > > > >> > > > and or transformations to the data (compress, encrypt,
> > > > deltas
> > > > > > > >> etc...).
> > > > > > > >> > > > While it is true that one could apply a backup process
> > to
> > > a
> > > > > > cloud
> > > > > > > >> > > snapshot,
> > > > > > > >> > > > this would be slow and inefficient requiring the data
> to
> > > be
> > > > > > moved
> > > > > > > >> > several
> > > > > > > >> > > > times and there are some major bottlenecks with cloud
> > > > > snapshots.
> > > > > > > >> With a
> > > > > > > >> > > > cloud snapshot - there seems to be no reasonable
> > > expectation
> > > > > of
> > > > > > > >> being
> > > > > > > >> > > able
> > > > > > > >> > > > to do differential snapshots (I think this depends on
> > the
> > > > > > > >> hypervisor)
> > > > > > > >> > and
> > > > > > > >> > > > if you do differential snapshots this will make file
> > > backups
> > > > > > from
> > > > > > > >> those
> > > > > > > >> > > > snapshots even more complicated to orchestrate.
> > > > > > > >> > > >
> > > > > > > >> > > > Suspect there needs to be a different thread on how to
> > > > better
> > > > > > > enable
> > > > > > > >> > > > backups, as a service. As per Paul's suggestion, but
> it
> > > is a
> > > > > > > related
> > > > > > > >> > > > workflow so relevant to this discussion.
> > > > > > > >> > > >
> > > > > > > >> > > > Ian
> > > > > > > >> > > >
> > > > > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > > > > >> > > mike.tutkowski@solidfire.com>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > To me it sounds like number two and number three are
> > > > > different
> > > > > > > >> uses
> > > > > > > >> > for
> > > > > > > >> > > > the
> > > > > > > >> > > > > same "thing"(which is totally fine).
> > > > > > > >> > > > >
> > > > > > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > > > > > >> asynchronously, do
> > > > > > > >> > > we
> > > > > > > >> > > > > see the SSVM as performing the export?
> > > > > > > >> > > > >
> > > > > > > >> > > > > To be backwards compatible with what we have in 4.6
> > and
> > > > > later
> > > > > > > for
> > > > > > > >> > > volume
> > > > > > > >> > > > > snapshots for managed storage, I think it might be
> > > easier
> > > > if
> > > > > > we
> > > > > > > >> pass
> > > > > > > >> > > in a
> > > > > > > >> > > > > flag that says whether or not to archive the SAN
> > > snapshot
> > > > > > > (which,
> > > > > > > >> I
> > > > > > > >> > > > think,
> > > > > > > >> > > > > is something that you suggested, as well,
> Pierre-Luc).
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > > > > pdion@cloudops.com
> > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Hi Mike,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > The reason behind the creation of a SAN snapshot
> > which
> > > > is
> > > > > > > >> exported
> > > > > > > >> > > into
> > > > > > > >> > > > > > secondary storage, is because creating a copy of
> the
> > > > .VHD
> > > > > > > >> directly
> > > > > > > >> > > > would
> > > > > > > >> > > > > > impact uptime of the VM as creating that copy take
> > > lots
> > > > of
> > > > > > > time.
> > > > > > > >> > Has
> > > > > > > >> > > > > oppose
> > > > > > > >> > > > > > to a SAN snapshot that is close to instantaneous
> > which
> > > > can
> > > > > > > >> > afterward
> > > > > > > >> > > be
> > > > > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I would suspect an extracted VolumeSnapshot taken
> > > from a
> > > > > SAN
> > > > > > > >> > snapshot
> > > > > > > >> > > > > could
> > > > > > > >> > > > > > have is SAN snapshot deleted to avoid duplica and
> > > space
> > > > > > > >> consumption
> > > > > > > >> > > on
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > Primary Storage side.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > I see 3 definitions in our current discussion
> > > regarding
> > > > > the
> > > > > > > term
> > > > > > > >> > > > snapshot
> > > > > > > >> > > > > > (these are not official terminology but by own
> > > > > > interpretation
> > > > > > > of
> > > > > > > >> > > them):
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's
> > > definition
> > > > > of
> > > > > > a
> > > > > > > >> > > > snapshot):
> > > > > > > >> > > > > > it's a volume snapshot at the storage level, point
> > in
> > > > time
> > > > > > of
> > > > > > > >> your
> > > > > > > >> > > > data.
> > > > > > > >> > > > > it
> > > > > > > >> > > > > > reside on the primary storage. Useful and
> efficient
> > > for
> > > > > > > software
> > > > > > > >> > side
> > > > > > > >> > > > > > incident.
> > > > > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack
> > VolumeSnapshot/
> > > > > cloud
> > > > > > > >> backup
> > > > > > > >> > > > aws-S3
> > > > > > > >> > > > > > style ): Point in time copy of the Virtual Disk
> that
> > > > > reside
> > > > > > > on a
> > > > > > > >> > > > > different
> > > > > > > >> > > > > > storage array then the original Volume. Facilitate
> > > data
> > > > > > > >> migration
> > > > > > > >> > > > between
> > > > > > > >> > > > > > clusters and, in case of primary storage incident,
> > > > Volume
> > > > > > > >> snapshots
> > > > > > > >> > > are
> > > > > > > >> > > > > not
> > > > > > > >> > > > > > impacted and can be reuse.
> > > > > > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines
> data
> > > that
> > > > > > also
> > > > > > > >> > > validate
> > > > > > > >> > > > > data
> > > > > > > >> > > > > > integrity, provide a storage efficient archiving
> > > method
> > > > > and
> > > > > > an
> > > > > > > >> > > > > independent
> > > > > > > >> > > > > > way to restore your data in case of an major
> > > > > infrastructure
> > > > > > > >> > disaster.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Regards,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > PL
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > > <javascript:;>
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > So, let's see if I currently follow the
> > > requirements:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > * Augment volume snapshots for managed storage
> to
> > > > > > > >> conditionally
> > > > > > > >> > > > export
> > > > > > > >> > > > > > data
> > > > > > > >> > > > > > > to NFS. The current process of taking a snapshot
> > on
> > > > the
> > > > > > SAN
> > > > > > > is
> > > > > > > >> > > fine,
> > > > > > > >> > > > > but
> > > > > > > >> > > > > > > we'd like the option to export the data to NFS,
> as
> > > > well.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Questions:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Once the data has been exported to NFS, do we
> keep
> > > the
> > > > > SAN
> > > > > > > >> > snapshot
> > > > > > > >> > > > or
> > > > > > > >> > > > > > > delete it?
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > If we are deleting the SAN snapshot, then why
> > don't
> > > we
> > > > > > just
> > > > > > > >> copy
> > > > > > > >> > > the
> > > > > > > >> > > > > VHD
> > > > > > > >> > > > > > > from primary to secondary the way we do today
> for
> > > > > > > non-managed
> > > > > > > >> > (i.e.
> > > > > > > >> > > > > > > traditional) storage? Why create a SAN snapshot
> in
> > > > that
> > > > > > > >> scenario?
> > > > > > > >> > > > > Perhaps
> > > > > > > >> > > > > > > to have the SSVM mount and perform the VHD copy
> to
> > > > > > secondary
> > > > > > > >> > > storage
> > > > > > > >> > > > > > > instead of a XenServer host?
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Thanks for the clarification.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > By the way, to me a backup is when you copy data
> > > from
> > > > > one
> > > > > > > >> storage
> > > > > > > >> > > > > system
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > another (regardless of features, if any, to
> > restore
> > > > the
> > > > > > data
> > > > > > > >> in
> > > > > > > >> > the
> > > > > > > >> > > > > > > future). A snapshot is a point-in-time view of
> the
> > > > data
> > > > > > of a
> > > > > > > >> > volume
> > > > > > > >> > > > and
> > > > > > > >> > > > > > > it's stored on the same storage system as the
> > > volume.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc
> Dion <
> > > > > > > >> > > pdion@cloudops.com
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > <javascript:;>>
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > That's fun to see that discussion happening. I
> > > 100%
> > > > > > agree
> > > > > > > >> with
> > > > > > > >> > > > Paul's
> > > > > > > >> > > > > > > > points of view. VolumeSnapshot are not a
> backup,
> > > > but I
> > > > > > do
> > > > > > > >> > > consider
> > > > > > > >> > > > > them
> > > > > > > >> > > > > > > as
> > > > > > > >> > > > > > > > a safety vest against Primary Storage failure,
> > > > because
> > > > > > > >> failure
> > > > > > > >> > > > append
> > > > > > > >> > > > > > > :-( .
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > The current proposal around snapshots that
> > reside
> > > on
> > > > > the
> > > > > > > >> > primary
> > > > > > > >> > > > > > storage
> > > > > > > >> > > > > > > or
> > > > > > > >> > > > > > > > snapshots that end in the Secondary Storage is
> > not
> > > > to
> > > > > > > >> address
> > > > > > > >> > > any
> > > > > > > >> > > > > kind
> > > > > > > >> > > > > > > of
> > > > > > > >> > > > > > > > backups requirement because a snapshot is not
> a
> > > > > backup,
> > > > > > > >> event
> > > > > > > >> > an
> > > > > > > >> > > > > > > extracted
> > > > > > > >> > > > > > > > VM snapshot.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > The main idea, and again this is for managed
> > > > storage;
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side
> > > snapshot
> > > > > > > >> capability
> > > > > > > >> > > for
> > > > > > > >> > > > > > fast
> > > > > > > >> > > > > > > > response time that support rollback to
> previous
> > > > > > timestamp,
> > > > > > > >> > create
> > > > > > > >> > > > new
> > > > > > > >> > > > > > > > volume and maybe create template.
> > > > > > > >> > > > > > > > not required to be a new API if the work is
> > > > > already
> > > > > > > >> done, I
> > > > > > > >> > > > think
> > > > > > > >> > > > > > > this
> > > > > > > >> > > > > > > > is a different behaviors than the user
> > expectation
> > > > of
> > > > > a
> > > > > > > >> > > > > > volume-snapshot.
> > > > > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current
> cloudstack
> > > > > > behavior
> > > > > > > >> that
> > > > > > > >> > > > create
> > > > > > > >> > > > > > an
> > > > > > > >> > > > > > > > extraction of a volume into SecondaryStorage
> > which
> > > > can
> > > > > > be
> > > > > > > >> reuse
> > > > > > > >> > > to
> > > > > > > >> > > > > > > create a
> > > > > > > >> > > > > > > > new volume into another Primary Storage. This
> > type
> > > > of
> > > > > > > >> snapshot
> > > > > > > >> > > is a
> > > > > > > >> > > > > > slow
> > > > > > > >> > > > > > > > job since yes it would have to copy the full
> > > volume
> > > > > size
> > > > > > > on
> > > > > > > >> the
> > > > > > > >> > > > > > Secondary
> > > > > > > >> > > > > > > > storage.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > PL
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq
> <
> > > > > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > > >> > > > > > <javascript:;>>
> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > I think I share you view on the 'Ideal
> world'.
> > > > > Backup
> > > > > > > (via
> > > > > > > >> > > Volume
> > > > > > > >> > > > > > > > > Snapshots) is a huge bottleneck in
> Cloudstack.
> > > > This
> > > > > is
> > > > > > > >> > > amplified
> > > > > > > >> > > > > > > > especially
> > > > > > > >> > > > > > > > > when you have a object storage as your
> > secondary
> > > > > > storage
> > > > > > > >> > > because
> > > > > > > >> > > > it
> > > > > > > >> > > > > > > > > requires two copies (one to an NFS staging
> > area
> > > > and
> > > > > > from
> > > > > > > >> > there
> > > > > > > >> > > to
> > > > > > > >> > > > > > > object
> > > > > > > >> > > > > > > > > storage). And not to mention that all these
> > > copies
> > > > > are
> > > > > > > >> > > consuming
> > > > > > > >> > > > > > > > hypervisor
> > > > > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> > > > > bottleneck
> > > > > > as
> > > > > > > >> all
> > > > > > > >> > > the
> > > > > > > >> > > > > > > Network
> > > > > > > >> > > > > > > > > and I/O flow through it. So our intention of
> > > > > proposing
> > > > > > > the
> > > > > > > >> > > > "Storage
> > > > > > > >> > > > > > > > > Snapshots" is to give a better way of
> achiving
> > > > > > snapshots
> > > > > > > >> > while
> > > > > > > >> > > > > still
> > > > > > > >> > > > > > > > > keeping the original definition of volume
> > > > snpashots
> > > > > > (ie
> > > > > > > >> > upload
> > > > > > > >> > > to
> > > > > > > >> > > > > sec
> > > > > > > >> > > > > > > > > storage).
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > But as Erik pointed out volume snapshots are
> > not
> > > > > > > backups.
> > > > > > > >> > They
> > > > > > > >> > > > > don't
> > > > > > > >> > > > > > > work
> > > > > > > >> > > > > > > > > form multi-disk LVM volume groups and
> dynamic
> > > > > disks. I
> > > > > > > am
> > > > > > > >> all
> > > > > > > >> > > in
> > > > > > > >> > > > > for
> > > > > > > >> > > > > > a
> > > > > > > >> > > > > > > > > better backup solution which handles these
> use
> > > > cases
> > > > > > and
> > > > > > > >> > > utilizes
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > > > storage's advanced features.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus
> <
> > > > > > > >> > > > > > paul.angus@shapeblue.com <javascript:;>
> > > <javascript:;>>
> > > > > > > >> > > > > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > In the beginning... there were CloudStack
> > > > > snapshots
> > > > > > > and
> > > > > > > >> > they
> > > > > > > >> > > > were
> > > > > > > >> > > > > > > > > actually
> > > > > > > >> > > > > > > > > > volume snapshots not hypervisor
> > point-in-time
> > > > > > > snapshots.
> > > > > > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > > > > > point-in-time
> > > > > > > >> > > > > hypervisor
> > > > > > > >> > > > > > > > > > snapshots) and we started referring to the
> > > > > original
> > > > > > > >> > snapshots
> > > > > > > >> > > > as
> > > > > > > >> > > > > > > volume
> > > > > > > >> > > > > > > > > > snapshots.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > CloudStack does not offer 'backups', but
> > many
> > > > > people
> > > > > > > use
> > > > > > > >> > > volume
> > > > > > > >> > > > > > > > snapshots
> > > > > > > >> > > > > > > > > > as backups. However you can't in-place
> > restore
> > > > > > volume
> > > > > > > >> > > snapshots
> > > > > > > >> > > > > and
> > > > > > > >> > > > > > > if
> > > > > > > >> > > > > > > > > you
> > > > > > > >> > > > > > > > > > have a VM with multiple volumes, the
> volume
> > > > > > snapshots
> > > > > > > >> must
> > > > > > > >> > be
> > > > > > > >> > > > > done
> > > > > > > >> > > > > > in
> > > > > > > >> > > > > > > > > > series, meaning that the state across all
> of
> > > the
> > > > > > > >> volumes is
> > > > > > > >> > > > > > unlikely
> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > > > > > be
> > > > > > > >> > > > > > > > > > consistent.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > 'Actual Backups' would enable all of the
> > > restore
> > > > > > > options
> > > > > > > >> > > which
> > > > > > > >> > > > > > users
> > > > > > > >> > > > > > > > > might
> > > > > > > >> > > > > > > > > > expect as well options as to where they
> > might
> > > be
> > > > > > > >> stored. In
> > > > > > > >> > > my
> > > > > > > >> > > > > > ideal
> > > > > > > >> > > > > > > > > world
> > > > > > > >> > > > > > > > > > they would also be able to leverage
> back-end
> > > > > > hardware
> > > > > > > >> (such
> > > > > > > >> > > as
> > > > > > > >> > > > > > > > Solidfire,
> > > > > > > >> > > > > > > > > > NetApp etc :) ) and software such as
> Veeam,
> > > > > > Commvault
> > > > > > > >> etc
> > > > > > > >> > to
> > > > > > > >> > > > > > > accelerate
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > > process.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > [image: ShapeBlue] <
> > http://www.shapeblue.com>
> > > > > > > >> > > > > > > > > > Paul Angus
> > > > > > > >> > > > > > > > > > VP Technology , ShapeBlue
> > > > > > > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> 0540*
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > >
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > |
> > > > > > > >> > > > > m:
> > > > > > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > >> > > > > > > > > > e: *paul.angus@shapeblue.com
> <javascript:;>
> > > > > > > >> > <javascript:;>
> > > > > > > >> > > |
> > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > > > >> > > > > <ja...@cloudyangus>
> > > > > > > >> > > > > > | w:
> > > > > > > >> > > > > > > > > > *www.shapeblue.com* <
> > http://www.shapeblue.com
> > > >
> > > > > > > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> > WC2N
> > > > > 4HS
> > > > > > UK
> > > > > > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated
> in
> > > > > England
> > > > > > &
> > > > > > > >> > Wales.
> > > > > > > >> > > > > > > ShapeBlue
> > > > > > > >> > > > > > > > > > Services India LLP is a company
> incorporated
> > > in
> > > > > > India
> > > > > > > >> and
> > > > > > > >> > is
> > > > > > > >> > > > > > operated
> > > > > > > >> > > > > > > > > under
> > > > > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> > Brasil
> > > > > > > >> Consultoria
> > > > > > > >> > > Ltda
> > > > > > > >> > > > > is
> > > > > > > >> > > > > > a
> > > > > > > >> > > > > > > > > > company incorporated in Brasil and is
> > operated
> > > > > under
> > > > > > > >> > license
> > > > > > > >> > > > from
> > > > > > > >> > > > > > > Shape
> > > > > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a
> company
> > > > > > registered
> > > > > > > >> by
> > > > > > > >> > The
> > > > > > > >> > > > > > > Republic
> > > > > > > >> > > > > > > > of
> > > > > > > >> > > > > > > > > > South Africa and is traded under license
> > from
> > > > > Shape
> > > > > > > Blue
> > > > > > > >> > Ltd.
> > > > > > > >> > > > > > > ShapeBlue
> > > > > > > >> > > > > > > > > is
> > > > > > > >> > > > > > > > > > a registered trademark.
> > > > > > > >> > > > > > > > > > This email and any attachments to it may
> be
> > > > > > > confidential
> > > > > > > >> > and
> > > > > > > >> > > > are
> > > > > > > >> > > > > > > > intended
> > > > > > > >> > > > > > > > > > solely for the use of the individual to
> whom
> > > it
> > > > is
> > > > > > > >> > addressed.
> > > > > > > >> > > > Any
> > > > > > > >> > > > > > > views
> > > > > > > >> > > > > > > > > or
> > > > > > > >> > > > > > > > > > opinions expressed are solely those of the
> > > > author
> > > > > > and
> > > > > > > do
> > > > > > > >> > not
> > > > > > > >> > > > > > > > necessarily
> > > > > > > >> > > > > > > > > > represent those of Shape Blue Ltd or
> related
> > > > > > > companies.
> > > > > > > >> If
> > > > > > > >> > > you
> > > > > > > >> > > > > are
> > > > > > > >> > > > > > > not
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > > intended recipient of this email, you must
> > > > neither
> > > > > > > take
> > > > > > > >> any
> > > > > > > >> > > > > action
> > > > > > > >> > > > > > > > based
> > > > > > > >> > > > > > > > > > upon its contents, nor copy or show it to
> > > > anyone.
> > > > > > > Please
> > > > > > > >> > > > contact
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > > > sender
> > > > > > > >> > > > > > > > > > if you believe you have received this
> email
> > in
> > > > > > error.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > -----Original Message-----
> > > > > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > syed1.mushtaq@gmail.com
> > > > > > > >> > > > > <javascript:;> <javascript:;>]
> > > > > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > > >> > > > > > > > > > To: dev@cloudstack.apache.org
> > <javascript:;>
> > > > > > > >> > <javascript:;>
> > > > > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature]
> Storage
> > > > > > Snapshots
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Paul,
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > When you say actual backups, how would it
> be
> > > > > > different
> > > > > > > >> from
> > > > > > > >> > > the
> > > > > > > >> > > > > > > Volume
> > > > > > > >> > > > > > > > > > Snapshots that exist currently. My
> > > understanding
> > > > > is
> > > > > > > that
> > > > > > > >> > > > Backups
> > > > > > > >> > > > > > end
> > > > > > > >> > > > > > > up
> > > > > > > >> > > > > > > > > in
> > > > > > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > > > > > point-in-time
> > > > > > > >> > state
> > > > > > > >> > > of
> > > > > > > >> > > > > > your
> > > > > > > >> > > > > > > > > volume
> > > > > > > >> > > > > > > > > > which can be restored back correct?
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > -Syed
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul
> Angus
> > <
> > > > > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> > > > <javascript:;>>
> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Hi Syed,
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > As I understand it, the SolidFire plugin
> > > will
> > > > > > export
> > > > > > > >> the
> > > > > > > >> > > > > snapshot
> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > > > > > > > secondary storage if the user requests a
> > > > > template
> > > > > > > from
> > > > > > > >> > the
> > > > > > > >> > > > > > snapshot
> > > > > > > >> > > > > > > > or
> > > > > > > >> > > > > > > > > > > wants to download the snapshot from the
> > > cloud.
> > > > > > This
> > > > > > > >> is a
> > > > > > > >> > > > good,
> > > > > > > >> > > > > > > > > > > pragmatic approach and yes Mike the
> > > SolidFire
> > > > > > > storage
> > > > > > > >> is
> > > > > > > >> > > > super
> > > > > > > >> > > > > > > > > > > reliable and snapshots on SolidFire
> arrays
> > > > take
> > > > > up
> > > > > > > >> next
> > > > > > > >> > to
> > > > > > > >> > > no
> > > > > > > >> > > > > > > space.
> > > > > > > >> > > > > > > > > > > BUT I think that we are talking about a
> > more
> > > > > > general
> > > > > > > >> > > purpose
> > > > > > > >> > > > > API,
> > > > > > > >> > > > > > > and
> > > > > > > >> > > > > > > > > > > other storage systems may not be as
> > awesome
> > > as
> > > > > > > Mike's.
> > > > > > > >> > > That's
> > > > > > > >> > > > > my
> > > > > > > >> > > > > > > > > > > concern. Also, the time to transfer for
> > say
> > > > 1TB
> > > > > to
> > > > > > > >> move
> > > > > > > >> > > from
> > > > > > > >> > > > > > > primary
> > > > > > > >> > > > > > > > > > > to sec storage and then create a VM
> > template
> > > > out
> > > > > > of
> > > > > > > it
> > > > > > > >> > may
> > > > > > > >> > > be
> > > > > > > >> > > > > too
> > > > > > > >> > > > > > > > long
> > > > > > > >> > > > > > > > > > for users.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term
> > volume
> > > > > > > snapshot
> > > > > > > >> for
> > > > > > > >> > > > > backup,
> > > > > > > >> > > > > > > > it's
> > > > > > > >> > > > > > > > > > > just that users want to do backups and a
> > > > volume
> > > > > > > >> snapshot
> > > > > > > >> > is
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > only
> > > > > > > >> > > > > > > > > > > type of snapshot that copies the disk
> > > > elsewhere
> > > > > > and
> > > > > > > >> can
> > > > > > > >> > be
> > > > > > > >> > > > > > > scheduled.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > I'm 'pondering' the implications of
> > enabling
> > > > > > actual
> > > > > > > >> > backups
> > > > > > > >> > > > > > > (through
> > > > > > > >> > > > > > > > > > > recognised backup providers) and the
> user
> > > > > > > requirements
> > > > > > > >> > > around
> > > > > > > >> > > > > > them
> > > > > > > >> > > > > > > > > > > (particularly restoration use cases) as
> a
> > > > > separate
> > > > > > > >> thread
> > > > > > > >> > > of
> > > > > > > >> > > > > > work.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > [image: ShapeBlue] <
> > > http://www.shapeblue.com>
> > > > > > Paul
> > > > > > > >> Angus
> > > > > > > >> > > VP
> > > > > > > >> > > > > > > > Technology
> > > > > > > >> > > > > > > > > > > , ShapeBlue
> > > > > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> > 0540*
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > >
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > >> > > > |
> > > > > > > >> > > > > m:
> > > > > > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com
> > <javascript:;>
> > > > > > > >> > > <javascript:;> |
> > > > > > > >> > > > > t: @cloudyangus*
> > > > > > > >> > > > > > > > > > > <paul.angus@shapeblue.com
> <javascript:;>
> > > > > > > >> <javascript:;>
> > > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > >> > > > > > > > > > > *www.shapeblue.com* <
> > > http://www.shapeblue.com
> > > > >
> > > > > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> London
> > > WC2N
> > > > > 4HS
> > > > > > > UK
> > > > > > > >> > Shape
> > > > > > > >> > > > > Blue
> > > > > > > >> > > > > > > Ltd
> > > > > > > >> > > > > > > > > > > is a company incorporated in England &
> > > Wales.
> > > > > > > >> ShapeBlue
> > > > > > > >> > > > > Services
> > > > > > > >> > > > > > > > India
> > > > > > > >> > > > > > > > > > > LLP is a company incorporated in India
> and
> > > is
> > > > > > > operated
> > > > > > > >> > > under
> > > > > > > >> > > > > > > license
> > > > > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> > > > > Consultoria
> > > > > > > >> Ltda
> > > > > > > >> > is
> > > > > > > >> > > a
> > > > > > > >> > > > > > > company
> > > > > > > >> > > > > > > > > > > incorporated in Brasil and is operated
> > under
> > > > > > license
> > > > > > > >> from
> > > > > > > >> > > > Shape
> > > > > > > >> > > > > > > Blue
> > > > > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> > > > > registered
> > > > > > by
> > > > > > > >> The
> > > > > > > >> > > > > Republic
> > > > > > > >> > > > > > > of
> > > > > > > >> > > > > > > > > > > South Africa and is traded under license
> > > from
> > > > > > Shape
> > > > > > > >> Blue
> > > > > > > >> > > Ltd.
> > > > > > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > > > > >> > > > > > > > > > > This email and any attachments to it may
> > be
> > > > > > > >> confidential
> > > > > > > >> > > and
> > > > > > > >> > > > > are
> > > > > > > >> > > > > > > > > > > intended solely for the use of the
> > > individual
> > > > to
> > > > > > > whom
> > > > > > > >> it
> > > > > > > >> > is
> > > > > > > >> > > > > > > > addressed.
> > > > > > > >> > > > > > > > > > > Any views or opinions expressed are
> solely
> > > > those
> > > > > > of
> > > > > > > >> the
> > > > > > > >> > > > author
> > > > > > > >> > > > > > and
> > > > > > > >> > > > > > > do
> > > > > > > >> > > > > > > > > > > not necessarily represent those of Shape
> > > Blue
> > > > > Ltd
> > > > > > or
> > > > > > > >> > > related
> > > > > > > >> > > > > > > > > > > companies. If you are not the intended
> > > > recipient
> > > > > > of
> > > > > > > >> this
> > > > > > > >> > > > email,
> > > > > > > >> > > > > > you
> > > > > > > >> > > > > > > > > > > must neither take any action based upon
> > its
> > > > > > > contents,
> > > > > > > >> nor
> > > > > > > >> > > > copy
> > > > > > > >> > > > > or
> > > > > > > >> > > > > > > > show
> > > > > > > >> > > > > > > > > > > it to anyone. Please contact the sender
> if
> > > you
> > > > > > > believe
> > > > > > > >> > you
> > > > > > > >> > > > have
> > > > > > > >> > > > > > > > > received
> > > > > > > >> > > > > > > > > > this email in error.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > -----Original Message-----
> > > > > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > > syed1.mushtaq@gmail.com
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > <javascript:;>]
> > > > > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org
> > > <javascript:;>
> > > > > > > >> > > <javascript:;>
> > > > > > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature]
> > Storage
> > > > > > > Snapshots
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > I think the terminology confusion comes
> > from
> > > > AWS
> > > > > > > where
> > > > > > > >> > they
> > > > > > > >> > > > do
> > > > > > > >> > > > > > EBS
> > > > > > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack
> > > sort
> > > > of
> > > > > > > >> followed
> > > > > > > >> > > > that.
> > > > > > > >> > > > > > And
> > > > > > > >> > > > > > > > as
> > > > > > > >> > > > > > > > > > > an end user who is oblivious to the
> > > internals
> > > > of
> > > > > > my
> > > > > > > >> > > provider,
> > > > > > > >> > > > > my
> > > > > > > >> > > > > > > > > > > expectation would be something similar
> to
> > > what
> > > > > AWS
> > > > > > > as
> > > > > > > >> > that
> > > > > > > >> > > is
> > > > > > > >> > > > > my
> > > > > > > >> > > > > > > > > > > biggest reference point.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > To your point Mike, I agree that a
> Primary
> > > > > Storage
> > > > > > > >> > failure
> > > > > > > >> > > on
> > > > > > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> > > > > motivations
> > > > > > > >> for us
> > > > > > > >> > > to
> > > > > > > >> > > > > push
> > > > > > > >> > > > > > > > data
> > > > > > > >> > > > > > > > > > > to secondary storage. Primary storage
> > > (atleast
> > > > > for
> > > > > > > us)
> > > > > > > >> > > costs
> > > > > > > >> > > > > > > around 3
> > > > > > > >> > > > > > > > > > > times as much as secondary storage and
> > > > snapshots
> > > > > > on
> > > > > > > >> > primary
> > > > > > > >> > > > > > storage
> > > > > > > >> > > > > > > > > > > are rarely used (especially for some of
> > our
> > > > > > > customers
> > > > > > > >> who
> > > > > > > >> > > do
> > > > > > > >> > > > > > daily
> > > > > > > >> > > > > > > > > > backups).
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike
> > > > Tutkowski
> > > > > <
> > > > > > > >> > > > > > > > > > > mike.tutkowski@solidfire.com
> > <javascript:;>
> > > > > > > >> > > <javascript:;>>
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Some of the weirdness is around
> > > terminology.
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > For most systems I've worked on, a
> > > snapshot
> > > > > and
> > > > > > a
> > > > > > > >> > backup
> > > > > > > >> > > > are
> > > > > > > >> > > > > > two
> > > > > > > >> > > > > > > > > > > > completely different things (but
> > > CloudStack
> > > > > has
> > > > > > > >> > > > traditionally
> > > > > > > >> > > > > > > used
> > > > > > > >> > > > > > > > > > > > the term "volume snapshot" to mean
> > > backup).
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here
> > and
> > > > say,
> > > > > > > >> though,
> > > > > > > >> > > that
> > > > > > > >> > > > > if
> > > > > > > >> > > > > > > your
> > > > > > > >> > > > > > > > > > > > primary storage is running on
> SolidFire
> > > that
> > > > > it
> > > > > > is
> > > > > > > >> > > unlikely
> > > > > > > >> > > > > > > you'll
> > > > > > > >> > > > > > > > > > > > encounter an issue where your primary
> > > > storage
> > > > > > goes
> > > > > > > >> > > offline
> > > > > > > >> > > > > (and
> > > > > > > >> > > > > > > > > > > > you'll even maintain your performance
> > > > > guarantees
> > > > > > > >> during
> > > > > > > >> > > > > failure
> > > > > > > >> > > > > > > > > > > > scenarios and upgrades, as well). That
> > > being
> > > > > the
> > > > > > > >> case,
> > > > > > > >> > it
> > > > > > > >> > > > is
> > > > > > > >> > > > > > less
> > > > > > > >> > > > > > > > > > > > useful to require a backup to Swift
> (but
> > > > it's
> > > > > > > >> perfectly
> > > > > > > >> > > OK
> > > > > > > >> > > > if
> > > > > > > >> > > > > > > > that's
> > > > > > > >> > > > > > > > > > > > what we want to do
> > > > > > > >> > > > > > > > > > > here).
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed
> > > Mushtaq
> > > > > > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com
> <javascript:;>
> > > > > > > >> > <javascript:;>>
> > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Paul,
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > I believe with the current
> > > implementation
> > > > of
> > > > > > > >> > Snapshots
> > > > > > > >> > > on
> > > > > > > >> > > > > > > managed
> > > > > > > >> > > > > > > > > > > > > storage
> > > > > > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never
> > > > exported
> > > > > > to
> > > > > > > >> the
> > > > > > > >> > > > > secondary
> > > > > > > >> > > > > > > > > > storage.
> > > > > > > >> > > > > > > > > > > > > While this solves the problem of
> > having
> > > > > > > snapshots
> > > > > > > >> > > taking
> > > > > > > >> > > > > > > forever
> > > > > > > >> > > > > > > > > > > > > to get to sec storage, this leaves
> us
> > > > with a
> > > > > > > >> > > > > > > > > > > > huge
> > > > > > > >> > > > > > > > > > > > > liability if our primary storage
> goes
> > > > down.
> > > > > We
> > > > > > > see
> > > > > > > >> > > > > snapshots
> > > > > > > >> > > > > > as
> > > > > > > >> > > > > > > > > > > > > our recovery path because we store
> > them
> > > in
> > > > > > Swift
> > > > > > > >> > which
> > > > > > > >> > > is
> > > > > > > >> > > > > > > > reliable
> > > > > > > >> > > > > > > > > > > > > and resilient to failures.
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > With Storage snpashots our goal is
> to
> > > have
> > > > > > > Volume
> > > > > > > >> > > > snapshots
> > > > > > > >> > > > > > > > always
> > > > > > > >> > > > > > > > > > > > > backed up to secondary storage and
> > > Storage
> > > > > > > >> Snapshots
> > > > > > > >> > > stay
> > > > > > > >> > > > > on
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > > > > primary
> > > > > > > >> > > > > > > > > > > > storage.
> > > > > > > >> > > > > > > > > > > > > A provider could potentially mix
> both
> > > > these
> > > > > > and
> > > > > > > >> solve
> > > > > > > >> > > the
> > > > > > > >> > > > > > > problem
> > > > > > > >> > > > > > > > > > > > > that you mentioned where you want to
> > > meet
> > > > > > user's
> > > > > > > >> > > > > expectation
> > > > > > > >> > > > > > > of a
> > > > > > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage)
> > > while
> > > > > > having
> > > > > > > >> an
> > > > > > > >> > > > ability
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on
> the
> > > > > device)
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > > > > > -Syed
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul
> > > Angus
> > > > > > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com
> > > <javascript:;>
> > > > > > > >> > > <javascript:;>>
> > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > HI guys,
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Could someone point me to the Jira
> > bug
> > > > of
> > > > > FS
> > > > > > > for
> > > > > > > >> > the
> > > > > > > >> > > > > > > > > > > > > > SAN-snapshot
> > > > > > > >> > > > > > > > > > > > feature
> > > > > > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > From my discussions with users and
> > > > > operators
> > > > > > > >> around
> > > > > > > >> > > > > > snapshots
> > > > > > > >> > > > > > > > > > > > > > I'd make
> > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > following observations:
> > > > > > > >> > > > > > > > > > > > > > a. 'users' use snapshots as
> backups
> > > > (both
> > > > > > > >> long-term
> > > > > > > >> > > and
> > > > > > > >> > > > > > short
> > > > > > > >> > > > > > > > > > > > > > term)
> > > > > > > >> > > > > > > > > > > > with
> > > > > > > >> > > > > > > > > > > > > > the expectation that they can use
> > them
> > > > for
> > > > > > > >> recovery
> > > > > > > >> > > if
> > > > > > > >> > > > > > > > required.
> > > > > > > >> > > > > > > > > > > > > > b. operators fall back to
> snapshots
> > if
> > > > > > > something
> > > > > > > >> > has
> > > > > > > >> > > > gone
> > > > > > > >> > > > > > > wrong
> > > > > > > >> > > > > > > > > > > > > > with primary storage.
> > > > > > > >> > > > > > > > > > > > > > c. users sometimes want to be able
> > to
> > > > > export
> > > > > > > >> > > snapshots
> > > > > > > >> > > > as
> > > > > > > >> > > > > > > well
> > > > > > > >> > > > > > > > > > > > > > as
> > > > > > > >> > > > > > > > > > > > create
> > > > > > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > > > > > >> > > > > > > > > > > > > > d. snapshots are a currently a
> > massive
> > > > > pain
> > > > > > > for
> > > > > > > >> > > > > operators,
> > > > > > > >> > > > > > I
> > > > > > > >> > > > > > > > > > > > > > know at
> > > > > > > >> > > > > > > > > > > > > least
> > > > > > > >> > > > > > > > > > > > > > one public cloud who have
> snapshots
> > > > which
> > > > > > > take 2
> > > > > > > >> > days
> > > > > > > >> > > > to
> > > > > > > >> > > > > > > > > complete.
> > > > > > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't
> be
> > > used
> > > > > for
> > > > > > > >> > multiple
> > > > > > > >> > > > LVM
> > > > > > > >> > > > > > > > disks.
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > I think the process Mike has used
> in
> > > the
> > > > > > > >> SolidFire
> > > > > > > >> > > > plugin
> > > > > > > >> > > > > > > (only
> > > > > > > >> > > > > > > > > > > > > > moving
> > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > disk image to secondary storage
> when
> > > you
> > > > > > > >> absolutely
> > > > > > > >> > > > have
> > > > > > > >> > > > > > to)
> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > a very
> > > > > > > >> > > > > > > > > > > > > good
> > > > > > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder
> > what
> > > > > > problems
> > > > > > > >> an
> > > > > > > >> > > > > operator
> > > > > > > >> > > > > > > > might
> > > > > > > >> > > > > > > > > > > > > experience
> > > > > > > >> > > > > > > > > > > > > > if they have an issue with a given
> > > > primary
> > > > > > > >> storage
> > > > > > > >> > > pool
> > > > > > > >> > > > > in
> > > > > > > >> > > > > > a
> > > > > > > >> > > > > > > > > > cluster.
> > > > > > > >> > > > > > > > > > > > (I
> > > > > > > >> > > > > > > > > > > > > > know that that is REALLY unlikely
> in
> > > the
> > > > > > > >> SolidFire
> > > > > > > >> > > case
> > > > > > > >> > > > > > Mike
> > > > > > > >> > > > > > > :)
> > > > > > > >> > > > > > > > > > > > > > ) And
> > > > > > > >> > > > > > > > > > > > if
> > > > > > > >> > > > > > > > > > > > > > the transfer from primary to
> > secondary
> > > > is
> > > > > > > slow,
> > > > > > > >> the
> > > > > > > >> > > > time
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > > > > > > > > being able
> > > > > > > >> > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > create a template or export the
> > volume
> > > > > will
> > > > > > be
> > > > > > > >> > slow.
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > So for me the issue is around
> making
> > > > sure
> > > > > > that
> > > > > > > >> the
> > > > > > > >> > > end
> > > > > > > >> > > > > > users
> > > > > > > >> > > > > > > > > > > > expectations
> > > > > > > >> > > > > > > > > > > > > > are met (while improving the
> > > > > > speed/efficiency
> > > > > > > of
> > > > > > > >> > the
> > > > > > > >> > > > back
> > > > > > > >> > > > > > > end)
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > > > > > http://www.shapeblue.com>
> > > > > > > >> Paul
> > > > > > > >> > > > Angus
> > > > > > > >> > > > > > VP
> > > > > > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > > > > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203
> > 603
> > > > > 0540*
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > >
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > >> > > > > > > |
> > > > > > > >> > > > > > > > m:
> > > > > > > >> > > > > > > > > > > > > > *+44 7711 418784*
> > > <+44%207711%20418784>
> > > > > > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;> | t:
> > > > > > > >> > > > > > @cloudyangus*
> > > > > > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com
> > > > <javascript:;>
> > > > > > > >> > > > <javascript:;>
> > > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > > > > > http://www.shapeblue.com
> > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > > > London
> > > > > > WC2N
> > > > > > > >> 4HS
> > > > > > > >> > UK
> > > > > > > >> > > > > Shape
> > > > > > > >> > > > > > > > Blue
> > > > > > > >> > > > > > > > > > > > > > Ltd is a company incorporated in
> > > > England &
> > > > > > > >> Wales.
> > > > > > > >> > > > > ShapeBlue
> > > > > > > >> > > > > > > > > > > > > > Services India LLP is a company
> > > > > incorporated
> > > > > > > in
> > > > > > > >> > India
> > > > > > > >> > > > and
> > > > > > > >> > > > > > is
> > > > > > > >> > > > > > > > > > > > > > operated
> > > > > > > >> > > > > > > > > > > > > under
> > > > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape
> > > Blue
> > > > > > Brasil
> > > > > > > >> > > > > Consultoria
> > > > > > > >> > > > > > > Ltda
> > > > > > > >> > > > > > > > > > > > > > is a company incorporated in
> Brasil
> > > and
> > > > is
> > > > > > > >> operated
> > > > > > > >> > > > under
> > > > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd.
> > ShapeBlue
> > > > SA
> > > > > > Pty
> > > > > > > >> Ltd
> > > > > > > >> > is
> > > > > > > >> > > a
> > > > > > > >> > > > > > > company
> > > > > > > >> > > > > > > > > > > > > > registered by The Republic
> > > > > > > >> > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > > > South Africa and is traded under
> > > license
> > > > > > from
> > > > > > > >> Shape
> > > > > > > >> > > > Blue
> > > > > > > >> > > > > > Ltd.
> > > > > > > >> > > > > > > > > > > > > > ShapeBlue
> > > > > > > >> > > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > a registered trademark.
> > > > > > > >> > > > > > > > > > > > > > This email and any attachments to
> it
> > > may
> > > > > be
> > > > > > > >> > > > confidential
> > > > > > > >> > > > > > and
> > > > > > > >> > > > > > > > are
> > > > > > > >> > > > > > > > > > > > intended
> > > > > > > >> > > > > > > > > > > > > > solely for the use of the
> individual
> > > to
> > > > > whom
> > > > > > > it
> > > > > > > >> is
> > > > > > > >> > > > > > addressed.
> > > > > > > >> > > > > > > > > > > > > > Any views
> > > > > > > >> > > > > > > > > > > > > or
> > > > > > > >> > > > > > > > > > > > > > opinions expressed are solely
> those
> > of
> > > > the
> > > > > > > >> author
> > > > > > > >> > and
> > > > > > > >> > > > do
> > > > > > > >> > > > > > not
> > > > > > > >> > > > > > > > > > > > necessarily
> > > > > > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd
> or
> > > > > related
> > > > > > > >> > > companies.
> > > > > > > >> > > > > If
> > > > > > > >> > > > > > > you
> > > > > > > >> > > > > > > > > > > > > > are not
> > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > intended recipient of this email,
> > you
> > > > must
> > > > > > > >> neither
> > > > > > > >> > > take
> > > > > > > >> > > > > any
> > > > > > > >> > > > > > > > > > > > > > action
> > > > > > > >> > > > > > > > > > > > based
> > > > > > > >> > > > > > > > > > > > > > upon its contents, nor copy or
> show
> > it
> > > > to
> > > > > > > >> anyone.
> > > > > > > >> > > > Please
> > > > > > > >> > > > > > > > contact
> > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > sender
> > > > > > > >> > > > > > > > > > > > > > if you believe you have received
> > this
> > > > > email
> > > > > > in
> > > > > > > >> > error.
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > -----Original Message-----
> > > > > > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > > > > > >> pdion@cloudops.com
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > <javascript:;>]
> > > > > > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016
> 12:56
> > > PM
> > > > > > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New
> Feature]
> > > > > Storage
> > > > > > > >> > Snapshots
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Hi Mike,
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > > > > > >> StorageSnapshot
> > > > > > > >> > > for
> > > > > > > >> > > > > > > managed
> > > > > > > >> > > > > > > > > > > > > > storage
> > > > > > > >> > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > because the VolumeSnapshot
> default,
> > or
> > > > > > > expected,
> > > > > > > >> > > > behavior
> > > > > > > >> > > > > > is
> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > archive snapshots into the
> Secondary
> > > > > > Storage.
> > > > > > > >> So a
> > > > > > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > > > > > >> > > > > > > > > > > > > for
> > > > > > > >> > > > > > > > > > > > > > snapshot that remain on the
> managed
> > > > > storage
> > > > > > > >> > > appliance.
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Quickly looking at the API doc
> and I
> > > > don't
> > > > > > > see a
> > > > > > > >> > > strong
> > > > > > > >> > > > > > > > > > > > > > requirement for volume snapshots
> to
> > be
> > > > > moved
> > > > > > > >> into
> > > > > > > >> > > > > secondary
> > > > > > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot
> > API
> > > > is
> > > > > > not
> > > > > > > >> > useful,
> > > > > > > >> > > > but
> > > > > > > >> > > > > > > both
> > > > > > > >> > > > > > > > > > > > > > use cases are required. A snapshot
> > > that
> > > > > > remain
> > > > > > > >> on
> > > > > > > >> > the
> > > > > > > >> > > > > > managed
> > > > > > > >> > > > > > > > > > > > > > storage, and another type of
> > > > > > > >> > > > > > > > > > > > snapshot
> > > > > > > >> > > > > > > > > > > > > > that end up into the secondary
> > > storage.
> > > > > > Since
> > > > > > > >> > you've
> > > > > > > >> > > > > done a
> > > > > > > >> > > > > > > lot
> > > > > > > >> > > > > > > > > > > > > > of
> > > > > > > >> > > > > > > > > > > > work,
> > > > > > > >> > > > > > > > > > > > > > might easier to just add a
> parameter
> > > to
> > > > > the
> > > > > > > >> current
> > > > > > > >> > > > > > snapshot
> > > > > > > >> > > > > > > > API
> > > > > > > >> > > > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > > > trigger an extraction of the
> storage
> > > > > > snapshot
> > > > > > > >> into
> > > > > > > >> > > the
> > > > > > > >> > > > > > > > secondary
> > > > > > > >> > > > > > > > > > > > storage?
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > PL
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM,
> Mike
> > > > > > > Tutkowski <
> > > > > > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> > > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > I think that all sounds
> reasonable
> > > > then
> > > > > -
> > > > > > > >> thanks!
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM,
> > Syed
> > > > > > > Mushtaq <
> > > > > > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com
> <javascript:;>
> > > > > > > >> <javascript:;>>
> > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >> You are correct Mike in terms
> of
> > > the
> > > > > > > >> > requirements.
> > > > > > > >> > > > One
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > our
> > > > > > > >> > > > > > > > > > > > earlier
> > > > > > > >> > > > > > > > > > > > > > >> iterations on this was to have
> an
> > > > > > argument
> > > > > > > to
> > > > > > > >> > the
> > > > > > > >> > > > > create
> > > > > > > >> > > > > > > > > > > > > > >> snapshot
> > > > > > > >> > > > > > > > > > > > API
> > > > > > > >> > > > > > > > > > > > > > >> which decides whether to backup
> > the
> > > > > > volume
> > > > > > > to
> > > > > > > >> > sec
> > > > > > > >> > > > > > storage
> > > > > > > >> > > > > > > > but
> > > > > > > >> > > > > > > > > > > > > > >> we realized it would make
> > > management
> > > > of
> > > > > > > >> > snapshots
> > > > > > > >> > > > > quite
> > > > > > > >> > > > > > > > messy
> > > > > > > >> > > > > > > > > > > > > > >> so we proposed a new api
> instead.
> > > > > > > >> > > > > > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM
> Mike
> > > > > > Tutkowski
> > > > > > > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > > > > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>>
> > > > > > > >> > > > > > > > > > > > > > >> wrote:
> > > > > > > >> > > > > > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > > > > > >>> Hi,
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> Just to make sure I understand
> > all
> > > > the
> > > > > > > >> > > requirements
> > > > > > > >> > > > > > here:
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> 1) This relates only to
> managed
> > > > > storage
> > > > > > > (1:1
> > > > > > > >> > > > mapping
> > > > > > > >> > > > > > > > between
> > > > > > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend
> SAN
> > > > > > volume).
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > > > > > (introduced
> > > > > > > >> in
> > > > > > > >> > > 4.6)
> > > > > > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> > > > > snapshot
> > > > > > on
> > > > > > > >> the
> > > > > > > >> > > SAN,
> > > > > > > >> > > > > and
> > > > > > > >> > > > > > > > > > > > > > >>> extend it via a config option
> > (or
> > > > > > > >> > > > > > > > > > > > > > >>> something) to not only take
> the
> > > SAN
> > > > > > > >> snapshot,
> > > > > > > >> > but
> > > > > > > >> > > > to
> > > > > > > >> > > > > > copy
> > > > > > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer
> > > only)
> > > > to
> > > > > > > NFS.
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always
> > > taken.
> > > > > > It's
> > > > > > > >> the
> > > > > > > >> > > > backup
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > NFS
> > > > > > > >> > > > > > > > > > > > > > >>> that is optional.
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> 4) Templates can be created
> from
> > > the
> > > > > > > >> snapshot
> > > > > > > >> > > > that's
> > > > > > > >> > > > > on
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be
> > > created
> > > > > > from
> > > > > > > >> the
> > > > > > > >> > > > > snapshot
> > > > > > > >> > > > > > > > > > > > > > >>> that's on
> > > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > >>> SAN (already works as long as
> > the
> > > > new
> > > > > > > >> > CloudStack
> > > > > > > >> > > > > volume
> > > > > > > >> > > > > > > > ends
> > > > > > > >> > > > > > > > > > > > > > >>> up on the same primary
> storage).
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> Would we have a need for a
> > storage
> > > > > > > snapshot
> > > > > > > >> API
> > > > > > > >> > > > then
> > > > > > > >> > > > > or
> > > > > > > >> > > > > > > > > > > > > > >>> would that just be the
> standard
> > > > volume
> > > > > > > >> snapshot
> > > > > > > >> > > > > without
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > >>> backup to
> > > > > > > >> > > > > > > > > > > NFS?
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> Thanks!
> > > > > > > >> > > > > > > > > > > > > > >>> Mike
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24
> PM,
> > > Syed
> > > > > > > Mushtaq
> > > > > > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> > > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>>
> > > > > > > >> > > > > > > > > > > > > > >>> wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > > > > > functionalities
> > > > > > > >> > > > > (snapshot
> > > > > > > >> > > > > > on
> > > > > > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > > > > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because
> > > Ideally,
> > > > we
> > > > > > > would
> > > > > > > >> > like
> > > > > > > >> > > > to
> > > > > > > >> > > > > > have
> > > > > > > >> > > > > > > > > both.
> > > > > > > >> > > > > > > > > > > > > > >>>> For example, some of our
> > > customers
> > > > > want
> > > > > > > to
> > > > > > > >> > > > implement
> > > > > > > >> > > > > > > their
> > > > > > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> > > > > encryption
> > > > > > > to
> > > > > > > >> > their
> > > > > > > >> > > > > > backups
> > > > > > > >> > > > > > > > > > > > > > >>>> which is a perfect use case
> for
> > > > > Storage
> > > > > > > >> > Snapshot
> > > > > > > >> > > > > while
> > > > > > > >> > > > > > > our
> > > > > > > >> > > > > > > > > > > > > > >>>> other customers will still
> keep
> > > > using
> > > > > > the
> > > > > > > >> > > standard
> > > > > > > >> > > > > > > volume
> > > > > > > >> > > > > > > > > > > snapshot.
> > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > >> > > > > > > > > > > > > > >>>> To keep things backward
> > > compatible,
> > > > > we
> > > > > > > can
> > > > > > > >> > add a
> > > > > > > >> > > > > > setting
> > > > > > > >> > > > > > > > > > > > > > >>>> which
> > > > > > > >> > > > > > > > > > > > says
> > > > > > > >> > > > > > > > > > > > > > >>>> to not upload on secondary
> > > storage,
> > > > > > > >> because,
> > > > > > > >> > > after
> > > > > > > >> > > > > > all,
> > > > > > > >> > > > > > > > you
> > > > > > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot
> first
> > > > when
> > > > > > > doing
> > > > > > > >> a
> > > > > > > >> > > > Volume
> > > > > > > >> > > > > > > > > Snapshot.
> > > > > > > >> > > > > > > > > > > > > > >>>> You could stop the process
> > there
> > > > and
> > > > > > not
> > > > > > > do
> > > > > > > >> > the
> > > > > > > >> > > > > > upload.
> > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > >> > > > > > > > > > > > > > >>>> What do you think about this
> > > > > approach?
> > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > > > > > >> > > > > > > > > > > > > > >>>> -Syed
> > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25
> PM,
> > > > Mike
> > > > > > > >> > Tutkowski <
> > > > > > > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > > > > > > >> <javascript:;>
> > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking
> > out
> > > > > load
> > > > > > > >> here,
> > > > > > > >> > but
> > > > > > > >> > > > if
> > > > > > > >> > > > > a
> > > > > > > >> > > > > > > > given
> > > > > > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't
> > > actually
> > > > > need
> > > > > > > to
> > > > > > > >> > > provide
> > > > > > > >> > > > > > both
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > > > ability
> > > > > > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and
> > > export
> > > > it
> > > > > > to
> > > > > > > >> NFS
> > > > > > > >> > (if
> > > > > > > >> > > > > just
> > > > > > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is
> OK),
> > > then
> > > > > we
> > > > > > > >> might
> > > > > > > >> > be
> > > > > > > >> > > > able
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > get
> > > > > > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > > > > > >> > > > > > > > > > > > API
> > > > > > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a
> > new
> > > > > > custom
> > > > > > > >> > > snapshot
> > > > > > > >> > > > > > > strategy
> > > > > > > >> > > > > > > > > > > > > > >>>>> and
> > > > > > > >> > > > > > > > > > > > data
> > > > > > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle
> the
> > > case
> > > > > > where
> > > > > > > >> the
> > > > > > > >> > > > > > CloudStack
> > > > > > > >> > > > > > > > > > > > > > >>>>> cloud
> > > > > > > >> > > > > > > > > > > > does
> > > > > > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > > > > > >> exported-to-NFS
> > > > > > > >> > > > > backup.
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>> In other words, the
> "default"
> > > > > behavior
> > > > > > > >> would
> > > > > > > >> > be
> > > > > > > >> > > > to
> > > > > > > >> > > > > > use
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data
> > > motion
> > > > > > > strategy
> > > > > > > >> > that
> > > > > > > >> > > > we
> > > > > > > >> > > > > > > > already
> > > > > > > >> > > > > > > > > > > > > > >>>>> have (the one that only
> takes
> > a
> > > > SAN
> > > > > > > >> > snapshot).
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud,
> > > however,
> > > > > > wants
> > > > > > > >> to
> > > > > > > >> > > take
> > > > > > > >> > > > a
> > > > > > > >> > > > > > SAN
> > > > > > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data
> > > > exported
> > > > > to
> > > > > > > >> NFS,
> > > > > > > >> > > then
> > > > > > > >> > > > we
> > > > > > > >> > > > > > > could
> > > > > > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing
> > > config
> > > > > > file
> > > > > > > to
> > > > > > > >> > make
> > > > > > > >> > > > use
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > a
> > > > > > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and
> data
> > > > > motion
> > > > > > > >> > strategy
> > > > > > > >> > > > that
> > > > > > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > > > > > >> > > > > > > > > > > > > > activities.
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior
> is
> > > > still
> > > > > > the
> > > > > > > >> > default
> > > > > > > >> > > > for
> > > > > > > >> > > > > > > > users,
> > > > > > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can
> > change
> > > > > this
> > > > > > > >> > behavior
> > > > > > > >> > > > via
> > > > > > > >> > > > > > > > > > > configuration.
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55
> > AM,
> > > > > Mike
> > > > > > > >> > > Tutkowski <
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> mike.tutkowski@solidfire.com
> > > > > > > >> <javascript:;>
> > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will
> need
> > to
> > > > > come
> > > > > > up
> > > > > > > >> > with a
> > > > > > > >> > > > > > viable
> > > > > > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some
> > reasonable
> > > > way
> > > > > > for
> > > > > > > >> them
> > > > > > > >> > > to
> > > > > > > >> > > > > move
> > > > > > > >> > > > > > > > from
> > > > > > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way
> > (and
> > > > > some
> > > > > > > >> obvious
> > > > > > > >> > > way
> > > > > > > >> > > > > > that
> > > > > > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > > > > > >> > > > > > > > > > > > to
> > > > > > > >> > > > > > > > > > > > > > do this).
> > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at
> 11:45
> > > AM,
> > > > > Syed
> > > > > > > >> > Mushtaq
> > > > > > > >> > > <
> > > > > > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > > > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about
> > the
> > > > > > upgrade
> > > > > > > >> path
> > > > > > > >> > > > > however,
> > > > > > > >> > > > > > > > > > > > > > >>>>>>> customers who are using
> 4.6
> > > and
> > > > > are
> > > > > > > on a
> > > > > > > >> > > > managed
> > > > > > > >> > > > > > > > storage
> > > > > > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the
> > same
> > > > > > > >> functionality
> > > > > > > >> > > > with
> > > > > > > >> > > > > > > Volume
> > > > > > > >> > > > > > > > > > > Snapshots.
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at
> 1:43
> > > PM,
> > > > > Syed
> > > > > > > >> > Mushtaq
> > > > > > > >> > > <
> > > > > > > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > > > > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> So if I understand
> > correctly,
> > > > > > > currently
> > > > > > > >> > > > taking a
> > > > > > > >> > > > > > > > Volume
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on
> a
> > > > > managed
> > > > > > > >> storage
> > > > > > > >> > > > keeps
> > > > > > > >> > > > > > it
> > > > > > > >> > > > > > > on
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a
> > part
> > > of
> > > > > > this
> > > > > > > >> > > feature,
> > > > > > > >> > > > we
> > > > > > > >> > > > > > can
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > > > > > > >> > > > > > > > > > > > that
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on
> managed
> > > > > storage
> > > > > > > are
> > > > > > > >> > > > uploaded
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This
> > would
> > > > > make
> > > > > > > the
> > > > > > > >> > > Volume
> > > > > > > >> > > > > > > Snapshot
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same
> > > > > regardless
> > > > > > of
> > > > > > > >> the
> > > > > > > >> > > > > storage
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for
> > > utilizing
> > > > > the
> > > > > > > >> > > efficient
> > > > > > > >> > > > > > > backend
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> storage
> > > > > > > >> > > > > > > > > > > > > > capabilities, we can use the new
> > > storage
> > > > > > > >> snapshots
> > > > > > > >> > > API.
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at
> 1:36
> > > PM,
> > > > > > Mike
> > > > > > > >> > > > Tutkowski <
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > mike.tutkowski@solidfire.com
> > > > > > > >> > <javascript:;>
> > > > > > > >> > > > > <javascript:;>>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we
> > need
> > > > to
> > > > > > > have a
> > > > > > > >> > plan
> > > > > > > >> > > > to
> > > > > > > >> > > > > > deal
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we
> > > already
> > > > > > have a
> > > > > > > >> > > feature
> > > > > > > >> > > > > (in
> > > > > > > >> > > > > > > 4.6
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> and
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you
> to
> > > use
> > > > > the
> > > > > > > >> > existing
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to
> > > > create a
> > > > > > > >> volume
> > > > > > > >> > > > > snapshot
> > > > > > > >> > > > > > > (for
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides
> on a
> > > > > backend
> > > > > > > SAN
> > > > > > > >> > > > (using a
> > > > > > > >> > > > > > > > custom
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a
> > > custom
> > > > > > data
> > > > > > > >> > motion
> > > > > > > >> > > > > > > strategy).
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in,
> > > then
> > > > > how
> > > > > > > >> should
> > > > > > > >> > > the
> > > > > > > >> > > > > > > original
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> implementation (present
> in
> > > 4.6
> > > > > and
> > > > > > > >> later)
> > > > > > > >> > > be
> > > > > > > >> > > > > > > changed?
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > > > > > > >> > > > > > > > > > > > is
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we
> support
> > > > > > customers
> > > > > > > >> who
> > > > > > > >> > > are
> > > > > > > >> > > > > > > already
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> using
> > > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot
> > API
> > > > to
> > > > > > take
> > > > > > > >> > > > snapshots
> > > > > > > >> > > > > > on a
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > > > > > > >> > > > > > > > > > > > > SAN?
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at
> > 11:27
> > > > AM,
> > > > > > > Will
> > > > > > > >> > > > Stevens <
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > > > > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to
> > create
> > > a
> > > > > > > Template
> > > > > > > >> > > from a
> > > > > > > >> > > > > > > > > > > > StorageSnapshot?
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the
> template
> > > be
> > > > > > stored
> > > > > > > >> in
> > > > > > > >> > the
> > > > > > > >> > > > > > > secondary
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal
> > > templates
> > > > > or
> > > > > > > will
> > > > > > > >> > that
> > > > > > > >> > > > be
> > > > > > > >> > > > > > > > handled
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > >> > > > > > > > > > > > > > vendor side?
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud
> > > > Solutions
> > > > > > > >> Experts
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|*
> Montreal
> > > *|*
> > > > > > Quebec
> > > > > > > >> *|*
> > > > > > > >> > > H3J
> > > > > > > >> > > > > 1S6
> > > > > > > >> > > > > > w
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw
> > > > @CloudOps_
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at
> > 1:22
> > > > PM,
> > > > > > > Syed
> > > > > > > >> > > > Mushtaq <
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> syed1.mushtaq@gmail.com
> > > > > > > >> <javascript:;>
> > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at
> > > 1:19
> > > > > PM,
> > > > > > > Will
> > > > > > > >> > > > Stevens
> > > > > > > >> > > > > <
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> > > > > > > >> <javascript:;>
> > > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked
> the
> > > > > Design
> > > > > > > >> Spec in
> > > > > > > >> > > the
> > > > > > > >> > > > > > Jira
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was
> > not
> > > > > clear
> > > > > > > in
> > > > > > > >> the
> > > > > > > >> > > > > > 'mention'
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it
> > shows
> > > > as a
> > > > > > > page
> > > > > > > >> > 'you
> > > > > > > >> > > do
> > > > > > > >> > > > > not
> > > > > > > >> > > > > > > > have
> > > > > > > >> > > > > > > > > > > permission to'.
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud
> > > > > Solutions
> > > > > > > >> Experts
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|*
> > Montreal
> > > > *|*
> > > > > > > Quebec
> > > > > > > >> > *|*
> > > > > > > >> > > > H3J
> > > > > > > >> > > > > > 1S6
> > > > > > > >> > > > > > > w
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw
> > > > > @CloudOps_
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016
> at
> > > 1:02
> > > > > PM,
> > > > > > > >> Syed
> > > > > > > >> > > Ahmed
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > > > >> <javascript:;>
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > >
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a
> > new
> > > > set
> > > > > > of
> > > > > > > >> APIs
> > > > > > > >> > to
> > > > > > > >> > > > do
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed
> > > > storage
> > > > > > > >> backends
> > > > > > > >> > > > like
> > > > > > > >> > > > > > > > > SolidFire.
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current
> > > > managed
> > > > > > > >> storage
> > > > > > > >> > > stay
> > > > > > > >> > > > > on
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is
> > contrary
> > > > to
> > > > > > what
> > > > > > > >> > > > CloudStack
> > > > > > > >> > > > > > > calls
> > > > > > > >> > > > > > > > > > > snpshots.
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots
> > on
> > > > > > storage
> > > > > > > >> and
> > > > > > > >> > > > keeping
> > > > > > > >> > > > > > it
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own
> > > > advantages
> > > > > > > >> > > > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally
> like
> > to
> > > > > have
> > > > > > > both
> > > > > > > >> > ways
> > > > > > > >> > > > of
> > > > > > > >> > > > > > > doing
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This
> > proposal
> > > > > adds
> > > > > > 4
> > > > > > > >> new
> > > > > > > >> > > APIs
> > > > > > > >> > > > to
> > > > > > > >> > > > > > > > create
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend
> > > > > storage.
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys
> think
> > > of
> > > > > this
> > > > > > > >> > > feature? I
> > > > > > > >> > > > > > would
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some
> > > > feedback.
> > > > > I
> > > > > > am
> > > > > > > >> > > working
> > > > > > > >> > > > on
> > > > > > > >> > > > > > > > making
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > > >> > > > > > > > > > > > spec
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but
> > wanted
> > > > to
> > > > > > > have a
> > > > > > > >> > high
> > > > > > > >> > > > > level
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first
> before
> > > > > starting
> > > > > > > to
> > > > > > > >> > work
> > > > > > > >> > > on
> > > > > > > >> > > > > it.
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> --
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack
> > > Developer,
> > > > > > > >> SolidFire
> > > > > > > >> > > Inc.*
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> e:
> > > > mike.tutkowski@solidfire.com
> > > > > > > >> > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the
> > world
> > > > uses
> > > > > > the
> > > > > > > >> > cloud
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> <
> > > > > > > >> > > > > > >
> > http://solidfire.com/solution/overview/?video=play
> > > > > > > >> > > > > > > > >*™
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>> --
> > > > > > > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack
> Developer,
> > > > > > SolidFire
> > > > > > > >> > Inc.*
> > > > > > > >> > > > > > > > > > > > > > >>>>>> e:
> > > mike.tutkowski@solidfire.com
> > > > > > > >> > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > > >> > > > > > > > > > > > > > >>>>>> Advancing the way the world
> > > uses
> > > > > the
> > > > > > > >> cloud
> > > > > > > >> > > > > > > > > > > > > > >>>>>> <
> > > > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > >> > > > > > > >*™*
> > > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>> --
> > > > > > > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack
> Developer,
> > > > > > SolidFire
> > > > > > > >> Inc.*
> > > > > > > >> > > > > > > > > > > > > > >>>>> e:
> > mike.tutkowski@solidfire.com
> > > > > > > >> > <javascript:;>
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > > > >> > > > > > > > > > > > > > >>>>> Advancing the way the world
> > uses
> > > > the
> > > > > > > cloud
> > > > > > > >> > > > > > > > > > > > > > >>>>> <
> > > > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > >> > > > > > > >*™*
> > > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>> --
> > > > > > > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > > > >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer,
> > > > > SolidFire
> > > > > > > >> Inc.*
> > > > > > > >> > > > > > > > > > > > > > >>> e:
> mike.tutkowski@solidfire.com
> > > > > > > >> <javascript:;>
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > > > > > >> > > > > > > > > > > > > > >>> Advancing the way the world
> uses
> > > the
> > > > > > cloud
> > > > > > > >> > > > > > > > > > > > > > >>> <
> > > > > > > >> > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > >> > > > > > >*™*
> > > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > > >> > > > > > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > > --
> > > > > > > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > > > > > >> > > > > > > > > > > > > > > *Senior CloudStack Developer,
> > > > SolidFire
> > > > > > > Inc.*
> > > > > > > >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > >> <javascript:;>
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > > > > > > > > > > o: 303.746.7302
> > > > > > > >> > > > > > > > > > > > > > > Advancing the way the world uses
> > the
> > > > > cloud
> > > > > > > >> > > > > > > > > > > > > > > <
> > > > > > > >> > > http://solidfire.com/solution/overview/?video=play
> > > > > > > >> > > > > >*™*
> > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Find out more about ShapeBlue and
> > our
> > > > > range
> > > > > > of
> > > > > > > >> > > > CloudStack
> > > > > > > >> > > > > > > > > > > > > > related
> > > > > > > >> > > > > > > > > > > > > services:
> > > > > > > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > > > > > >> > > |
> > > > > > > >> > > > > > > > CSForge –
> > > > > > > >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > > > > > > >> > > > > > > http://shapeblue.com/csforge/
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > CloudStack Consulting
> > > > > > > >> > > > > > > > > > > > > > <
> > > > > > http://shapeblue.com/cloudstack-consultancy/
> > > > > > > >
> > > > > > > >> |
> > > > > > > >> > > > > > > > > > > > > CloudStack
> > > > > > > >> > > > > > > > > > > > > > Software Engineering
> > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > >> > > > > > > > > > > > > > <
> > > > > > > >> > > >
> http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > |
> > > > > > > >> > > > > > > > > > > > > > CloudStack Bootcamp Training
> Courses
> > > > > > > >> > > > > > > > > > > > > > <
> > > > > http://shapeblue.com/cloudstack-training/>
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > --
> > > > > > > >> > > > > > > > > > > > *Mike Tutkowski*
> > > > > > > >> > > > > > > > > > > > *Senior CloudStack Developer,
> SolidFire
> > > > Inc.*
> > > > > > > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > <javascript:;>
> > > > > > > >> > > > > <javascript:;>
> > > > > > > >> > > > > > > > > > > > o: 303.746.7302
> > > > > > > >> > > > > > > > > > > > Advancing the way the world uses the
> > cloud
> > > > > > > >> > > > > > > > > > > > <
> > > > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > >> > >*™*
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Find out more about ShapeBlue and our
> > range
> > > of
> > > > > > > >> CloudStack
> > > > > > > >> > > > > related
> > > > > > > >> > > > > > > > > > services:
> > > > > > > >> > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > >> > > > > > > > > > > <
> > > > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > > > >
> > > > > > > >> |
> > > > > > > >> > > > > CSForge –
> > > > > > > >> > > > > > > > rapid
> > > > > > > >> > > > > > > > > > > IaaS deployment framework <
> > > > > > > >> http://shapeblue.com/csforge/
> > > > > > > >> > >
> > > > > > > >> > > > > > > CloudStack
> > > > > > > >> > > > > > > > > > > Consulting <
> > > > > > > >> http://shapeblue.com/cloudstack-consultancy/
> > > > > > > >> > >
> > > > > > > >> > > |
> > > > > > > >> > > > > > > > CloudStack
> > > > > > > >> > > > > > > > > > > Software Engineering
> > > > > > > >> > > > > > > > > > > <
> > > > > > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > >> > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > >> > > > > > > > > > > <
> > > > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > > > >> > >
> > > > > > > >> > > |
> > > > > > > >> > > > > > > > CloudStack
> > > > > > > >> > > > > > > > > > > Bootcamp Training Courses <
> > > > > > > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > Find out more about ShapeBlue and our
> range
> > of
> > > > > > > >> CloudStack
> > > > > > > >> > > > related
> > > > > > > >> > > > > > > > > services:
> > > > > > > >> > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > >> > > > > > > > > > <
> > > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > > >
> > > > > > > |
> > > > > > > >> > > > CSForge –
> > > > > > > >> > > > > > > rapid
> > > > > > > >> > > > > > > > > > IaaS deployment framework <
> > > > > > > >> http://shapeblue.com/csforge/>
> > > > > > > >> > > > > > > > > > CloudStack Consulting <
> > > > > > > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > > > > >> > > > > > > |
> > > > > > > >> > > > > > > > > CloudStack
> > > > > > > >> > > > > > > > > > Software Engineering
> > > > > > > >> > > > > > > > > > <
> > > > > > > http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > >> > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > >> > > > > > > > > > <
> > > > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > > > > > >> > |
> > > > > > > >> > > > > > > CloudStack
> > > > > > > >> > > > > > > > > > Bootcamp Training Courses <
> > > > > > > >> > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > --
> > > > > > > >> > > > > > > *Mike Tutkowski*
> > > > > > > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > <javascript:;>
> > > > > > > >> > > > > > > o: 303.746.7302
> > > > > > > >> > > > > > > Advancing the way the world uses the cloud
> > > > > > > >> > > > > > > <
> > http://solidfire.com/solution/overview/?video=play
> > > > >*™*
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > --
> > > > > > > >> > > > > *Mike Tutkowski*
> > > > > > > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > >> > > > > o: 303.746.7302
> > > > > > > >> > > > > Advancing the way the world uses the cloud
> > > > > > > >> > > > > <http://solidfire.com/solution/overview/?video=play
> > >*™*
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > --
> > > > > > > >> > > > Ian Rae
> > > > > > > >> > > > CEO | PDG
> > > > > > > >> > > > c: 514.944.4008
> > > > > > > >> > > >
> > > > > > > >> > > > CloudOps | Cloud Infrastructure and Networking
> Solutions
> > > > > > > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada |
> > H3J
> > > > 1S6
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > *Mike Tutkowski*
> > > > > > > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > >> > e: mike.tutkowski@solidfire.com
> > > > > > > >> > o: 303.746.7302
> > > > > > > >> > Advancing the way the world uses the cloud
> > > > > > > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Mike Tutkowski*
> > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > > o: 303.746.7302
> > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Mike Tutkowski*
> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > o: 303.746.7302
> > > > > > > Advancing the way the world uses the cloud
> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > Find out more about ShapeBlue and our range of CloudStack related
> > services:
> > > IaaS Cloud Design & Build
> > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> > CloudStack
> > > Software Engineering
> > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > CloudStack Infrastructure Support
> > > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

I agree with Mike's concern about backward compatibility. We can add a
global flag which makes sure that the way volume snapshots work currently
on managed storage (stay on the device) is retained after upgrade. We can
then safely implement the Storage Snapshot API while making the Volume
Snpashot API move the snapshot to Secondary Storage.

Sounds good guys?

-Syed


On Mon, Feb 8, 2016 at 2:56 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> Here's what we have for snapshots for managed storage as of 4.6, Paul:
>
> 1. VM snapshots (no proposed changes to this).
>
> 2. Volume snapshots that do not end up on secondary storage, but rather are
> stored on a SAN (effectively storing snapshots on primary storage).
>
> Pierre-Luc is saying he'd like this for snapshots for managed storage:
>
> A. VM snapshots (no proposed changes to this).
>
> B. Volume snapshots that export to secondary storage.
>
> C. New: Storage snapshots that behave like 2 (above).
>
> I like Pierre-Luc's ideas there, but the problem is backward compatibility.
>
> If customers who were using managed storage with volume snapshots in 4.6
> were getting their snapshots put on a SAN, then in 4.9 - all of a sudden -
> their new snapshots are put on secondary storage (unless they explicitly
> change over to using the new Storage snapshots feature).
>
>
>
> On Mon, Feb 8, 2016 at 12:32 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
>
> > Just to make sure I'm on the same page, are we talking about;
> > https://issues.apache.org/jira/browse/CLOUDSTACK-9278 ?
> >
> > The FS reads (to me) more like 1a + the possibility to export to
> secondary
> > storage if required?
> > Have I understood correctly?
> > I have seen [1a] implemented for VMware by NetApp in their beta
> CloudStack
> > plugin (pleased I can say that without Mike beating me up now). No
> changes
> > to the CloudStack API were required. (nb it didn't export to secondary
> > storage).
> >
> >
> >
> > 1. VM Snapshot (point-in-time hypervisor based snapshots)
> > 1a. SAN assisted VM snapshots (point-in-time hypervisor snapshot takes
> > place on transparently SAN to avoid performance issue in disk chains)
> > 2. SAN Snapshot (Storage Snapshot) - NEW
> > 3. Volume Snapshot (current old/slow transfer to secstorage)
> > 4. Backup - JUST AN IDEAL.
> >
> >
> >
> >
> > [image: ShapeBlue] <http://www.shapeblue.com>
> > Paul Angus
> > VP Technology ,  ShapeBlue
> > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > *+44 7711 418784* <+44%207711%20418784>
> > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > <pa...@cloudyangus>  |  w:
> > *www.shapeblue.com* <http://www.shapeblue.com>
> > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> > Services India LLP is a company incorporated in India and is operated
> under
> > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > company incorporated in Brasil and is operated under license from Shape
> > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
> is
> > a registered trademark.
> > This email and any attachments to it may be confidential and are intended
> > solely for the use of the individual to whom it is addressed. Any views
> or
> > opinions expressed are solely those of the author and do not necessarily
> > represent those of Shape Blue Ltd or related companies. If you are not
> the
> > intended recipient of this email, you must neither take any action based
> > upon its contents, nor copy or show it to anyone. Please contact the
> sender
> > if you believe you have received this email in error.
> >
> >
> > -----Original Message-----
> > From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
> > Sent: Monday, February 8, 2016 7:16 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [Propose][New Feature] Storage Snapshots
> >
> > Correct, Will.
> >
> > That Global Settings would only be for managed storage. Non-managed
> > (traditional) volume snapshots are completely un-impacted by this
> feature.
> >
> > If we need to sometimes keep the snapshots on the SAN and sometimes push
> > them to secondary storage, we'll need a more robust solution than Global
> > Settings, though.
> >
> > On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > Sorry. I missed a bit of context when I responded. The global setting
> > > would be only for the managed storage case, currently being called
> > Storage
> > > Snapshots, and is only to determine if a copy is pushed to secondary
> > > storage right? The global setting would not change the behavior of the
> > > Volume Snapshots right?
> > >
> > > I was referring to the need for Volume Snapshots and Storage Snapshots
> to
> > > exist together. Disregard my comment. I caught up on context after I
> > > posted. My bad...
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > *CloudOps* *| *Cloud Solutions Experts
> > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > w cloudops.com *|* tw @CloudOps_
> > >
> > > On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com
> > > > wrote:
> > >
> > > > Hey Will,
> > > >
> > > > Who's picking the behavior? Is it the cloud provider or the end user?
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <wstevens@cloudops.com
> >
> > > > wrote:
> > > >
> > > > > I don't think a global setting is a good option because we need
> both
> > > > > functionality to be available at the same time and for different
> use
> > > > cases
> > > > > to be able to pick which they choose.
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > w cloudops.com *|* tw @CloudOps_
> > > > >
> > > > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > Now that I re-read your e-mail, it dawned on me: The end-user
> > doesn't
> > > > > care
> > > > > > where the snapshot is.
> > > > > >
> > > > > > If that's true, then we should perhaps control this via Global
> > > Settings
> > > > > or
> > > > > > something.
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > > > mike.tutkowski@solidfire.com> wrote:
> > > > > >
> > > > > > > It's not ideal - true, but it does allow us to be backward
> > > > compatible.
> > > > > > >
> > > > > > > If you have other ideas, though, about how to maintain backward
> > > > > > > compatibility, I'm definitely open to hear them.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > > > syed1.mushtaq@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Mike,
> > > > > > >>
> > > > > > >> Adding a flag to createSnapshot was the first and the most
> > obvious
> > > > > thing
> > > > > > >> that came to our minds. The problem that I had with this was
> > that:
> > > > > > >>
> > > > > > >> 1) I feel it is exposing something to the end user that is
> > > internal
> > > > to
> > > > > > the
> > > > > > >> cloud.
> > > > > > >>
> > > > > > >> 2) We have to follow two different ways of restore/deletion in
> > the
> > > > > same
> > > > > > >> code path depending on where the Snapshot resides which I find
> > > kind
> > > > > of a
> > > > > > >> bad design.
> > > > > > >>
> > > > > > >> But if exposing a archive flag to the end user is acceptable
> > then
> > > we
> > > > > can
> > > > > > >> definitely use this instead of adding the StorageSnapshot API
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> -Syed
> > > > > > >>
> > > > > > >>
> > > > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > > > >> mike.tutkowski@solidfire.com
> > > > > > >> > wrote:
> > > > > > >>
> > > > > > >> > Hi Pierre-Luc,
> > > > > > >> >
> > > > > > >> > My recommendation would be this:
> > > > > > >> >
> > > > > > >> > Add an "archive" flag to the current volume-snapshot API.
> Its
> > > > > default
> > > > > > >> would
> > > > > > >> > be "false" because that would be backward compatible with
> how
> > > 4.6
> > > > > has
> > > > > > >> > volume snapshots implemented (i.e. they stay on the SAN in
> > 4.6,
> > > > 4.7,
> > > > > > and
> > > > > > >> > 4.8).
> > > > > > >> >
> > > > > > >> > If you set archive=true, then we would perform a background
> > > > > migration
> > > > > > of
> > > > > > >> > the snapshot from the SAN to the secondary storage (then
> > delete
> > > > the
> > > > > > SAN
> > > > > > >> > snapshot).
> > > > > > >> >
> > > > > > >> > That archive parameter would only be valid for managed
> > storage.
> > > > > > >> >
> > > > > > >> > Sound reasonable?
> > > > > > >> >
> > > > > > >> > Also, a VM snapshot that includes disks provided by managed
> > > > storage
> > > > > > >> should
> > > > > > >> > work.
> > > > > > >> >
> > > > > > >> > Talk to you later,
> > > > > > >> > Mike
> > > > > > >> >
> > > > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > > > pdion@cloudops.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Mike,
> > > > > > >> > >
> > > > > > >> > > In terms of API's, would you prefer introducing a
> parameter
> > to
> > > > the
> > > > > > >> > existing
> > > > > > >> > > VolumeSnapshot, example: extract={true|false} with a
> > > default
> > > > > > value
> > > > > > >> of
> > > > > > >> > > true which would extract snapshot into the secondary
> > storage,
> > > > > which
> > > > > > >> is
> > > > > > >> > the
> > > > > > >> > > current default behavior. Then for SAN snapshot that
> remain
> > on
> > > > the
> > > > > > >> SAN we
> > > > > > >> > > would just set "extract=false" ? as oppose to create a new
> > > > > > >> > > StorageSnapshot API ?
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Paul,
> > > > > > >> > >
> > > > > > >> > > From what I'm seeing so far, we can't do a VM-snapshot
> when
> > > > using
> > > > > > >> managed
> > > > > > >> > > storage for VM having more than one Volume. For the reason
> > > that
> > > > > > >> snapshot
> > > > > > >> > > are performed outside of the hypervisor awareness and
> > > > > > asynchronously.
> > > > > > >> If
> > > > > > >> > > someone have a way to address that, it would make thinks
> > much
> > > > more
> > > > > > >> > > attractive.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <
> irae@cloudops.com
> > >
> > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > I think a service provider backup scenario is more
> likely
> > to
> > > > > take
> > > > > > >> > > advantage
> > > > > > >> > > > of SAN snapshot. There are a few reasons for this.
> > > Traditional
> > > > > > >> backups
> > > > > > >> > > > involve access to the file system, and there is an
> > > expectation
> > > > > > that
> > > > > > >> > this
> > > > > > >> > > > can be done with reasonably short time frames without
> > > > negatively
> > > > > > >> > > impacting
> > > > > > >> > > > VM performance, and that the backup orchestrator can
> apply
> > > > > various
> > > > > > >> > logic
> > > > > > >> > > > and or transformations to the data (compress, encrypt,
> > > deltas
> > > > > > >> etc...).
> > > > > > >> > > > While it is true that one could apply a backup process
> to
> > a
> > > > > cloud
> > > > > > >> > > snapshot,
> > > > > > >> > > > this would be slow and inefficient requiring the data to
> > be
> > > > > moved
> > > > > > >> > several
> > > > > > >> > > > times and there are some major bottlenecks with cloud
> > > > snapshots.
> > > > > > >> With a
> > > > > > >> > > > cloud snapshot - there seems to be no reasonable
> > expectation
> > > > of
> > > > > > >> being
> > > > > > >> > > able
> > > > > > >> > > > to do differential snapshots (I think this depends on
> the
> > > > > > >> hypervisor)
> > > > > > >> > and
> > > > > > >> > > > if you do differential snapshots this will make file
> > backups
> > > > > from
> > > > > > >> those
> > > > > > >> > > > snapshots even more complicated to orchestrate.
> > > > > > >> > > >
> > > > > > >> > > > Suspect there needs to be a different thread on how to
> > > better
> > > > > > enable
> > > > > > >> > > > backups, as a service. As per Paul's suggestion, but it
> > is a
> > > > > > related
> > > > > > >> > > > workflow so relevant to this discussion.
> > > > > > >> > > >
> > > > > > >> > > > Ian
> > > > > > >> > > >
> > > > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > > > >> > > mike.tutkowski@solidfire.com>
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > To me it sounds like number two and number three are
> > > > different
> > > > > > >> uses
> > > > > > >> > for
> > > > > > >> > > > the
> > > > > > >> > > > > same "thing"(which is totally fine).
> > > > > > >> > > > >
> > > > > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > > > > >> asynchronously, do
> > > > > > >> > > we
> > > > > > >> > > > > see the SSVM as performing the export?
> > > > > > >> > > > >
> > > > > > >> > > > > To be backwards compatible with what we have in 4.6
> and
> > > > later
> > > > > > for
> > > > > > >> > > volume
> > > > > > >> > > > > snapshots for managed storage, I think it might be
> > easier
> > > if
> > > > > we
> > > > > > >> pass
> > > > > > >> > > in a
> > > > > > >> > > > > flag that says whether or not to archive the SAN
> > snapshot
> > > > > > (which,
> > > > > > >> I
> > > > > > >> > > > think,
> > > > > > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > > > > > >> > > > >
> > > > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > > > pdion@cloudops.com
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hi Mike,
> > > > > > >> > > > > >
> > > > > > >> > > > > > The reason behind the creation of a SAN snapshot
> which
> > > is
> > > > > > >> exported
> > > > > > >> > > into
> > > > > > >> > > > > > secondary storage, is because creating a copy of the
> > > .VHD
> > > > > > >> directly
> > > > > > >> > > > would
> > > > > > >> > > > > > impact uptime of the VM as creating that copy take
> > lots
> > > of
> > > > > > time.
> > > > > > >> > Has
> > > > > > >> > > > > oppose
> > > > > > >> > > > > > to a SAN snapshot that is close to instantaneous
> which
> > > can
> > > > > > >> > afterward
> > > > > > >> > > be
> > > > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > > > >> > > > > >
> > > > > > >> > > > > > I would suspect an extracted VolumeSnapshot taken
> > from a
> > > > SAN
> > > > > > >> > snapshot
> > > > > > >> > > > > could
> > > > > > >> > > > > > have is SAN snapshot deleted to avoid duplica and
> > space
> > > > > > >> consumption
> > > > > > >> > > on
> > > > > > >> > > > > the
> > > > > > >> > > > > > Primary Storage side.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > I see 3 definitions in our current discussion
> > regarding
> > > > the
> > > > > > term
> > > > > > >> > > > snapshot
> > > > > > >> > > > > > (these are not official terminology but by own
> > > > > interpretation
> > > > > > of
> > > > > > >> > > them):
> > > > > > >> > > > > >
> > > > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's
> > definition
> > > > of
> > > > > a
> > > > > > >> > > > snapshot):
> > > > > > >> > > > > > it's a volume snapshot at the storage level, point
> in
> > > time
> > > > > of
> > > > > > >> your
> > > > > > >> > > > data.
> > > > > > >> > > > > it
> > > > > > >> > > > > > reside on the primary storage. Useful and efficient
> > for
> > > > > > software
> > > > > > >> > side
> > > > > > >> > > > > > incident.
> > > > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack
> VolumeSnapshot/
> > > > cloud
> > > > > > >> backup
> > > > > > >> > > > aws-S3
> > > > > > >> > > > > > style ): Point in time copy of the Virtual Disk that
> > > > reside
> > > > > > on a
> > > > > > >> > > > > different
> > > > > > >> > > > > > storage array then the original Volume. Facilitate
> > data
> > > > > > >> migration
> > > > > > >> > > > between
> > > > > > >> > > > > > clusters and, in case of primary storage incident,
> > > Volume
> > > > > > >> snapshots
> > > > > > >> > > are
> > > > > > >> > > > > not
> > > > > > >> > > > > > impacted and can be reuse.
> > > > > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data
> > that
> > > > > also
> > > > > > >> > > validate
> > > > > > >> > > > > data
> > > > > > >> > > > > > integrity, provide a storage efficient archiving
> > method
> > > > and
> > > > > an
> > > > > > >> > > > > independent
> > > > > > >> > > > > > way to restore your data in case of an major
> > > > infrastructure
> > > > > > >> > disaster.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Regards,
> > > > > > >> > > > > >
> > > > > > >> > > > > > PL
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > So, let's see if I currently follow the
> > requirements:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > * Augment volume snapshots for managed storage to
> > > > > > >> conditionally
> > > > > > >> > > > export
> > > > > > >> > > > > > data
> > > > > > >> > > > > > > to NFS. The current process of taking a snapshot
> on
> > > the
> > > > > SAN
> > > > > > is
> > > > > > >> > > fine,
> > > > > > >> > > > > but
> > > > > > >> > > > > > > we'd like the option to export the data to NFS, as
> > > well.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Questions:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Once the data has been exported to NFS, do we keep
> > the
> > > > SAN
> > > > > > >> > snapshot
> > > > > > >> > > > or
> > > > > > >> > > > > > > delete it?
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > If we are deleting the SAN snapshot, then why
> don't
> > we
> > > > > just
> > > > > > >> copy
> > > > > > >> > > the
> > > > > > >> > > > > VHD
> > > > > > >> > > > > > > from primary to secondary the way we do today for
> > > > > > non-managed
> > > > > > >> > (i.e.
> > > > > > >> > > > > > > traditional) storage? Why create a SAN snapshot in
> > > that
> > > > > > >> scenario?
> > > > > > >> > > > > Perhaps
> > > > > > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> > > > > secondary
> > > > > > >> > > storage
> > > > > > >> > > > > > > instead of a XenServer host?
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks for the clarification.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > By the way, to me a backup is when you copy data
> > from
> > > > one
> > > > > > >> storage
> > > > > > >> > > > > system
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > another (regardless of features, if any, to
> restore
> > > the
> > > > > data
> > > > > > >> in
> > > > > > >> > the
> > > > > > >> > > > > > > future). A snapshot is a point-in-time view of the
> > > data
> > > > > of a
> > > > > > >> > volume
> > > > > > >> > > > and
> > > > > > >> > > > > > > it's stored on the same storage system as the
> > volume.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > > > > >> > > pdion@cloudops.com
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > <javascript:;>>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > That's fun to see that discussion happening. I
> > 100%
> > > > > agree
> > > > > > >> with
> > > > > > >> > > > Paul's
> > > > > > >> > > > > > > > points of view. VolumeSnapshot are not a backup,
> > > but I
> > > > > do
> > > > > > >> > > consider
> > > > > > >> > > > > them
> > > > > > >> > > > > > > as
> > > > > > >> > > > > > > > a safety vest against Primary Storage failure,
> > > because
> > > > > > >> failure
> > > > > > >> > > > append
> > > > > > >> > > > > > > :-( .
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > The current proposal around snapshots that
> reside
> > on
> > > > the
> > > > > > >> > primary
> > > > > > >> > > > > > storage
> > > > > > >> > > > > > > or
> > > > > > >> > > > > > > > snapshots that end in the Secondary Storage is
> not
> > > to
> > > > > > >> address
> > > > > > >> > > any
> > > > > > >> > > > > kind
> > > > > > >> > > > > > > of
> > > > > > >> > > > > > > > backups requirement because a snapshot is not a
> > > > backup,
> > > > > > >> event
> > > > > > >> > an
> > > > > > >> > > > > > > extracted
> > > > > > >> > > > > > > > VM snapshot.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > The main idea, and again this is for managed
> > > storage;
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side
> > snapshot
> > > > > > >> capability
> > > > > > >> > > for
> > > > > > >> > > > > > fast
> > > > > > >> > > > > > > > response time that support rollback to previous
> > > > > timestamp,
> > > > > > >> > create
> > > > > > >> > > > new
> > > > > > >> > > > > > > > volume and maybe create template.
> > > > > > >> > > > > > > > not required to be a new API if the work is
> > > > already
> > > > > > >> done, I
> > > > > > >> > > > think
> > > > > > >> > > > > > > this
> > > > > > >> > > > > > > > is a different behaviors than the user
> expectation
> > > of
> > > > a
> > > > > > >> > > > > > volume-snapshot.
> > > > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> > > > > behavior
> > > > > > >> that
> > > > > > >> > > > create
> > > > > > >> > > > > > an
> > > > > > >> > > > > > > > extraction of a volume into SecondaryStorage
> which
> > > can
> > > > > be
> > > > > > >> reuse
> > > > > > >> > > to
> > > > > > >> > > > > > > create a
> > > > > > >> > > > > > > > new volume into another Primary Storage. This
> type
> > > of
> > > > > > >> snapshot
> > > > > > >> > > is a
> > > > > > >> > > > > > slow
> > > > > > >> > > > > > > > job since yes it would have to copy the full
> > volume
> > > > size
> > > > > > on
> > > > > > >> the
> > > > > > >> > > > > > Secondary
> > > > > > >> > > > > > > > storage.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > PL
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > >> > > > > > <javascript:;>>
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > I think I share you view on the 'Ideal world'.
> > > > Backup
> > > > > > (via
> > > > > > >> > > Volume
> > > > > > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack.
> > > This
> > > > is
> > > > > > >> > > amplified
> > > > > > >> > > > > > > > especially
> > > > > > >> > > > > > > > > when you have a object storage as your
> secondary
> > > > > storage
> > > > > > >> > > because
> > > > > > >> > > > it
> > > > > > >> > > > > > > > > requires two copies (one to an NFS staging
> area
> > > and
> > > > > from
> > > > > > >> > there
> > > > > > >> > > to
> > > > > > >> > > > > > > object
> > > > > > >> > > > > > > > > storage). And not to mention that all these
> > copies
> > > > are
> > > > > > >> > > consuming
> > > > > > >> > > > > > > > hypervisor
> > > > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> > > > bottleneck
> > > > > as
> > > > > > >> all
> > > > > > >> > > the
> > > > > > >> > > > > > > Network
> > > > > > >> > > > > > > > > and I/O flow through it. So our intention of
> > > > proposing
> > > > > > the
> > > > > > >> > > > "Storage
> > > > > > >> > > > > > > > > Snapshots" is to give a better way of achiving
> > > > > snapshots
> > > > > > >> > while
> > > > > > >> > > > > still
> > > > > > >> > > > > > > > > keeping the original definition of volume
> > > snpashots
> > > > > (ie
> > > > > > >> > upload
> > > > > > >> > > to
> > > > > > >> > > > > sec
> > > > > > >> > > > > > > > > storage).
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > But as Erik pointed out volume snapshots are
> not
> > > > > > backups.
> > > > > > >> > They
> > > > > > >> > > > > don't
> > > > > > >> > > > > > > work
> > > > > > >> > > > > > > > > form multi-disk LVM volume groups and dynamic
> > > > disks. I
> > > > > > am
> > > > > > >> all
> > > > > > >> > > in
> > > > > > >> > > > > for
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > > > better backup solution which handles these use
> > > cases
> > > > > and
> > > > > > >> > > utilizes
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > > storage's advanced features.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > > > > >> > > > > > paul.angus@shapeblue.com <javascript:;>
> > <javascript:;>>
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > In the beginning... there were CloudStack
> > > > snapshots
> > > > > > and
> > > > > > >> > they
> > > > > > >> > > > were
> > > > > > >> > > > > > > > > actually
> > > > > > >> > > > > > > > > > volume snapshots not hypervisor
> point-in-time
> > > > > > snapshots.
> > > > > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > > > > point-in-time
> > > > > > >> > > > > hypervisor
> > > > > > >> > > > > > > > > > snapshots) and we started referring to the
> > > > original
> > > > > > >> > snapshots
> > > > > > >> > > > as
> > > > > > >> > > > > > > volume
> > > > > > >> > > > > > > > > > snapshots.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > CloudStack does not offer 'backups', but
> many
> > > > people
> > > > > > use
> > > > > > >> > > volume
> > > > > > >> > > > > > > > snapshots
> > > > > > >> > > > > > > > > > as backups. However you can't in-place
> restore
> > > > > volume
> > > > > > >> > > snapshots
> > > > > > >> > > > > and
> > > > > > >> > > > > > > if
> > > > > > >> > > > > > > > > you
> > > > > > >> > > > > > > > > > have a VM with multiple volumes, the volume
> > > > > snapshots
> > > > > > >> must
> > > > > > >> > be
> > > > > > >> > > > > done
> > > > > > >> > > > > > in
> > > > > > >> > > > > > > > > > series, meaning that the state across all of
> > the
> > > > > > >> volumes is
> > > > > > >> > > > > > unlikely
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > be
> > > > > > >> > > > > > > > > > consistent.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > 'Actual Backups' would enable all of the
> > restore
> > > > > > options
> > > > > > >> > > which
> > > > > > >> > > > > > users
> > > > > > >> > > > > > > > > might
> > > > > > >> > > > > > > > > > expect as well options as to where they
> might
> > be
> > > > > > >> stored. In
> > > > > > >> > > my
> > > > > > >> > > > > > ideal
> > > > > > >> > > > > > > > > world
> > > > > > >> > > > > > > > > > they would also be able to leverage back-end
> > > > > hardware
> > > > > > >> (such
> > > > > > >> > > as
> > > > > > >> > > > > > > > Solidfire,
> > > > > > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> > > > > Commvault
> > > > > > >> etc
> > > > > > >> > to
> > > > > > >> > > > > > > accelerate
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > process.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > [image: ShapeBlue] <
> http://www.shapeblue.com>
> > > > > > >> > > > > > > > > > Paul Angus
> > > > > > >> > > > > > > > > > VP Technology , ShapeBlue
> > > > > > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > >> > > > > > > > > >
> > > > > > >> > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > |
> > > > > > >> > > > > m:
> > > > > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > >> > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > > > > >> > <javascript:;>
> > > > > > >> > > |
> > > > > > >> > > > > t: @cloudyangus*
> > > > > > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > > >> > > > > <ja...@cloudyangus>
> > > > > > >> > > > > > | w:
> > > > > > >> > > > > > > > > > *www.shapeblue.com* <
> http://www.shapeblue.com
> > >
> > > > > > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> WC2N
> > > > 4HS
> > > > > UK
> > > > > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in
> > > > England
> > > > > &
> > > > > > >> > Wales.
> > > > > > >> > > > > > > ShapeBlue
> > > > > > >> > > > > > > > > > Services India LLP is a company incorporated
> > in
> > > > > India
> > > > > > >> and
> > > > > > >> > is
> > > > > > >> > > > > > operated
> > > > > > >> > > > > > > > > under
> > > > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> Brasil
> > > > > > >> Consultoria
> > > > > > >> > > Ltda
> > > > > > >> > > > > is
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > > > > company incorporated in Brasil and is
> operated
> > > > under
> > > > > > >> > license
> > > > > > >> > > > from
> > > > > > >> > > > > > > Shape
> > > > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > > > registered
> > > > > > >> by
> > > > > > >> > The
> > > > > > >> > > > > > > Republic
> > > > > > >> > > > > > > > of
> > > > > > >> > > > > > > > > > South Africa and is traded under license
> from
> > > > Shape
> > > > > > Blue
> > > > > > >> > Ltd.
> > > > > > >> > > > > > > ShapeBlue
> > > > > > >> > > > > > > > > is
> > > > > > >> > > > > > > > > > a registered trademark.
> > > > > > >> > > > > > > > > > This email and any attachments to it may be
> > > > > > confidential
> > > > > > >> > and
> > > > > > >> > > > are
> > > > > > >> > > > > > > > intended
> > > > > > >> > > > > > > > > > solely for the use of the individual to whom
> > it
> > > is
> > > > > > >> > addressed.
> > > > > > >> > > > Any
> > > > > > >> > > > > > > views
> > > > > > >> > > > > > > > > or
> > > > > > >> > > > > > > > > > opinions expressed are solely those of the
> > > author
> > > > > and
> > > > > > do
> > > > > > >> > not
> > > > > > >> > > > > > > > necessarily
> > > > > > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > > > > > companies.
> > > > > > >> If
> > > > > > >> > > you
> > > > > > >> > > > > are
> > > > > > >> > > > > > > not
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > intended recipient of this email, you must
> > > neither
> > > > > > take
> > > > > > >> any
> > > > > > >> > > > > action
> > > > > > >> > > > > > > > based
> > > > > > >> > > > > > > > > > upon its contents, nor copy or show it to
> > > anyone.
> > > > > > Please
> > > > > > >> > > > contact
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > sender
> > > > > > >> > > > > > > > > > if you believe you have received this email
> in
> > > > > error.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > -----Original Message-----
> > > > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > syed1.mushtaq@gmail.com
> > > > > > >> > > > > <javascript:;> <javascript:;>]
> > > > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > >> > > > > > > > > > To: dev@cloudstack.apache.org
> <javascript:;>
> > > > > > >> > <javascript:;>
> > > > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > > > Snapshots
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Paul,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > When you say actual backups, how would it be
> > > > > different
> > > > > > >> from
> > > > > > >> > > the
> > > > > > >> > > > > > > Volume
> > > > > > >> > > > > > > > > > Snapshots that exist currently. My
> > understanding
> > > > is
> > > > > > that
> > > > > > >> > > > Backups
> > > > > > >> > > > > > end
> > > > > > >> > > > > > > up
> > > > > > >> > > > > > > > > in
> > > > > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > > > > point-in-time
> > > > > > >> > state
> > > > > > >> > > of
> > > > > > >> > > > > > your
> > > > > > >> > > > > > > > > volume
> > > > > > >> > > > > > > > > > which can be restored back correct?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > -Syed
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus
> <
> > > > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> > > <javascript:;>>
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > Hi Syed,
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > As I understand it, the SolidFire plugin
> > will
> > > > > export
> > > > > > >> the
> > > > > > >> > > > > snapshot
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > > > secondary storage if the user requests a
> > > > template
> > > > > > from
> > > > > > >> > the
> > > > > > >> > > > > > snapshot
> > > > > > >> > > > > > > > or
> > > > > > >> > > > > > > > > > > wants to download the snapshot from the
> > cloud.
> > > > > This
> > > > > > >> is a
> > > > > > >> > > > good,
> > > > > > >> > > > > > > > > > > pragmatic approach and yes Mike the
> > SolidFire
> > > > > > storage
> > > > > > >> is
> > > > > > >> > > > super
> > > > > > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays
> > > take
> > > > up
> > > > > > >> next
> > > > > > >> > to
> > > > > > >> > > no
> > > > > > >> > > > > > > space.
> > > > > > >> > > > > > > > > > > BUT I think that we are talking about a
> more
> > > > > general
> > > > > > >> > > purpose
> > > > > > >> > > > > API,
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > > > > other storage systems may not be as
> awesome
> > as
> > > > > > Mike's.
> > > > > > >> > > That's
> > > > > > >> > > > > my
> > > > > > >> > > > > > > > > > > concern. Also, the time to transfer for
> say
> > > 1TB
> > > > to
> > > > > > >> move
> > > > > > >> > > from
> > > > > > >> > > > > > > primary
> > > > > > >> > > > > > > > > > > to sec storage and then create a VM
> template
> > > out
> > > > > of
> > > > > > it
> > > > > > >> > may
> > > > > > >> > > be
> > > > > > >> > > > > too
> > > > > > >> > > > > > > > long
> > > > > > >> > > > > > > > > > for users.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term
> volume
> > > > > > snapshot
> > > > > > >> for
> > > > > > >> > > > > backup,
> > > > > > >> > > > > > > > it's
> > > > > > >> > > > > > > > > > > just that users want to do backups and a
> > > volume
> > > > > > >> snapshot
> > > > > > >> > is
> > > > > > >> > > > the
> > > > > > >> > > > > > > only
> > > > > > >> > > > > > > > > > > type of snapshot that copies the disk
> > > elsewhere
> > > > > and
> > > > > > >> can
> > > > > > >> > be
> > > > > > >> > > > > > > scheduled.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > I'm 'pondering' the implications of
> enabling
> > > > > actual
> > > > > > >> > backups
> > > > > > >> > > > > > > (through
> > > > > > >> > > > > > > > > > > recognised backup providers) and the user
> > > > > > requirements
> > > > > > >> > > around
> > > > > > >> > > > > > them
> > > > > > >> > > > > > > > > > > (particularly restoration use cases) as a
> > > > separate
> > > > > > >> thread
> > > > > > >> > > of
> > > > > > >> > > > > > work.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > [image: ShapeBlue] <
> > http://www.shapeblue.com>
> > > > > Paul
> > > > > > >> Angus
> > > > > > >> > > VP
> > > > > > >> > > > > > > > Technology
> > > > > > >> > > > > > > > > > > , ShapeBlue
> > > > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> 0540*
> > > > > > >> > > > > > > > > > >
> > > > > > >> > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > >> > > > |
> > > > > > >> > > > > m:
> > > > > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com
> <javascript:;>
> > > > > > >> > > <javascript:;> |
> > > > > > >> > > > > t: @cloudyangus*
> > > > > > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > > >> <javascript:;>
> > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > >> > > > > > > > > > > *www.shapeblue.com* <
> > http://www.shapeblue.com
> > > >
> > > > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> > WC2N
> > > > 4HS
> > > > > > UK
> > > > > > >> > Shape
> > > > > > >> > > > > Blue
> > > > > > >> > > > > > > Ltd
> > > > > > >> > > > > > > > > > > is a company incorporated in England &
> > Wales.
> > > > > > >> ShapeBlue
> > > > > > >> > > > > Services
> > > > > > >> > > > > > > > India
> > > > > > >> > > > > > > > > > > LLP is a company incorporated in India and
> > is
> > > > > > operated
> > > > > > >> > > under
> > > > > > >> > > > > > > license
> > > > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> > > > Consultoria
> > > > > > >> Ltda
> > > > > > >> > is
> > > > > > >> > > a
> > > > > > >> > > > > > > company
> > > > > > >> > > > > > > > > > > incorporated in Brasil and is operated
> under
> > > > > license
> > > > > > >> from
> > > > > > >> > > > Shape
> > > > > > >> > > > > > > Blue
> > > > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> > > > registered
> > > > > by
> > > > > > >> The
> > > > > > >> > > > > Republic
> > > > > > >> > > > > > > of
> > > > > > >> > > > > > > > > > > South Africa and is traded under license
> > from
> > > > > Shape
> > > > > > >> Blue
> > > > > > >> > > Ltd.
> > > > > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > > > >> > > > > > > > > > > This email and any attachments to it may
> be
> > > > > > >> confidential
> > > > > > >> > > and
> > > > > > >> > > > > are
> > > > > > >> > > > > > > > > > > intended solely for the use of the
> > individual
> > > to
> > > > > > whom
> > > > > > >> it
> > > > > > >> > is
> > > > > > >> > > > > > > > addressed.
> > > > > > >> > > > > > > > > > > Any views or opinions expressed are solely
> > > those
> > > > > of
> > > > > > >> the
> > > > > > >> > > > author
> > > > > > >> > > > > > and
> > > > > > >> > > > > > > do
> > > > > > >> > > > > > > > > > > not necessarily represent those of Shape
> > Blue
> > > > Ltd
> > > > > or
> > > > > > >> > > related
> > > > > > >> > > > > > > > > > > companies. If you are not the intended
> > > recipient
> > > > > of
> > > > > > >> this
> > > > > > >> > > > email,
> > > > > > >> > > > > > you
> > > > > > >> > > > > > > > > > > must neither take any action based upon
> its
> > > > > > contents,
> > > > > > >> nor
> > > > > > >> > > > copy
> > > > > > >> > > > > or
> > > > > > >> > > > > > > > show
> > > > > > >> > > > > > > > > > > it to anyone. Please contact the sender if
> > you
> > > > > > believe
> > > > > > >> > you
> > > > > > >> > > > have
> > > > > > >> > > > > > > > > received
> > > > > > >> > > > > > > > > > this email in error.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > -----Original Message-----
> > > > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > syed1.mushtaq@gmail.com
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > <javascript:;>]
> > > > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org
> > <javascript:;>
> > > > > > >> > > <javascript:;>
> > > > > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature]
> Storage
> > > > > > Snapshots
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > I think the terminology confusion comes
> from
> > > AWS
> > > > > > where
> > > > > > >> > they
> > > > > > >> > > > do
> > > > > > >> > > > > > EBS
> > > > > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack
> > sort
> > > of
> > > > > > >> followed
> > > > > > >> > > > that.
> > > > > > >> > > > > > And
> > > > > > >> > > > > > > > as
> > > > > > >> > > > > > > > > > > an end user who is oblivious to the
> > internals
> > > of
> > > > > my
> > > > > > >> > > provider,
> > > > > > >> > > > > my
> > > > > > >> > > > > > > > > > > expectation would be something similar to
> > what
> > > > AWS
> > > > > > as
> > > > > > >> > that
> > > > > > >> > > is
> > > > > > >> > > > > my
> > > > > > >> > > > > > > > > > > biggest reference point.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > To your point Mike, I agree that a Primary
> > > > Storage
> > > > > > >> > failure
> > > > > > >> > > on
> > > > > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> > > > motivations
> > > > > > >> for us
> > > > > > >> > > to
> > > > > > >> > > > > push
> > > > > > >> > > > > > > > data
> > > > > > >> > > > > > > > > > > to secondary storage. Primary storage
> > (atleast
> > > > for
> > > > > > us)
> > > > > > >> > > costs
> > > > > > >> > > > > > > around 3
> > > > > > >> > > > > > > > > > > times as much as secondary storage and
> > > snapshots
> > > > > on
> > > > > > >> > primary
> > > > > > >> > > > > > storage
> > > > > > >> > > > > > > > > > > are rarely used (especially for some of
> our
> > > > > > customers
> > > > > > >> who
> > > > > > >> > > do
> > > > > > >> > > > > > daily
> > > > > > >> > > > > > > > > > backups).
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike
> > > Tutkowski
> > > > <
> > > > > > >> > > > > > > > > > > mike.tutkowski@solidfire.com
> <javascript:;>
> > > > > > >> > > <javascript:;>>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Some of the weirdness is around
> > terminology.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > For most systems I've worked on, a
> > snapshot
> > > > and
> > > > > a
> > > > > > >> > backup
> > > > > > >> > > > are
> > > > > > >> > > > > > two
> > > > > > >> > > > > > > > > > > > completely different things (but
> > CloudStack
> > > > has
> > > > > > >> > > > traditionally
> > > > > > >> > > > > > > used
> > > > > > >> > > > > > > > > > > > the term "volume snapshot" to mean
> > backup).
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here
> and
> > > say,
> > > > > > >> though,
> > > > > > >> > > that
> > > > > > >> > > > > if
> > > > > > >> > > > > > > your
> > > > > > >> > > > > > > > > > > > primary storage is running on SolidFire
> > that
> > > > it
> > > > > is
> > > > > > >> > > unlikely
> > > > > > >> > > > > > > you'll
> > > > > > >> > > > > > > > > > > > encounter an issue where your primary
> > > storage
> > > > > goes
> > > > > > >> > > offline
> > > > > > >> > > > > (and
> > > > > > >> > > > > > > > > > > > you'll even maintain your performance
> > > > guarantees
> > > > > > >> during
> > > > > > >> > > > > failure
> > > > > > >> > > > > > > > > > > > scenarios and upgrades, as well). That
> > being
> > > > the
> > > > > > >> case,
> > > > > > >> > it
> > > > > > >> > > > is
> > > > > > >> > > > > > less
> > > > > > >> > > > > > > > > > > > useful to require a backup to Swift (but
> > > it's
> > > > > > >> perfectly
> > > > > > >> > > OK
> > > > > > >> > > > if
> > > > > > >> > > > > > > > that's
> > > > > > >> > > > > > > > > > > > what we want to do
> > > > > > >> > > > > > > > > > > here).
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed
> > Mushtaq
> > > > > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > > > > > >> > <javascript:;>>
> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Paul,
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > I believe with the current
> > implementation
> > > of
> > > > > > >> > Snapshots
> > > > > > >> > > on
> > > > > > >> > > > > > > managed
> > > > > > >> > > > > > > > > > > > > storage
> > > > > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never
> > > exported
> > > > > to
> > > > > > >> the
> > > > > > >> > > > > secondary
> > > > > > >> > > > > > > > > > storage.
> > > > > > >> > > > > > > > > > > > > While this solves the problem of
> having
> > > > > > snapshots
> > > > > > >> > > taking
> > > > > > >> > > > > > > forever
> > > > > > >> > > > > > > > > > > > > to get to sec storage, this leaves us
> > > with a
> > > > > > >> > > > > > > > > > > > huge
> > > > > > >> > > > > > > > > > > > > liability if our primary storage goes
> > > down.
> > > > We
> > > > > > see
> > > > > > >> > > > > snapshots
> > > > > > >> > > > > > as
> > > > > > >> > > > > > > > > > > > > our recovery path because we store
> them
> > in
> > > > > Swift
> > > > > > >> > which
> > > > > > >> > > is
> > > > > > >> > > > > > > > reliable
> > > > > > >> > > > > > > > > > > > > and resilient to failures.
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > With Storage snpashots our goal is to
> > have
> > > > > > Volume
> > > > > > >> > > > snapshots
> > > > > > >> > > > > > > > always
> > > > > > >> > > > > > > > > > > > > backed up to secondary storage and
> > Storage
> > > > > > >> Snapshots
> > > > > > >> > > stay
> > > > > > >> > > > > on
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > > > > primary
> > > > > > >> > > > > > > > > > > > storage.
> > > > > > >> > > > > > > > > > > > > A provider could potentially mix both
> > > these
> > > > > and
> > > > > > >> solve
> > > > > > >> > > the
> > > > > > >> > > > > > > problem
> > > > > > >> > > > > > > > > > > > > that you mentioned where you want to
> > meet
> > > > > user's
> > > > > > >> > > > > expectation
> > > > > > >> > > > > > > of a
> > > > > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage)
> > while
> > > > > having
> > > > > > >> an
> > > > > > >> > > > ability
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the
> > > > device)
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > > > > -Syed
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul
> > Angus
> > > > > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com
> > <javascript:;>
> > > > > > >> > > <javascript:;>>
> > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > HI guys,
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Could someone point me to the Jira
> bug
> > > of
> > > > FS
> > > > > > for
> > > > > > >> > the
> > > > > > >> > > > > > > > > > > > > > SAN-snapshot
> > > > > > >> > > > > > > > > > > > feature
> > > > > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > From my discussions with users and
> > > > operators
> > > > > > >> around
> > > > > > >> > > > > > snapshots
> > > > > > >> > > > > > > > > > > > > > I'd make
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > following observations:
> > > > > > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups
> > > (both
> > > > > > >> long-term
> > > > > > >> > > and
> > > > > > >> > > > > > short
> > > > > > >> > > > > > > > > > > > > > term)
> > > > > > >> > > > > > > > > > > > with
> > > > > > >> > > > > > > > > > > > > > the expectation that they can use
> them
> > > for
> > > > > > >> recovery
> > > > > > >> > > if
> > > > > > >> > > > > > > > required.
> > > > > > >> > > > > > > > > > > > > > b. operators fall back to snapshots
> if
> > > > > > something
> > > > > > >> > has
> > > > > > >> > > > gone
> > > > > > >> > > > > > > wrong
> > > > > > >> > > > > > > > > > > > > > with primary storage.
> > > > > > >> > > > > > > > > > > > > > c. users sometimes want to be able
> to
> > > > export
> > > > > > >> > > snapshots
> > > > > > >> > > > as
> > > > > > >> > > > > > > well
> > > > > > >> > > > > > > > > > > > > > as
> > > > > > >> > > > > > > > > > > > create
> > > > > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > > > > >> > > > > > > > > > > > > > d. snapshots are a currently a
> massive
> > > > pain
> > > > > > for
> > > > > > >> > > > > operators,
> > > > > > >> > > > > > I
> > > > > > >> > > > > > > > > > > > > > know at
> > > > > > >> > > > > > > > > > > > > least
> > > > > > >> > > > > > > > > > > > > > one public cloud who have snapshots
> > > which
> > > > > > take 2
> > > > > > >> > days
> > > > > > >> > > > to
> > > > > > >> > > > > > > > > complete.
> > > > > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be
> > used
> > > > for
> > > > > > >> > multiple
> > > > > > >> > > > LVM
> > > > > > >> > > > > > > > disks.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > I think the process Mike has used in
> > the
> > > > > > >> SolidFire
> > > > > > >> > > > plugin
> > > > > > >> > > > > > > (only
> > > > > > >> > > > > > > > > > > > > > moving
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > disk image to secondary storage when
> > you
> > > > > > >> absolutely
> > > > > > >> > > > have
> > > > > > >> > > > > > to)
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > > > > > > > a very
> > > > > > >> > > > > > > > > > > > > good
> > > > > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder
> what
> > > > > problems
> > > > > > >> an
> > > > > > >> > > > > operator
> > > > > > >> > > > > > > > might
> > > > > > >> > > > > > > > > > > > > experience
> > > > > > >> > > > > > > > > > > > > > if they have an issue with a given
> > > primary
> > > > > > >> storage
> > > > > > >> > > pool
> > > > > > >> > > > > in
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > > > > cluster.
> > > > > > >> > > > > > > > > > > > (I
> > > > > > >> > > > > > > > > > > > > > know that that is REALLY unlikely in
> > the
> > > > > > >> SolidFire
> > > > > > >> > > case
> > > > > > >> > > > > > Mike
> > > > > > >> > > > > > > :)
> > > > > > >> > > > > > > > > > > > > > ) And
> > > > > > >> > > > > > > > > > > > if
> > > > > > >> > > > > > > > > > > > > > the transfer from primary to
> secondary
> > > is
> > > > > > slow,
> > > > > > >> the
> > > > > > >> > > > time
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > > > > > > > being able
> > > > > > >> > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > create a template or export the
> volume
> > > > will
> > > > > be
> > > > > > >> > slow.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > So for me the issue is around making
> > > sure
> > > > > that
> > > > > > >> the
> > > > > > >> > > end
> > > > > > >> > > > > > users
> > > > > > >> > > > > > > > > > > > expectations
> > > > > > >> > > > > > > > > > > > > > are met (while improving the
> > > > > speed/efficiency
> > > > > > of
> > > > > > >> > the
> > > > > > >> > > > back
> > > > > > >> > > > > > > end)
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > > > > http://www.shapeblue.com>
> > > > > > >> Paul
> > > > > > >> > > > Angus
> > > > > > >> > > > > > VP
> > > > > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > > > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203
> 603
> > > > 0540*
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > >
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > >> > > > > > > |
> > > > > > >> > > > > > > > m:
> > > > > > >> > > > > > > > > > > > > > *+44 7711 418784*
> > <+44%207711%20418784>
> > > > > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;> | t:
> > > > > > >> > > > > > @cloudyangus*
> > > > > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com
> > > <javascript:;>
> > > > > > >> > > > <javascript:;>
> > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > > > > http://www.shapeblue.com
> > > > > > >
> > > > > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > > London
> > > > > WC2N
> > > > > > >> 4HS
> > > > > > >> > UK
> > > > > > >> > > > > Shape
> > > > > > >> > > > > > > > Blue
> > > > > > >> > > > > > > > > > > > > > Ltd is a company incorporated in
> > > England &
> > > > > > >> Wales.
> > > > > > >> > > > > ShapeBlue
> > > > > > >> > > > > > > > > > > > > > Services India LLP is a company
> > > > incorporated
> > > > > > in
> > > > > > >> > India
> > > > > > >> > > > and
> > > > > > >> > > > > > is
> > > > > > >> > > > > > > > > > > > > > operated
> > > > > > >> > > > > > > > > > > > > under
> > > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape
> > Blue
> > > > > Brasil
> > > > > > >> > > > > Consultoria
> > > > > > >> > > > > > > Ltda
> > > > > > >> > > > > > > > > > > > > > is a company incorporated in Brasil
> > and
> > > is
> > > > > > >> operated
> > > > > > >> > > > under
> > > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd.
> ShapeBlue
> > > SA
> > > > > Pty
> > > > > > >> Ltd
> > > > > > >> > is
> > > > > > >> > > a
> > > > > > >> > > > > > > company
> > > > > > >> > > > > > > > > > > > > > registered by The Republic
> > > > > > >> > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > South Africa and is traded under
> > license
> > > > > from
> > > > > > >> Shape
> > > > > > >> > > > Blue
> > > > > > >> > > > > > Ltd.
> > > > > > >> > > > > > > > > > > > > > ShapeBlue
> > > > > > >> > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > a registered trademark.
> > > > > > >> > > > > > > > > > > > > > This email and any attachments to it
> > may
> > > > be
> > > > > > >> > > > confidential
> > > > > > >> > > > > > and
> > > > > > >> > > > > > > > are
> > > > > > >> > > > > > > > > > > > intended
> > > > > > >> > > > > > > > > > > > > > solely for the use of the individual
> > to
> > > > whom
> > > > > > it
> > > > > > >> is
> > > > > > >> > > > > > addressed.
> > > > > > >> > > > > > > > > > > > > > Any views
> > > > > > >> > > > > > > > > > > > > or
> > > > > > >> > > > > > > > > > > > > > opinions expressed are solely those
> of
> > > the
> > > > > > >> author
> > > > > > >> > and
> > > > > > >> > > > do
> > > > > > >> > > > > > not
> > > > > > >> > > > > > > > > > > > necessarily
> > > > > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or
> > > > related
> > > > > > >> > > companies.
> > > > > > >> > > > > If
> > > > > > >> > > > > > > you
> > > > > > >> > > > > > > > > > > > > > are not
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > intended recipient of this email,
> you
> > > must
> > > > > > >> neither
> > > > > > >> > > take
> > > > > > >> > > > > any
> > > > > > >> > > > > > > > > > > > > > action
> > > > > > >> > > > > > > > > > > > based
> > > > > > >> > > > > > > > > > > > > > upon its contents, nor copy or show
> it
> > > to
> > > > > > >> anyone.
> > > > > > >> > > > Please
> > > > > > >> > > > > > > > contact
> > > > > > >> > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > sender
> > > > > > >> > > > > > > > > > > > > > if you believe you have received
> this
> > > > email
> > > > > in
> > > > > > >> > error.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > -----Original Message-----
> > > > > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > > > > >> pdion@cloudops.com
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > <javascript:;>]
> > > > > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56
> > PM
> > > > > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature]
> > > > Storage
> > > > > > >> > Snapshots
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Hi Mike,
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > > > > >> StorageSnapshot
> > > > > > >> > > for
> > > > > > >> > > > > > > managed
> > > > > > >> > > > > > > > > > > > > > storage
> > > > > > >> > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > because the VolumeSnapshot default,
> or
> > > > > > expected,
> > > > > > >> > > > behavior
> > > > > > >> > > > > > is
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> > > > > Storage.
> > > > > > >> So a
> > > > > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > > > > >> > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > snapshot that remain on the managed
> > > > storage
> > > > > > >> > > appliance.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Quickly looking at the API doc and I
> > > don't
> > > > > > see a
> > > > > > >> > > strong
> > > > > > >> > > > > > > > > > > > > > requirement for volume snapshots to
> be
> > > > moved
> > > > > > >> into
> > > > > > >> > > > > secondary
> > > > > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot
> API
> > > is
> > > > > not
> > > > > > >> > useful,
> > > > > > >> > > > but
> > > > > > >> > > > > > > both
> > > > > > >> > > > > > > > > > > > > > use cases are required. A snapshot
> > that
> > > > > remain
> > > > > > >> on
> > > > > > >> > the
> > > > > > >> > > > > > managed
> > > > > > >> > > > > > > > > > > > > > storage, and another type of
> > > > > > >> > > > > > > > > > > > snapshot
> > > > > > >> > > > > > > > > > > > > > that end up into the secondary
> > storage.
> > > > > Since
> > > > > > >> > you've
> > > > > > >> > > > > done a
> > > > > > >> > > > > > > lot
> > > > > > >> > > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > work,
> > > > > > >> > > > > > > > > > > > > > might easier to just add a parameter
> > to
> > > > the
> > > > > > >> current
> > > > > > >> > > > > > snapshot
> > > > > > >> > > > > > > > API
> > > > > > >> > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > > trigger an extraction of the storage
> > > > > snapshot
> > > > > > >> into
> > > > > > >> > > the
> > > > > > >> > > > > > > > secondary
> > > > > > >> > > > > > > > > > > > storage?
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > PL
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > > > > > Tutkowski <
> > > > > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > I think that all sounds reasonable
> > > then
> > > > -
> > > > > > >> thanks!
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM,
> Syed
> > > > > > Mushtaq <
> > > > > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > >> <javascript:;>>
> > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >> You are correct Mike in terms of
> > the
> > > > > > >> > requirements.
> > > > > > >> > > > One
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > our
> > > > > > >> > > > > > > > > > > > earlier
> > > > > > >> > > > > > > > > > > > > > >> iterations on this was to have an
> > > > > argument
> > > > > > to
> > > > > > >> > the
> > > > > > >> > > > > create
> > > > > > >> > > > > > > > > > > > > > >> snapshot
> > > > > > >> > > > > > > > > > > > API
> > > > > > >> > > > > > > > > > > > > > >> which decides whether to backup
> the
> > > > > volume
> > > > > > to
> > > > > > >> > sec
> > > > > > >> > > > > > storage
> > > > > > >> > > > > > > > but
> > > > > > >> > > > > > > > > > > > > > >> we realized it would make
> > management
> > > of
> > > > > > >> > snapshots
> > > > > > >> > > > > quite
> > > > > > >> > > > > > > > messy
> > > > > > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > > > > >> > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> > > > > Tutkowski
> > > > > > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>>
> > > > > > >> > > > > > > > > > > > > > >> wrote:
> > > > > > >> > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > >>> Hi,
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> Just to make sure I understand
> all
> > > the
> > > > > > >> > > requirements
> > > > > > >> > > > > > here:
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 1) This relates only to managed
> > > > storage
> > > > > > (1:1
> > > > > > >> > > > mapping
> > > > > > >> > > > > > > > between
> > > > > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> > > > > volume).
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > > > > (introduced
> > > > > > >> in
> > > > > > >> > > 4.6)
> > > > > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> > > > snapshot
> > > > > on
> > > > > > >> the
> > > > > > >> > > SAN,
> > > > > > >> > > > > and
> > > > > > >> > > > > > > > > > > > > > >>> extend it via a config option
> (or
> > > > > > >> > > > > > > > > > > > > > >>> something) to not only take the
> > SAN
> > > > > > >> snapshot,
> > > > > > >> > but
> > > > > > >> > > > to
> > > > > > >> > > > > > copy
> > > > > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer
> > only)
> > > to
> > > > > > NFS.
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always
> > taken.
> > > > > It's
> > > > > > >> the
> > > > > > >> > > > backup
> > > > > > >> > > > > to
> > > > > > >> > > > > > > NFS
> > > > > > >> > > > > > > > > > > > > > >>> that is optional.
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 4) Templates can be created from
> > the
> > > > > > >> snapshot
> > > > > > >> > > > that's
> > > > > > >> > > > > on
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be
> > created
> > > > > from
> > > > > > >> the
> > > > > > >> > > > > snapshot
> > > > > > >> > > > > > > > > > > > > > >>> that's on
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>> SAN (already works as long as
> the
> > > new
> > > > > > >> > CloudStack
> > > > > > >> > > > > volume
> > > > > > >> > > > > > > > ends
> > > > > > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> Would we have a need for a
> storage
> > > > > > snapshot
> > > > > > >> API
> > > > > > >> > > > then
> > > > > > >> > > > > or
> > > > > > >> > > > > > > > > > > > > > >>> would that just be the standard
> > > volume
> > > > > > >> snapshot
> > > > > > >> > > > > without
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>> backup to
> > > > > > >> > > > > > > > > > > NFS?
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> Thanks!
> > > > > > >> > > > > > > > > > > > > > >>> Mike
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM,
> > Syed
> > > > > > Mushtaq
> > > > > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>>
> > > > > > >> > > > > > > > > > > > > > >>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > > > > functionalities
> > > > > > >> > > > > (snapshot
> > > > > > >> > > > > > on
> > > > > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > > > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because
> > Ideally,
> > > we
> > > > > > would
> > > > > > >> > like
> > > > > > >> > > > to
> > > > > > >> > > > > > have
> > > > > > >> > > > > > > > > both.
> > > > > > >> > > > > > > > > > > > > > >>>> For example, some of our
> > customers
> > > > want
> > > > > > to
> > > > > > >> > > > implement
> > > > > > >> > > > > > > their
> > > > > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> > > > encryption
> > > > > > to
> > > > > > >> > their
> > > > > > >> > > > > > backups
> > > > > > >> > > > > > > > > > > > > > >>>> which is a perfect use case for
> > > > Storage
> > > > > > >> > Snapshot
> > > > > > >> > > > > while
> > > > > > >> > > > > > > our
> > > > > > >> > > > > > > > > > > > > > >>>> other customers will still keep
> > > using
> > > > > the
> > > > > > >> > > standard
> > > > > > >> > > > > > > volume
> > > > > > >> > > > > > > > > > > snapshot.
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>> To keep things backward
> > compatible,
> > > > we
> > > > > > can
> > > > > > >> > add a
> > > > > > >> > > > > > setting
> > > > > > >> > > > > > > > > > > > > > >>>> which
> > > > > > >> > > > > > > > > > > > says
> > > > > > >> > > > > > > > > > > > > > >>>> to not upload on secondary
> > storage,
> > > > > > >> because,
> > > > > > >> > > after
> > > > > > >> > > > > > all,
> > > > > > >> > > > > > > > you
> > > > > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first
> > > when
> > > > > > doing
> > > > > > >> a
> > > > > > >> > > > Volume
> > > > > > >> > > > > > > > > Snapshot.
> > > > > > >> > > > > > > > > > > > > > >>>> You could stop the process
> there
> > > and
> > > > > not
> > > > > > do
> > > > > > >> > the
> > > > > > >> > > > > > upload.
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>> What do you think about this
> > > > approach?
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > > > > >> > > > > > > > > > > > > > >>>> -Syed
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM,
> > > Mike
> > > > > > >> > Tutkowski <
> > > > > > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking
> out
> > > > load
> > > > > > >> here,
> > > > > > >> > but
> > > > > > >> > > > if
> > > > > > >> > > > > a
> > > > > > >> > > > > > > > given
> > > > > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't
> > actually
> > > > need
> > > > > > to
> > > > > > >> > > provide
> > > > > > >> > > > > > both
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > ability
> > > > > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and
> > export
> > > it
> > > > > to
> > > > > > >> NFS
> > > > > > >> > (if
> > > > > > >> > > > > just
> > > > > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK),
> > then
> > > > we
> > > > > > >> might
> > > > > > >> > be
> > > > > > >> > > > able
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > get
> > > > > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > > > > >> > > > > > > > > > > > API
> > > > > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a
> new
> > > > > custom
> > > > > > >> > > snapshot
> > > > > > >> > > > > > > strategy
> > > > > > >> > > > > > > > > > > > > > >>>>> and
> > > > > > >> > > > > > > > > > > > data
> > > > > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the
> > case
> > > > > where
> > > > > > >> the
> > > > > > >> > > > > > CloudStack
> > > > > > >> > > > > > > > > > > > > > >>>>> cloud
> > > > > > >> > > > > > > > > > > > does
> > > > > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > > > > >> exported-to-NFS
> > > > > > >> > > > > backup.
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> In other words, the "default"
> > > > behavior
> > > > > > >> would
> > > > > > >> > be
> > > > > > >> > > > to
> > > > > > >> > > > > > use
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data
> > motion
> > > > > > strategy
> > > > > > >> > that
> > > > > > >> > > > we
> > > > > > >> > > > > > > > already
> > > > > > >> > > > > > > > > > > > > > >>>>> have (the one that only takes
> a
> > > SAN
> > > > > > >> > snapshot).
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud,
> > however,
> > > > > wants
> > > > > > >> to
> > > > > > >> > > take
> > > > > > >> > > > a
> > > > > > >> > > > > > SAN
> > > > > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data
> > > exported
> > > > to
> > > > > > >> NFS,
> > > > > > >> > > then
> > > > > > >> > > > we
> > > > > > >> > > > > > > could
> > > > > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing
> > config
> > > > > file
> > > > > > to
> > > > > > >> > make
> > > > > > >> > > > use
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > a
> > > > > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data
> > > > motion
> > > > > > >> > strategy
> > > > > > >> > > > that
> > > > > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > > > > >> > > > > > > > > > > > > > activities.
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is
> > > still
> > > > > the
> > > > > > >> > default
> > > > > > >> > > > for
> > > > > > >> > > > > > > > users,
> > > > > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can
> change
> > > > this
> > > > > > >> > behavior
> > > > > > >> > > > via
> > > > > > >> > > > > > > > > > > configuration.
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55
> AM,
> > > > Mike
> > > > > > >> > > Tutkowski <
> > > > > > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need
> to
> > > > come
> > > > > up
> > > > > > >> > with a
> > > > > > >> > > > > > viable
> > > > > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some
> reasonable
> > > way
> > > > > for
> > > > > > >> them
> > > > > > >> > > to
> > > > > > >> > > > > move
> > > > > > >> > > > > > > > from
> > > > > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way
> (and
> > > > some
> > > > > > >> obvious
> > > > > > >> > > way
> > > > > > >> > > > > > that
> > > > > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > > > > >> > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > do this).
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45
> > AM,
> > > > Syed
> > > > > > >> > Mushtaq
> > > > > > >> > > <
> > > > > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about
> the
> > > > > upgrade
> > > > > > >> path
> > > > > > >> > > > > however,
> > > > > > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6
> > and
> > > > are
> > > > > > on a
> > > > > > >> > > > managed
> > > > > > >> > > > > > > > storage
> > > > > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the
> same
> > > > > > >> functionality
> > > > > > >> > > > with
> > > > > > >> > > > > > > Volume
> > > > > > >> > > > > > > > > > > Snapshots.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43
> > PM,
> > > > Syed
> > > > > > >> > Mushtaq
> > > > > > >> > > <
> > > > > > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> So if I understand
> correctly,
> > > > > > currently
> > > > > > >> > > > taking a
> > > > > > >> > > > > > > > Volume
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a
> > > > managed
> > > > > > >> storage
> > > > > > >> > > > keeps
> > > > > > >> > > > > > it
> > > > > > >> > > > > > > on
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a
> part
> > of
> > > > > this
> > > > > > >> > > feature,
> > > > > > >> > > > we
> > > > > > >> > > > > > can
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed
> > > > storage
> > > > > > are
> > > > > > >> > > > uploaded
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This
> would
> > > > make
> > > > > > the
> > > > > > >> > > Volume
> > > > > > >> > > > > > > Snapshot
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same
> > > > regardless
> > > > > of
> > > > > > >> the
> > > > > > >> > > > > storage
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for
> > utilizing
> > > > the
> > > > > > >> > > efficient
> > > > > > >> > > > > > > backend
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> storage
> > > > > > >> > > > > > > > > > > > > > capabilities, we can use the new
> > storage
> > > > > > >> snapshots
> > > > > > >> > > API.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36
> > PM,
> > > > > Mike
> > > > > > >> > > > Tutkowski <
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> mike.tutkowski@solidfire.com
> > > > > > >> > <javascript:;>
> > > > > > >> > > > > <javascript:;>>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we
> need
> > > to
> > > > > > have a
> > > > > > >> > plan
> > > > > > >> > > > to
> > > > > > >> > > > > > deal
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we
> > already
> > > > > have a
> > > > > > >> > > feature
> > > > > > >> > > > > (in
> > > > > > >> > > > > > > 4.6
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> and
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to
> > use
> > > > the
> > > > > > >> > existing
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to
> > > create a
> > > > > > >> volume
> > > > > > >> > > > > snapshot
> > > > > > >> > > > > > > (for
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a
> > > > backend
> > > > > > SAN
> > > > > > >> > > > (using a
> > > > > > >> > > > > > > > custom
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a
> > custom
> > > > > data
> > > > > > >> > motion
> > > > > > >> > > > > > > strategy).
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in,
> > then
> > > > how
> > > > > > >> should
> > > > > > >> > > the
> > > > > > >> > > > > > > original
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in
> > 4.6
> > > > and
> > > > > > >> later)
> > > > > > >> > > be
> > > > > > >> > > > > > > changed?
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > > > > > >> > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support
> > > > > customers
> > > > > > >> who
> > > > > > >> > > are
> > > > > > >> > > > > > > already
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> using
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot
> API
> > > to
> > > > > take
> > > > > > >> > > > snapshots
> > > > > > >> > > > > > on a
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > > > > > >> > > > > > > > > > > > > SAN?
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at
> 11:27
> > > AM,
> > > > > > Will
> > > > > > >> > > > Stevens <
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to
> create
> > a
> > > > > > Template
> > > > > > >> > > from a
> > > > > > >> > > > > > > > > > > > StorageSnapshot?
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template
> > be
> > > > > stored
> > > > > > >> in
> > > > > > >> > the
> > > > > > >> > > > > > > secondary
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal
> > templates
> > > > or
> > > > > > will
> > > > > > >> > that
> > > > > > >> > > > be
> > > > > > >> > > > > > > > handled
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > >> > > > > > > > > > > > > > vendor side?
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud
> > > Solutions
> > > > > > >> Experts
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal
> > *|*
> > > > > Quebec
> > > > > > >> *|*
> > > > > > >> > > H3J
> > > > > > >> > > > > 1S6
> > > > > > >> > > > > > w
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw
> > > @CloudOps_
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at
> 1:22
> > > PM,
> > > > > > Syed
> > > > > > >> > > > Mushtaq <
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at
> > 1:19
> > > > PM,
> > > > > > Will
> > > > > > >> > > > Stevens
> > > > > > >> > > > > <
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the
> > > > Design
> > > > > > >> Spec in
> > > > > > >> > > the
> > > > > > >> > > > > > Jira
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was
> not
> > > > clear
> > > > > > in
> > > > > > >> the
> > > > > > >> > > > > > 'mention'
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it
> shows
> > > as a
> > > > > > page
> > > > > > >> > 'you
> > > > > > >> > > do
> > > > > > >> > > > > not
> > > > > > >> > > > > > > > have
> > > > > > >> > > > > > > > > > > permission to'.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud
> > > > Solutions
> > > > > > >> Experts
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|*
> Montreal
> > > *|*
> > > > > > Quebec
> > > > > > >> > *|*
> > > > > > >> > > > H3J
> > > > > > >> > > > > > 1S6
> > > > > > >> > > > > > > w
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw
> > > > @CloudOps_
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at
> > 1:02
> > > > PM,
> > > > > > >> Syed
> > > > > > >> > > Ahmed
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > >
> > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a
> new
> > > set
> > > > > of
> > > > > > >> APIs
> > > > > > >> > to
> > > > > > >> > > > do
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed
> > > storage
> > > > > > >> backends
> > > > > > >> > > > like
> > > > > > >> > > > > > > > > SolidFire.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current
> > > managed
> > > > > > >> storage
> > > > > > >> > > stay
> > > > > > >> > > > > on
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is
> contrary
> > > to
> > > > > what
> > > > > > >> > > > CloudStack
> > > > > > >> > > > > > > calls
> > > > > > >> > > > > > > > > > > snpshots.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots
> on
> > > > > storage
> > > > > > >> and
> > > > > > >> > > > keeping
> > > > > > >> > > > > > it
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own
> > > advantages
> > > > > > >> > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like
> to
> > > > have
> > > > > > both
> > > > > > >> > ways
> > > > > > >> > > > of
> > > > > > >> > > > > > > doing
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This
> proposal
> > > > adds
> > > > > 4
> > > > > > >> new
> > > > > > >> > > APIs
> > > > > > >> > > > to
> > > > > > >> > > > > > > > create
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend
> > > > storage.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think
> > of
> > > > this
> > > > > > >> > > feature? I
> > > > > > >> > > > > > would
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some
> > > feedback.
> > > > I
> > > > > am
> > > > > > >> > > working
> > > > > > >> > > > on
> > > > > > >> > > > > > > > making
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > >> > > > > > > > > > > > spec
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but
> wanted
> > > to
> > > > > > have a
> > > > > > >> > high
> > > > > > >> > > > > level
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before
> > > > starting
> > > > > > to
> > > > > > >> > work
> > > > > > >> > > on
> > > > > > >> > > > > it.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> --
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack
> > Developer,
> > > > > > >> SolidFire
> > > > > > >> > > Inc.*
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> e:
> > > mike.tutkowski@solidfire.com
> > > > > > >> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the
> world
> > > uses
> > > > > the
> > > > > > >> > cloud
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> <
> > > > > > >> > > > > > >
> http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > > > > >*™
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>> --
> > > > > > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer,
> > > > > SolidFire
> > > > > > >> > Inc.*
> > > > > > >> > > > > > > > > > > > > > >>>>>> e:
> > mike.tutkowski@solidfire.com
> > > > > > >> > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > >>>>>> Advancing the way the world
> > uses
> > > > the
> > > > > > >> cloud
> > > > > > >> > > > > > > > > > > > > > >>>>>> <
> > > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > > > >*™*
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> --
> > > > > > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer,
> > > > > SolidFire
> > > > > > >> Inc.*
> > > > > > >> > > > > > > > > > > > > > >>>>> e:
> mike.tutkowski@solidfire.com
> > > > > > >> > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > >>>>> Advancing the way the world
> uses
> > > the
> > > > > > cloud
> > > > > > >> > > > > > > > > > > > > > >>>>> <
> > > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > > > >*™*
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> --
> > > > > > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer,
> > > > SolidFire
> > > > > > >> Inc.*
> > > > > > >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > >>> Advancing the way the world uses
> > the
> > > > > cloud
> > > > > > >> > > > > > > > > > > > > > >>> <
> > > > > > >> > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > > >*™*
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > --
> > > > > > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > > *Senior CloudStack Developer,
> > > SolidFire
> > > > > > Inc.*
> > > > > > >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > > o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > > Advancing the way the world uses
> the
> > > > cloud
> > > > > > >> > > > > > > > > > > > > > > <
> > > > > > >> > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > >*™*
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Find out more about ShapeBlue and
> our
> > > > range
> > > > > of
> > > > > > >> > > > CloudStack
> > > > > > >> > > > > > > > > > > > > > related
> > > > > > >> > > > > > > > > > > > > services:
> > > > > > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > >> > > > > > > > > > > > > > <
> > > > > > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > > > > >> > > |
> > > > > > >> > > > > > > > CSForge –
> > > > > > >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > > > > > >> > > > > > > http://shapeblue.com/csforge/
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > CloudStack Consulting
> > > > > > >> > > > > > > > > > > > > > <
> > > > > http://shapeblue.com/cloudstack-consultancy/
> > > > > > >
> > > > > > >> |
> > > > > > >> > > > > > > > > > > > > CloudStack
> > > > > > >> > > > > > > > > > > > > > Software Engineering
> > > > > > >> > > > > > > > > > > > > > <
> > > > > > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > > > > > >> > > > >
> > > > > > >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > >> > > > > > > > > > > > > > <
> > > > > > >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > > >> > > > > >
> > > > > > >> > > > > > |
> > > > > > >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > >> > > > > > > > > > > > > > <
> > > > http://shapeblue.com/cloudstack-training/>
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > --
> > > > > > >> > > > > > > > > > > > *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> > > Inc.*
> > > > > > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > o: 303.746.7302
> > > > > > >> > > > > > > > > > > > Advancing the way the world uses the
> cloud
> > > > > > >> > > > > > > > > > > > <
> > > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > >*™*
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Find out more about ShapeBlue and our
> range
> > of
> > > > > > >> CloudStack
> > > > > > >> > > > > related
> > > > > > >> > > > > > > > > > services:
> > > > > > >> > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > >> > > > > > > > > > > <
> > > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > > >
> > > > > > >> |
> > > > > > >> > > > > CSForge –
> > > > > > >> > > > > > > > rapid
> > > > > > >> > > > > > > > > > > IaaS deployment framework <
> > > > > > >> http://shapeblue.com/csforge/
> > > > > > >> > >
> > > > > > >> > > > > > > CloudStack
> > > > > > >> > > > > > > > > > > Consulting <
> > > > > > >> http://shapeblue.com/cloudstack-consultancy/
> > > > > > >> > >
> > > > > > >> > > |
> > > > > > >> > > > > > > > CloudStack
> > > > > > >> > > > > > > > > > > Software Engineering
> > > > > > >> > > > > > > > > > > <
> > > > > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > >> > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > >> > > > > > > > > > > <
> > > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > > >> > >
> > > > > > >> > > |
> > > > > > >> > > > > > > > CloudStack
> > > > > > >> > > > > > > > > > > Bootcamp Training Courses <
> > > > > > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > Find out more about ShapeBlue and our range
> of
> > > > > > >> CloudStack
> > > > > > >> > > > related
> > > > > > >> > > > > > > > > services:
> > > > > > >> > > > > > > > > > IaaS Cloud Design & Build
> > > > > > >> > > > > > > > > > <
> > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > >
> > > > > > |
> > > > > > >> > > > CSForge –
> > > > > > >> > > > > > > rapid
> > > > > > >> > > > > > > > > > IaaS deployment framework <
> > > > > > >> http://shapeblue.com/csforge/>
> > > > > > >> > > > > > > > > > CloudStack Consulting <
> > > > > > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > > > >> > > > > > > |
> > > > > > >> > > > > > > > > CloudStack
> > > > > > >> > > > > > > > > > Software Engineering
> > > > > > >> > > > > > > > > > <
> > > > > > http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > >> > > > > > > > > > CloudStack Infrastructure Support
> > > > > > >> > > > > > > > > > <
> > > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > > > > >> > |
> > > > > > >> > > > > > > CloudStack
> > > > > > >> > > > > > > > > > Bootcamp Training Courses <
> > > > > > >> > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > --
> > > > > > >> > > > > > > *Mike Tutkowski*
> > > > > > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > <javascript:;>
> > > > > > >> > > > > > > o: 303.746.7302
> > > > > > >> > > > > > > Advancing the way the world uses the cloud
> > > > > > >> > > > > > > <
> http://solidfire.com/solution/overview/?video=play
> > > >*™*
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > --
> > > > > > >> > > > > *Mike Tutkowski*
> > > > > > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > >> > > > > o: 303.746.7302
> > > > > > >> > > > > Advancing the way the world uses the cloud
> > > > > > >> > > > > <http://solidfire.com/solution/overview/?video=play
> >*™*
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > --
> > > > > > >> > > > Ian Rae
> > > > > > >> > > > CEO | PDG
> > > > > > >> > > > c: 514.944.4008
> > > > > > >> > > >
> > > > > > >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> > > > > > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada |
> H3J
> > > 1S6
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > *Mike Tutkowski*
> > > > > > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >> > e: mike.tutkowski@solidfire.com
> > > > > > >> > o: 303.746.7302
> > > > > > >> > Advancing the way the world uses the cloud
> > > > > > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Mike Tutkowski*
> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > o: 303.746.7302
> > > > > > > Advancing the way the world uses the cloud
> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > e: mike.tutkowski@solidfire.com
> > > > > > o: 303.746.7302
> > > > > > Advancing the way the world uses the cloud
> > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> > Find out more about ShapeBlue and our range of CloudStack related
> services:
> > IaaS Cloud Design & Build
> > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > IaaS deployment framework <http://shapeblue.com/csforge/>
> > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack
> > Software Engineering
> > <http://shapeblue.com/cloudstack-software-engineering/>
> > CloudStack Infrastructure Support
> > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Here's what we have for snapshots for managed storage as of 4.6, Paul:

1. VM snapshots (no proposed changes to this).

2. Volume snapshots that do not end up on secondary storage, but rather are
stored on a SAN (effectively storing snapshots on primary storage).

Pierre-Luc is saying he'd like this for snapshots for managed storage:

A. VM snapshots (no proposed changes to this).

B. Volume snapshots that export to secondary storage.

C. New: Storage snapshots that behave like 2 (above).

I like Pierre-Luc's ideas there, but the problem is backward compatibility.

If customers who were using managed storage with volume snapshots in 4.6
were getting their snapshots put on a SAN, then in 4.9 - all of a sudden -
their new snapshots are put on secondary storage (unless they explicitly
change over to using the new Storage snapshots feature).



On Mon, Feb 8, 2016 at 12:32 PM, Paul Angus <pa...@shapeblue.com>
wrote:

> Just to make sure I'm on the same page, are we talking about;
> https://issues.apache.org/jira/browse/CLOUDSTACK-9278 ?
>
> The FS reads (to me) more like 1a + the possibility to export to secondary
> storage if required?
> Have I understood correctly?
> I have seen [1a] implemented for VMware by NetApp in their beta CloudStack
> plugin (pleased I can say that without Mike beating me up now). No changes
> to the CloudStack API were required. (nb it didn't export to secondary
> storage).
>
>
>
> 1. VM Snapshot (point-in-time hypervisor based snapshots)
> 1a. SAN assisted VM snapshots (point-in-time hypervisor snapshot takes
> place on transparently SAN to avoid performance issue in disk chains)
> 2. SAN Snapshot (Storage Snapshot) - NEW
> 3. Volume Snapshot (current old/slow transfer to secstorage)
> 4. Backup - JUST AN IDEAL.
>
>
>
>
> [image: ShapeBlue] <http://www.shapeblue.com>
> Paul Angus
> VP Technology ,  ShapeBlue
> d:  *+44 203 617 0528 | s: +44 203 603 0540*
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> *+44 7711 418784* <+44%207711%20418784>
> e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> <pa...@cloudyangus>  |  w:
> *www.shapeblue.com* <http://www.shapeblue.com>
> a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> Services India LLP is a company incorporated in India and is operated under
> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> company incorporated in Brasil and is operated under license from Shape
> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
> a registered trademark.
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error.
>
>
> -----Original Message-----
> From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
> Sent: Monday, February 8, 2016 7:16 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [Propose][New Feature] Storage Snapshots
>
> Correct, Will.
>
> That Global Settings would only be for managed storage. Non-managed
> (traditional) volume snapshots are completely un-impacted by this feature.
>
> If we need to sometimes keep the snapshots on the SAN and sometimes push
> them to secondary storage, we'll need a more robust solution than Global
> Settings, though.
>
> On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > Sorry. I missed a bit of context when I responded. The global setting
> > would be only for the managed storage case, currently being called
> Storage
> > Snapshots, and is only to determine if a copy is pushed to secondary
> > storage right? The global setting would not change the behavior of the
> > Volume Snapshots right?
> >
> > I was referring to the need for Volume Snapshots and Storage Snapshots to
> > exist together. Disregard my comment. I caught up on context after I
> > posted. My bad...
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com
> > > wrote:
> >
> > > Hey Will,
> > >
> > > Who's picking the behavior? Is it the cloud provider or the end user?
> > >
> > > Thanks
> > >
> > > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > I don't think a global setting is a good option because we need both
> > > > functionality to be available at the same time and for different use
> > > cases
> > > > to be able to pick which they choose.
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > *CloudOps* *| *Cloud Solutions Experts
> > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > w cloudops.com *|* tw @CloudOps_
> > > >
> > > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com
> > > > > wrote:
> > > >
> > > > > Now that I re-read your e-mail, it dawned on me: The end-user
> doesn't
> > > > care
> > > > > where the snapshot is.
> > > > >
> > > > > If that's true, then we should perhaps control this via Global
> > Settings
> > > > or
> > > > > something.
> > > > >
> > > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com> wrote:
> > > > >
> > > > > > It's not ideal - true, but it does allow us to be backward
> > > compatible.
> > > > > >
> > > > > > If you have other ideas, though, about how to maintain backward
> > > > > > compatibility, I'm definitely open to hear them.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > > syed1.mushtaq@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Mike,
> > > > > >>
> > > > > >> Adding a flag to createSnapshot was the first and the most
> obvious
> > > > thing
> > > > > >> that came to our minds. The problem that I had with this was
> that:
> > > > > >>
> > > > > >> 1) I feel it is exposing something to the end user that is
> > internal
> > > to
> > > > > the
> > > > > >> cloud.
> > > > > >>
> > > > > >> 2) We have to follow two different ways of restore/deletion in
> the
> > > > same
> > > > > >> code path depending on where the Snapshot resides which I find
> > kind
> > > > of a
> > > > > >> bad design.
> > > > > >>
> > > > > >> But if exposing a archive flag to the end user is acceptable
> then
> > we
> > > > can
> > > > > >> definitely use this instead of adding the StorageSnapshot API
> > > > > >>
> > > > > >> Thanks,
> > > > > >> -Syed
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > > >> mike.tutkowski@solidfire.com
> > > > > >> > wrote:
> > > > > >>
> > > > > >> > Hi Pierre-Luc,
> > > > > >> >
> > > > > >> > My recommendation would be this:
> > > > > >> >
> > > > > >> > Add an "archive" flag to the current volume-snapshot API. Its
> > > > default
> > > > > >> would
> > > > > >> > be "false" because that would be backward compatible with how
> > 4.6
> > > > has
> > > > > >> > volume snapshots implemented (i.e. they stay on the SAN in
> 4.6,
> > > 4.7,
> > > > > and
> > > > > >> > 4.8).
> > > > > >> >
> > > > > >> > If you set archive=true, then we would perform a background
> > > > migration
> > > > > of
> > > > > >> > the snapshot from the SAN to the secondary storage (then
> delete
> > > the
> > > > > SAN
> > > > > >> > snapshot).
> > > > > >> >
> > > > > >> > That archive parameter would only be valid for managed
> storage.
> > > > > >> >
> > > > > >> > Sound reasonable?
> > > > > >> >
> > > > > >> > Also, a VM snapshot that includes disks provided by managed
> > > storage
> > > > > >> should
> > > > > >> > work.
> > > > > >> >
> > > > > >> > Talk to you later,
> > > > > >> > Mike
> > > > > >> >
> > > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > > pdion@cloudops.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Mike,
> > > > > >> > >
> > > > > >> > > In terms of API's, would you prefer introducing a parameter
> to
> > > the
> > > > > >> > existing
> > > > > >> > > VolumeSnapshot, example: extract={true|false} with a
> > default
> > > > > value
> > > > > >> of
> > > > > >> > > true which would extract snapshot into the secondary
> storage,
> > > > which
> > > > > >> is
> > > > > >> > the
> > > > > >> > > current default behavior. Then for SAN snapshot that remain
> on
> > > the
> > > > > >> SAN we
> > > > > >> > > would just set "extract=false" ? as oppose to create a new
> > > > > >> > > StorageSnapshot API ?
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Paul,
> > > > > >> > >
> > > > > >> > > From what I'm seeing so far, we can't do a VM-snapshot when
> > > using
> > > > > >> managed
> > > > > >> > > storage for VM having more than one Volume. For the reason
> > that
> > > > > >> snapshot
> > > > > >> > > are performed outside of the hypervisor awareness and
> > > > > asynchronously.
> > > > > >> If
> > > > > >> > > someone have a way to address that, it would make thinks
> much
> > > more
> > > > > >> > > attractive.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <irae@cloudops.com
> >
> > > > wrote:
> > > > > >> > >
> > > > > >> > > > I think a service provider backup scenario is more likely
> to
> > > > take
> > > > > >> > > advantage
> > > > > >> > > > of SAN snapshot. There are a few reasons for this.
> > Traditional
> > > > > >> backups
> > > > > >> > > > involve access to the file system, and there is an
> > expectation
> > > > > that
> > > > > >> > this
> > > > > >> > > > can be done with reasonably short time frames without
> > > negatively
> > > > > >> > > impacting
> > > > > >> > > > VM performance, and that the backup orchestrator can apply
> > > > various
> > > > > >> > logic
> > > > > >> > > > and or transformations to the data (compress, encrypt,
> > deltas
> > > > > >> etc...).
> > > > > >> > > > While it is true that one could apply a backup process to
> a
> > > > cloud
> > > > > >> > > snapshot,
> > > > > >> > > > this would be slow and inefficient requiring the data to
> be
> > > > moved
> > > > > >> > several
> > > > > >> > > > times and there are some major bottlenecks with cloud
> > > snapshots.
> > > > > >> With a
> > > > > >> > > > cloud snapshot - there seems to be no reasonable
> expectation
> > > of
> > > > > >> being
> > > > > >> > > able
> > > > > >> > > > to do differential snapshots (I think this depends on the
> > > > > >> hypervisor)
> > > > > >> > and
> > > > > >> > > > if you do differential snapshots this will make file
> backups
> > > > from
> > > > > >> those
> > > > > >> > > > snapshots even more complicated to orchestrate.
> > > > > >> > > >
> > > > > >> > > > Suspect there needs to be a different thread on how to
> > better
> > > > > enable
> > > > > >> > > > backups, as a service. As per Paul's suggestion, but it
> is a
> > > > > related
> > > > > >> > > > workflow so relevant to this discussion.
> > > > > >> > > >
> > > > > >> > > > Ian
> > > > > >> > > >
> > > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > > >> > > mike.tutkowski@solidfire.com>
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > To me it sounds like number two and number three are
> > > different
> > > > > >> uses
> > > > > >> > for
> > > > > >> > > > the
> > > > > >> > > > > same "thing"(which is totally fine).
> > > > > >> > > > >
> > > > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > > > >> asynchronously, do
> > > > > >> > > we
> > > > > >> > > > > see the SSVM as performing the export?
> > > > > >> > > > >
> > > > > >> > > > > To be backwards compatible with what we have in 4.6 and
> > > later
> > > > > for
> > > > > >> > > volume
> > > > > >> > > > > snapshots for managed storage, I think it might be
> easier
> > if
> > > > we
> > > > > >> pass
> > > > > >> > > in a
> > > > > >> > > > > flag that says whether or not to archive the SAN
> snapshot
> > > > > (which,
> > > > > >> I
> > > > > >> > > > think,
> > > > > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > > > > >> > > > >
> > > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > > pdion@cloudops.com
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi Mike,
> > > > > >> > > > > >
> > > > > >> > > > > > The reason behind the creation of a SAN snapshot which
> > is
> > > > > >> exported
> > > > > >> > > into
> > > > > >> > > > > > secondary storage, is because creating a copy of the
> > .VHD
> > > > > >> directly
> > > > > >> > > > would
> > > > > >> > > > > > impact uptime of the VM as creating that copy take
> lots
> > of
> > > > > time.
> > > > > >> > Has
> > > > > >> > > > > oppose
> > > > > >> > > > > > to a SAN snapshot that is close to instantaneous which
> > can
> > > > > >> > afterward
> > > > > >> > > be
> > > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > > >> > > > > >
> > > > > >> > > > > > I would suspect an extracted VolumeSnapshot taken
> from a
> > > SAN
> > > > > >> > snapshot
> > > > > >> > > > > could
> > > > > >> > > > > > have is SAN snapshot deleted to avoid duplica and
> space
> > > > > >> consumption
> > > > > >> > > on
> > > > > >> > > > > the
> > > > > >> > > > > > Primary Storage side.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > I see 3 definitions in our current discussion
> regarding
> > > the
> > > > > term
> > > > > >> > > > snapshot
> > > > > >> > > > > > (these are not official terminology but by own
> > > > interpretation
> > > > > of
> > > > > >> > > them):
> > > > > >> > > > > >
> > > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's
> definition
> > > of
> > > > a
> > > > > >> > > > snapshot):
> > > > > >> > > > > > it's a volume snapshot at the storage level, point in
> > time
> > > > of
> > > > > >> your
> > > > > >> > > > data.
> > > > > >> > > > > it
> > > > > >> > > > > > reside on the primary storage. Useful and efficient
> for
> > > > > software
> > > > > >> > side
> > > > > >> > > > > > incident.
> > > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/
> > > cloud
> > > > > >> backup
> > > > > >> > > > aws-S3
> > > > > >> > > > > > style ): Point in time copy of the Virtual Disk that
> > > reside
> > > > > on a
> > > > > >> > > > > different
> > > > > >> > > > > > storage array then the original Volume. Facilitate
> data
> > > > > >> migration
> > > > > >> > > > between
> > > > > >> > > > > > clusters and, in case of primary storage incident,
> > Volume
> > > > > >> snapshots
> > > > > >> > > are
> > > > > >> > > > > not
> > > > > >> > > > > > impacted and can be reuse.
> > > > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data
> that
> > > > also
> > > > > >> > > validate
> > > > > >> > > > > data
> > > > > >> > > > > > integrity, provide a storage efficient archiving
> method
> > > and
> > > > an
> > > > > >> > > > > independent
> > > > > >> > > > > > way to restore your data in case of an major
> > > infrastructure
> > > > > >> > disaster.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Regards,
> > > > > >> > > > > >
> > > > > >> > > > > > PL
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > So, let's see if I currently follow the
> requirements:
> > > > > >> > > > > > >
> > > > > >> > > > > > > * Augment volume snapshots for managed storage to
> > > > > >> conditionally
> > > > > >> > > > export
> > > > > >> > > > > > data
> > > > > >> > > > > > > to NFS. The current process of taking a snapshot on
> > the
> > > > SAN
> > > > > is
> > > > > >> > > fine,
> > > > > >> > > > > but
> > > > > >> > > > > > > we'd like the option to export the data to NFS, as
> > well.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Questions:
> > > > > >> > > > > > >
> > > > > >> > > > > > > Once the data has been exported to NFS, do we keep
> the
> > > SAN
> > > > > >> > snapshot
> > > > > >> > > > or
> > > > > >> > > > > > > delete it?
> > > > > >> > > > > > >
> > > > > >> > > > > > > If we are deleting the SAN snapshot, then why don't
> we
> > > > just
> > > > > >> copy
> > > > > >> > > the
> > > > > >> > > > > VHD
> > > > > >> > > > > > > from primary to secondary the way we do today for
> > > > > non-managed
> > > > > >> > (i.e.
> > > > > >> > > > > > > traditional) storage? Why create a SAN snapshot in
> > that
> > > > > >> scenario?
> > > > > >> > > > > Perhaps
> > > > > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> > > > secondary
> > > > > >> > > storage
> > > > > >> > > > > > > instead of a XenServer host?
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks for the clarification.
> > > > > >> > > > > > >
> > > > > >> > > > > > > By the way, to me a backup is when you copy data
> from
> > > one
> > > > > >> storage
> > > > > >> > > > > system
> > > > > >> > > > > > to
> > > > > >> > > > > > > another (regardless of features, if any, to restore
> > the
> > > > data
> > > > > >> in
> > > > > >> > the
> > > > > >> > > > > > > future). A snapshot is a point-in-time view of the
> > data
> > > > of a
> > > > > >> > volume
> > > > > >> > > > and
> > > > > >> > > > > > > it's stored on the same storage system as the
> volume.
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > > > >> > > pdion@cloudops.com
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > <javascript:;>>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > That's fun to see that discussion happening. I
> 100%
> > > > agree
> > > > > >> with
> > > > > >> > > > Paul's
> > > > > >> > > > > > > > points of view. VolumeSnapshot are not a backup,
> > but I
> > > > do
> > > > > >> > > consider
> > > > > >> > > > > them
> > > > > >> > > > > > > as
> > > > > >> > > > > > > > a safety vest against Primary Storage failure,
> > because
> > > > > >> failure
> > > > > >> > > > append
> > > > > >> > > > > > > :-( .
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > The current proposal around snapshots that reside
> on
> > > the
> > > > > >> > primary
> > > > > >> > > > > > storage
> > > > > >> > > > > > > or
> > > > > >> > > > > > > > snapshots that end in the Secondary Storage is not
> > to
> > > > > >> address
> > > > > >> > > any
> > > > > >> > > > > kind
> > > > > >> > > > > > > of
> > > > > >> > > > > > > > backups requirement because a snapshot is not a
> > > backup,
> > > > > >> event
> > > > > >> > an
> > > > > >> > > > > > > extracted
> > > > > >> > > > > > > > VM snapshot.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > The main idea, and again this is for managed
> > storage;
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side
> snapshot
> > > > > >> capability
> > > > > >> > > for
> > > > > >> > > > > > fast
> > > > > >> > > > > > > > response time that support rollback to previous
> > > > timestamp,
> > > > > >> > create
> > > > > >> > > > new
> > > > > >> > > > > > > > volume and maybe create template.
> > > > > >> > > > > > > > not required to be a new API if the work is
> > > already
> > > > > >> done, I
> > > > > >> > > > think
> > > > > >> > > > > > > this
> > > > > >> > > > > > > > is a different behaviors than the user expectation
> > of
> > > a
> > > > > >> > > > > > volume-snapshot.
> > > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> > > > behavior
> > > > > >> that
> > > > > >> > > > create
> > > > > >> > > > > > an
> > > > > >> > > > > > > > extraction of a volume into SecondaryStorage which
> > can
> > > > be
> > > > > >> reuse
> > > > > >> > > to
> > > > > >> > > > > > > create a
> > > > > >> > > > > > > > new volume into another Primary Storage. This type
> > of
> > > > > >> snapshot
> > > > > >> > > is a
> > > > > >> > > > > > slow
> > > > > >> > > > > > > > job since yes it would have to copy the full
> volume
> > > size
> > > > > on
> > > > > >> the
> > > > > >> > > > > > Secondary
> > > > > >> > > > > > > > storage.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > PL
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > >> > > > > > <javascript:;>>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > I think I share you view on the 'Ideal world'.
> > > Backup
> > > > > (via
> > > > > >> > > Volume
> > > > > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack.
> > This
> > > is
> > > > > >> > > amplified
> > > > > >> > > > > > > > especially
> > > > > >> > > > > > > > > when you have a object storage as your secondary
> > > > storage
> > > > > >> > > because
> > > > > >> > > > it
> > > > > >> > > > > > > > > requires two copies (one to an NFS staging area
> > and
> > > > from
> > > > > >> > there
> > > > > >> > > to
> > > > > >> > > > > > > object
> > > > > >> > > > > > > > > storage). And not to mention that all these
> copies
> > > are
> > > > > >> > > consuming
> > > > > >> > > > > > > > hypervisor
> > > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> > > bottleneck
> > > > as
> > > > > >> all
> > > > > >> > > the
> > > > > >> > > > > > > Network
> > > > > >> > > > > > > > > and I/O flow through it. So our intention of
> > > proposing
> > > > > the
> > > > > >> > > > "Storage
> > > > > >> > > > > > > > > Snapshots" is to give a better way of achiving
> > > > snapshots
> > > > > >> > while
> > > > > >> > > > > still
> > > > > >> > > > > > > > > keeping the original definition of volume
> > snpashots
> > > > (ie
> > > > > >> > upload
> > > > > >> > > to
> > > > > >> > > > > sec
> > > > > >> > > > > > > > > storage).
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > > > > backups.
> > > > > >> > They
> > > > > >> > > > > don't
> > > > > >> > > > > > > work
> > > > > >> > > > > > > > > form multi-disk LVM volume groups and dynamic
> > > disks. I
> > > > > am
> > > > > >> all
> > > > > >> > > in
> > > > > >> > > > > for
> > > > > >> > > > > > a
> > > > > >> > > > > > > > > better backup solution which handles these use
> > cases
> > > > and
> > > > > >> > > utilizes
> > > > > >> > > > > the
> > > > > >> > > > > > > > > storage's advanced features.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > > > >> > > > > > paul.angus@shapeblue.com <javascript:;>
> <javascript:;>>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > In the beginning... there were CloudStack
> > > snapshots
> > > > > and
> > > > > >> > they
> > > > > >> > > > were
> > > > > >> > > > > > > > > actually
> > > > > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > > > > snapshots.
> > > > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > > > point-in-time
> > > > > >> > > > > hypervisor
> > > > > >> > > > > > > > > > snapshots) and we started referring to the
> > > original
> > > > > >> > snapshots
> > > > > >> > > > as
> > > > > >> > > > > > > volume
> > > > > >> > > > > > > > > > snapshots.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > CloudStack does not offer 'backups', but many
> > > people
> > > > > use
> > > > > >> > > volume
> > > > > >> > > > > > > > snapshots
> > > > > >> > > > > > > > > > as backups. However you can't in-place restore
> > > > volume
> > > > > >> > > snapshots
> > > > > >> > > > > and
> > > > > >> > > > > > > if
> > > > > >> > > > > > > > > you
> > > > > >> > > > > > > > > > have a VM with multiple volumes, the volume
> > > > snapshots
> > > > > >> must
> > > > > >> > be
> > > > > >> > > > > done
> > > > > >> > > > > > in
> > > > > >> > > > > > > > > > series, meaning that the state across all of
> the
> > > > > >> volumes is
> > > > > >> > > > > > unlikely
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > be
> > > > > >> > > > > > > > > > consistent.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > 'Actual Backups' would enable all of the
> restore
> > > > > options
> > > > > >> > > which
> > > > > >> > > > > > users
> > > > > >> > > > > > > > > might
> > > > > >> > > > > > > > > > expect as well options as to where they might
> be
> > > > > >> stored. In
> > > > > >> > > my
> > > > > >> > > > > > ideal
> > > > > >> > > > > > > > > world
> > > > > >> > > > > > > > > > they would also be able to leverage back-end
> > > > hardware
> > > > > >> (such
> > > > > >> > > as
> > > > > >> > > > > > > > Solidfire,
> > > > > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> > > > Commvault
> > > > > >> etc
> > > > > >> > to
> > > > > >> > > > > > > accelerate
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > process.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > >> > > > > > > > > > Paul Angus
> > > > > >> > > > > > > > > > VP Technology , ShapeBlue
> > > > > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > >> > > > > > > > > >
> > > > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > |
> > > > > >> > > > > m:
> > > > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > >> > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > > > >> > <javascript:;>
> > > > > >> > > |
> > > > > >> > > > > t: @cloudyangus*
> > > > > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > >> > > > > <ja...@cloudyangus>
> > > > > >> > > > > > | w:
> > > > > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com
> >
> > > > > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> > > 4HS
> > > > UK
> > > > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in
> > > England
> > > > &
> > > > > >> > Wales.
> > > > > >> > > > > > > ShapeBlue
> > > > > >> > > > > > > > > > Services India LLP is a company incorporated
> in
> > > > India
> > > > > >> and
> > > > > >> > is
> > > > > >> > > > > > operated
> > > > > >> > > > > > > > > under
> > > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > > > >> Consultoria
> > > > > >> > > Ltda
> > > > > >> > > > > is
> > > > > >> > > > > > a
> > > > > >> > > > > > > > > > company incorporated in Brasil and is operated
> > > under
> > > > > >> > license
> > > > > >> > > > from
> > > > > >> > > > > > > Shape
> > > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > > registered
> > > > > >> by
> > > > > >> > The
> > > > > >> > > > > > > Republic
> > > > > >> > > > > > > > of
> > > > > >> > > > > > > > > > South Africa and is traded under license from
> > > Shape
> > > > > Blue
> > > > > >> > Ltd.
> > > > > >> > > > > > > ShapeBlue
> > > > > >> > > > > > > > > is
> > > > > >> > > > > > > > > > a registered trademark.
> > > > > >> > > > > > > > > > This email and any attachments to it may be
> > > > > confidential
> > > > > >> > and
> > > > > >> > > > are
> > > > > >> > > > > > > > intended
> > > > > >> > > > > > > > > > solely for the use of the individual to whom
> it
> > is
> > > > > >> > addressed.
> > > > > >> > > > Any
> > > > > >> > > > > > > views
> > > > > >> > > > > > > > > or
> > > > > >> > > > > > > > > > opinions expressed are solely those of the
> > author
> > > > and
> > > > > do
> > > > > >> > not
> > > > > >> > > > > > > > necessarily
> > > > > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > > > > companies.
> > > > > >> If
> > > > > >> > > you
> > > > > >> > > > > are
> > > > > >> > > > > > > not
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > intended recipient of this email, you must
> > neither
> > > > > take
> > > > > >> any
> > > > > >> > > > > action
> > > > > >> > > > > > > > based
> > > > > >> > > > > > > > > > upon its contents, nor copy or show it to
> > anyone.
> > > > > Please
> > > > > >> > > > contact
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > sender
> > > > > >> > > > > > > > > > if you believe you have received this email in
> > > > error.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > -----Original Message-----
> > > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > > syed1.mushtaq@gmail.com
> > > > > >> > > > > <javascript:;> <javascript:;>]
> > > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > > >> > <javascript:;>
> > > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > > Snapshots
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Paul,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > When you say actual backups, how would it be
> > > > different
> > > > > >> from
> > > > > >> > > the
> > > > > >> > > > > > > Volume
> > > > > >> > > > > > > > > > Snapshots that exist currently. My
> understanding
> > > is
> > > > > that
> > > > > >> > > > Backups
> > > > > >> > > > > > end
> > > > > >> > > > > > > up
> > > > > >> > > > > > > > > in
> > > > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > > > point-in-time
> > > > > >> > state
> > > > > >> > > of
> > > > > >> > > > > > your
> > > > > >> > > > > > > > > volume
> > > > > >> > > > > > > > > > which can be restored back correct?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > -Syed
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> > <javascript:;>>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > Hi Syed,
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > As I understand it, the SolidFire plugin
> will
> > > > export
> > > > > >> the
> > > > > >> > > > > snapshot
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > > secondary storage if the user requests a
> > > template
> > > > > from
> > > > > >> > the
> > > > > >> > > > > > snapshot
> > > > > >> > > > > > > > or
> > > > > >> > > > > > > > > > > wants to download the snapshot from the
> cloud.
> > > > This
> > > > > >> is a
> > > > > >> > > > good,
> > > > > >> > > > > > > > > > > pragmatic approach and yes Mike the
> SolidFire
> > > > > storage
> > > > > >> is
> > > > > >> > > > super
> > > > > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays
> > take
> > > up
> > > > > >> next
> > > > > >> > to
> > > > > >> > > no
> > > > > >> > > > > > > space.
> > > > > >> > > > > > > > > > > BUT I think that we are talking about a more
> > > > general
> > > > > >> > > purpose
> > > > > >> > > > > API,
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > > > > other storage systems may not be as awesome
> as
> > > > > Mike's.
> > > > > >> > > That's
> > > > > >> > > > > my
> > > > > >> > > > > > > > > > > concern. Also, the time to transfer for say
> > 1TB
> > > to
> > > > > >> move
> > > > > >> > > from
> > > > > >> > > > > > > primary
> > > > > >> > > > > > > > > > > to sec storage and then create a VM template
> > out
> > > > of
> > > > > it
> > > > > >> > may
> > > > > >> > > be
> > > > > >> > > > > too
> > > > > >> > > > > > > > long
> > > > > >> > > > > > > > > > for users.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > > > > snapshot
> > > > > >> for
> > > > > >> > > > > backup,
> > > > > >> > > > > > > > it's
> > > > > >> > > > > > > > > > > just that users want to do backups and a
> > volume
> > > > > >> snapshot
> > > > > >> > is
> > > > > >> > > > the
> > > > > >> > > > > > > only
> > > > > >> > > > > > > > > > > type of snapshot that copies the disk
> > elsewhere
> > > > and
> > > > > >> can
> > > > > >> > be
> > > > > >> > > > > > > scheduled.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> > > > actual
> > > > > >> > backups
> > > > > >> > > > > > > (through
> > > > > >> > > > > > > > > > > recognised backup providers) and the user
> > > > > requirements
> > > > > >> > > around
> > > > > >> > > > > > them
> > > > > >> > > > > > > > > > > (particularly restoration use cases) as a
> > > separate
> > > > > >> thread
> > > > > >> > > of
> > > > > >> > > > > > work.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > [image: ShapeBlue] <
> http://www.shapeblue.com>
> > > > Paul
> > > > > >> Angus
> > > > > >> > > VP
> > > > > >> > > > > > > > Technology
> > > > > >> > > > > > > > > > > , ShapeBlue
> > > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > >> > > > > > > > > > >
> > > > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > >> > > > |
> > > > > >> > > > > m:
> > > > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > > > >> > > <javascript:;> |
> > > > > >> > > > > t: @cloudyangus*
> > > > > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > >> <javascript:;>
> > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > >> > > > > > > > > > > *www.shapeblue.com* <
> http://www.shapeblue.com
> > >
> > > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> WC2N
> > > 4HS
> > > > > UK
> > > > > >> > Shape
> > > > > >> > > > > Blue
> > > > > >> > > > > > > Ltd
> > > > > >> > > > > > > > > > > is a company incorporated in England &
> Wales.
> > > > > >> ShapeBlue
> > > > > >> > > > > Services
> > > > > >> > > > > > > > India
> > > > > >> > > > > > > > > > > LLP is a company incorporated in India and
> is
> > > > > operated
> > > > > >> > > under
> > > > > >> > > > > > > license
> > > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> > > Consultoria
> > > > > >> Ltda
> > > > > >> > is
> > > > > >> > > a
> > > > > >> > > > > > > company
> > > > > >> > > > > > > > > > > incorporated in Brasil and is operated under
> > > > license
> > > > > >> from
> > > > > >> > > > Shape
> > > > > >> > > > > > > Blue
> > > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> > > registered
> > > > by
> > > > > >> The
> > > > > >> > > > > Republic
> > > > > >> > > > > > > of
> > > > > >> > > > > > > > > > > South Africa and is traded under license
> from
> > > > Shape
> > > > > >> Blue
> > > > > >> > > Ltd.
> > > > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > > >> > > > > > > > > > > This email and any attachments to it may be
> > > > > >> confidential
> > > > > >> > > and
> > > > > >> > > > > are
> > > > > >> > > > > > > > > > > intended solely for the use of the
> individual
> > to
> > > > > whom
> > > > > >> it
> > > > > >> > is
> > > > > >> > > > > > > > addressed.
> > > > > >> > > > > > > > > > > Any views or opinions expressed are solely
> > those
> > > > of
> > > > > >> the
> > > > > >> > > > author
> > > > > >> > > > > > and
> > > > > >> > > > > > > do
> > > > > >> > > > > > > > > > > not necessarily represent those of Shape
> Blue
> > > Ltd
> > > > or
> > > > > >> > > related
> > > > > >> > > > > > > > > > > companies. If you are not the intended
> > recipient
> > > > of
> > > > > >> this
> > > > > >> > > > email,
> > > > > >> > > > > > you
> > > > > >> > > > > > > > > > > must neither take any action based upon its
> > > > > contents,
> > > > > >> nor
> > > > > >> > > > copy
> > > > > >> > > > > or
> > > > > >> > > > > > > > show
> > > > > >> > > > > > > > > > > it to anyone. Please contact the sender if
> you
> > > > > believe
> > > > > >> > you
> > > > > >> > > > have
> > > > > >> > > > > > > > > received
> > > > > >> > > > > > > > > > this email in error.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > -----Original Message-----
> > > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > syed1.mushtaq@gmail.com
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > <javascript:;>]
> > > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org
> <javascript:;>
> > > > > >> > > <javascript:;>
> > > > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > > > Snapshots
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > I think the terminology confusion comes from
> > AWS
> > > > > where
> > > > > >> > they
> > > > > >> > > > do
> > > > > >> > > > > > EBS
> > > > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack
> sort
> > of
> > > > > >> followed
> > > > > >> > > > that.
> > > > > >> > > > > > And
> > > > > >> > > > > > > > as
> > > > > >> > > > > > > > > > > an end user who is oblivious to the
> internals
> > of
> > > > my
> > > > > >> > > provider,
> > > > > >> > > > > my
> > > > > >> > > > > > > > > > > expectation would be something similar to
> what
> > > AWS
> > > > > as
> > > > > >> > that
> > > > > >> > > is
> > > > > >> > > > > my
> > > > > >> > > > > > > > > > > biggest reference point.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > To your point Mike, I agree that a Primary
> > > Storage
> > > > > >> > failure
> > > > > >> > > on
> > > > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> > > motivations
> > > > > >> for us
> > > > > >> > > to
> > > > > >> > > > > push
> > > > > >> > > > > > > > data
> > > > > >> > > > > > > > > > > to secondary storage. Primary storage
> (atleast
> > > for
> > > > > us)
> > > > > >> > > costs
> > > > > >> > > > > > > around 3
> > > > > >> > > > > > > > > > > times as much as secondary storage and
> > snapshots
> > > > on
> > > > > >> > primary
> > > > > >> > > > > > storage
> > > > > >> > > > > > > > > > > are rarely used (especially for some of our
> > > > > customers
> > > > > >> who
> > > > > >> > > do
> > > > > >> > > > > > daily
> > > > > >> > > > > > > > > > backups).
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike
> > Tutkowski
> > > <
> > > > > >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > > >> > > <javascript:;>>
> > > > > >> > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Some of the weirdness is around
> terminology.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > For most systems I've worked on, a
> snapshot
> > > and
> > > > a
> > > > > >> > backup
> > > > > >> > > > are
> > > > > >> > > > > > two
> > > > > >> > > > > > > > > > > > completely different things (but
> CloudStack
> > > has
> > > > > >> > > > traditionally
> > > > > >> > > > > > > used
> > > > > >> > > > > > > > > > > > the term "volume snapshot" to mean
> backup).
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and
> > say,
> > > > > >> though,
> > > > > >> > > that
> > > > > >> > > > > if
> > > > > >> > > > > > > your
> > > > > >> > > > > > > > > > > > primary storage is running on SolidFire
> that
> > > it
> > > > is
> > > > > >> > > unlikely
> > > > > >> > > > > > > you'll
> > > > > >> > > > > > > > > > > > encounter an issue where your primary
> > storage
> > > > goes
> > > > > >> > > offline
> > > > > >> > > > > (and
> > > > > >> > > > > > > > > > > > you'll even maintain your performance
> > > guarantees
> > > > > >> during
> > > > > >> > > > > failure
> > > > > >> > > > > > > > > > > > scenarios and upgrades, as well). That
> being
> > > the
> > > > > >> case,
> > > > > >> > it
> > > > > >> > > > is
> > > > > >> > > > > > less
> > > > > >> > > > > > > > > > > > useful to require a backup to Swift (but
> > it's
> > > > > >> perfectly
> > > > > >> > > OK
> > > > > >> > > > if
> > > > > >> > > > > > > > that's
> > > > > >> > > > > > > > > > > > what we want to do
> > > > > >> > > > > > > > > > > here).
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed
> Mushtaq
> > > > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > > > > >> > <javascript:;>>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Paul,
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > I believe with the current
> implementation
> > of
> > > > > >> > Snapshots
> > > > > >> > > on
> > > > > >> > > > > > > managed
> > > > > >> > > > > > > > > > > > > storage
> > > > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never
> > exported
> > > > to
> > > > > >> the
> > > > > >> > > > > secondary
> > > > > >> > > > > > > > > > storage.
> > > > > >> > > > > > > > > > > > > While this solves the problem of having
> > > > > snapshots
> > > > > >> > > taking
> > > > > >> > > > > > > forever
> > > > > >> > > > > > > > > > > > > to get to sec storage, this leaves us
> > with a
> > > > > >> > > > > > > > > > > > huge
> > > > > >> > > > > > > > > > > > > liability if our primary storage goes
> > down.
> > > We
> > > > > see
> > > > > >> > > > > snapshots
> > > > > >> > > > > > as
> > > > > >> > > > > > > > > > > > > our recovery path because we store them
> in
> > > > Swift
> > > > > >> > which
> > > > > >> > > is
> > > > > >> > > > > > > > reliable
> > > > > >> > > > > > > > > > > > > and resilient to failures.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > With Storage snpashots our goal is to
> have
> > > > > Volume
> > > > > >> > > > snapshots
> > > > > >> > > > > > > > always
> > > > > >> > > > > > > > > > > > > backed up to secondary storage and
> Storage
> > > > > >> Snapshots
> > > > > >> > > stay
> > > > > >> > > > > on
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > > primary
> > > > > >> > > > > > > > > > > > storage.
> > > > > >> > > > > > > > > > > > > A provider could potentially mix both
> > these
> > > > and
> > > > > >> solve
> > > > > >> > > the
> > > > > >> > > > > > > problem
> > > > > >> > > > > > > > > > > > > that you mentioned where you want to
> meet
> > > > user's
> > > > > >> > > > > expectation
> > > > > >> > > > > > > of a
> > > > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage)
> while
> > > > having
> > > > > >> an
> > > > > >> > > > ability
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the
> > > device)
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > > > > -Syed
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul
> Angus
> > > > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com
> <javascript:;>
> > > > > >> > > <javascript:;>>
> > > > > >> > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > HI guys,
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Could someone point me to the Jira bug
> > of
> > > FS
> > > > > for
> > > > > >> > the
> > > > > >> > > > > > > > > > > > > > SAN-snapshot
> > > > > >> > > > > > > > > > > > feature
> > > > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > From my discussions with users and
> > > operators
> > > > > >> around
> > > > > >> > > > > > snapshots
> > > > > >> > > > > > > > > > > > > > I'd make
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > following observations:
> > > > > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups
> > (both
> > > > > >> long-term
> > > > > >> > > and
> > > > > >> > > > > > short
> > > > > >> > > > > > > > > > > > > > term)
> > > > > >> > > > > > > > > > > > with
> > > > > >> > > > > > > > > > > > > > the expectation that they can use them
> > for
> > > > > >> recovery
> > > > > >> > > if
> > > > > >> > > > > > > > required.
> > > > > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > > > > something
> > > > > >> > has
> > > > > >> > > > gone
> > > > > >> > > > > > > wrong
> > > > > >> > > > > > > > > > > > > > with primary storage.
> > > > > >> > > > > > > > > > > > > > c. users sometimes want to be able to
> > > export
> > > > > >> > > snapshots
> > > > > >> > > > as
> > > > > >> > > > > > > well
> > > > > >> > > > > > > > > > > > > > as
> > > > > >> > > > > > > > > > > > create
> > > > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > > > >> > > > > > > > > > > > > > d. snapshots are a currently a massive
> > > pain
> > > > > for
> > > > > >> > > > > operators,
> > > > > >> > > > > > I
> > > > > >> > > > > > > > > > > > > > know at
> > > > > >> > > > > > > > > > > > > least
> > > > > >> > > > > > > > > > > > > > one public cloud who have snapshots
> > which
> > > > > take 2
> > > > > >> > days
> > > > > >> > > > to
> > > > > >> > > > > > > > > complete.
> > > > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be
> used
> > > for
> > > > > >> > multiple
> > > > > >> > > > LVM
> > > > > >> > > > > > > > disks.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > I think the process Mike has used in
> the
> > > > > >> SolidFire
> > > > > >> > > > plugin
> > > > > >> > > > > > > (only
> > > > > >> > > > > > > > > > > > > > moving
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > disk image to secondary storage when
> you
> > > > > >> absolutely
> > > > > >> > > > have
> > > > > >> > > > > > to)
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > > > > > > > a very
> > > > > >> > > > > > > > > > > > > good
> > > > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> > > > problems
> > > > > >> an
> > > > > >> > > > > operator
> > > > > >> > > > > > > > might
> > > > > >> > > > > > > > > > > > > experience
> > > > > >> > > > > > > > > > > > > > if they have an issue with a given
> > primary
> > > > > >> storage
> > > > > >> > > pool
> > > > > >> > > > > in
> > > > > >> > > > > > a
> > > > > >> > > > > > > > > > cluster.
> > > > > >> > > > > > > > > > > > (I
> > > > > >> > > > > > > > > > > > > > know that that is REALLY unlikely in
> the
> > > > > >> SolidFire
> > > > > >> > > case
> > > > > >> > > > > > Mike
> > > > > >> > > > > > > :)
> > > > > >> > > > > > > > > > > > > > ) And
> > > > > >> > > > > > > > > > > > if
> > > > > >> > > > > > > > > > > > > > the transfer from primary to secondary
> > is
> > > > > slow,
> > > > > >> the
> > > > > >> > > > time
> > > > > >> > > > > to
> > > > > >> > > > > > > > > > > > > > being able
> > > > > >> > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > create a template or export the volume
> > > will
> > > > be
> > > > > >> > slow.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > So for me the issue is around making
> > sure
> > > > that
> > > > > >> the
> > > > > >> > > end
> > > > > >> > > > > > users
> > > > > >> > > > > > > > > > > > expectations
> > > > > >> > > > > > > > > > > > > > are met (while improving the
> > > > speed/efficiency
> > > > > of
> > > > > >> > the
> > > > > >> > > > back
> > > > > >> > > > > > > end)
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > > > http://www.shapeblue.com>
> > > > > >> Paul
> > > > > >> > > > Angus
> > > > > >> > > > > > VP
> > > > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> > > 0540*
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > >
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > >> > > > > > > |
> > > > > >> > > > > > > > m:
> > > > > >> > > > > > > > > > > > > > *+44 7711 418784*
> <+44%207711%20418784>
> > > > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > <javascript:;>
> > > > > >> > > > > <javascript:;> | t:
> > > > > >> > > > > > @cloudyangus*
> > > > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com
> > <javascript:;>
> > > > > >> > > > <javascript:;>
> > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > > > http://www.shapeblue.com
> > > > > >
> > > > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > London
> > > > WC2N
> > > > > >> 4HS
> > > > > >> > UK
> > > > > >> > > > > Shape
> > > > > >> > > > > > > > Blue
> > > > > >> > > > > > > > > > > > > > Ltd is a company incorporated in
> > England &
> > > > > >> Wales.
> > > > > >> > > > > ShapeBlue
> > > > > >> > > > > > > > > > > > > > Services India LLP is a company
> > > incorporated
> > > > > in
> > > > > >> > India
> > > > > >> > > > and
> > > > > >> > > > > > is
> > > > > >> > > > > > > > > > > > > > operated
> > > > > >> > > > > > > > > > > > > under
> > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape
> Blue
> > > > Brasil
> > > > > >> > > > > Consultoria
> > > > > >> > > > > > > Ltda
> > > > > >> > > > > > > > > > > > > > is a company incorporated in Brasil
> and
> > is
> > > > > >> operated
> > > > > >> > > > under
> > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue
> > SA
> > > > Pty
> > > > > >> Ltd
> > > > > >> > is
> > > > > >> > > a
> > > > > >> > > > > > > company
> > > > > >> > > > > > > > > > > > > > registered by The Republic
> > > > > >> > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > South Africa and is traded under
> license
> > > > from
> > > > > >> Shape
> > > > > >> > > > Blue
> > > > > >> > > > > > Ltd.
> > > > > >> > > > > > > > > > > > > > ShapeBlue
> > > > > >> > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > a registered trademark.
> > > > > >> > > > > > > > > > > > > > This email and any attachments to it
> may
> > > be
> > > > > >> > > > confidential
> > > > > >> > > > > > and
> > > > > >> > > > > > > > are
> > > > > >> > > > > > > > > > > > intended
> > > > > >> > > > > > > > > > > > > > solely for the use of the individual
> to
> > > whom
> > > > > it
> > > > > >> is
> > > > > >> > > > > > addressed.
> > > > > >> > > > > > > > > > > > > > Any views
> > > > > >> > > > > > > > > > > > > or
> > > > > >> > > > > > > > > > > > > > opinions expressed are solely those of
> > the
> > > > > >> author
> > > > > >> > and
> > > > > >> > > > do
> > > > > >> > > > > > not
> > > > > >> > > > > > > > > > > > necessarily
> > > > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or
> > > related
> > > > > >> > > companies.
> > > > > >> > > > > If
> > > > > >> > > > > > > you
> > > > > >> > > > > > > > > > > > > > are not
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > intended recipient of this email, you
> > must
> > > > > >> neither
> > > > > >> > > take
> > > > > >> > > > > any
> > > > > >> > > > > > > > > > > > > > action
> > > > > >> > > > > > > > > > > > based
> > > > > >> > > > > > > > > > > > > > upon its contents, nor copy or show it
> > to
> > > > > >> anyone.
> > > > > >> > > > Please
> > > > > >> > > > > > > > contact
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > sender
> > > > > >> > > > > > > > > > > > > > if you believe you have received this
> > > email
> > > > in
> > > > > >> > error.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > -----Original Message-----
> > > > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > > > >> pdion@cloudops.com
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > <javascript:;>]
> > > > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56
> PM
> > > > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature]
> > > Storage
> > > > > >> > Snapshots
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Hi Mike,
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > > > >> StorageSnapshot
> > > > > >> > > for
> > > > > >> > > > > > > managed
> > > > > >> > > > > > > > > > > > > > storage
> > > > > >> > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > > > > expected,
> > > > > >> > > > behavior
> > > > > >> > > > > > is
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> > > > Storage.
> > > > > >> So a
> > > > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > > > >> > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > snapshot that remain on the managed
> > > storage
> > > > > >> > > appliance.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Quickly looking at the API doc and I
> > don't
> > > > > see a
> > > > > >> > > strong
> > > > > >> > > > > > > > > > > > > > requirement for volume snapshots to be
> > > moved
> > > > > >> into
> > > > > >> > > > > secondary
> > > > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API
> > is
> > > > not
> > > > > >> > useful,
> > > > > >> > > > but
> > > > > >> > > > > > > both
> > > > > >> > > > > > > > > > > > > > use cases are required. A snapshot
> that
> > > > remain
> > > > > >> on
> > > > > >> > the
> > > > > >> > > > > > managed
> > > > > >> > > > > > > > > > > > > > storage, and another type of
> > > > > >> > > > > > > > > > > > snapshot
> > > > > >> > > > > > > > > > > > > > that end up into the secondary
> storage.
> > > > Since
> > > > > >> > you've
> > > > > >> > > > > done a
> > > > > >> > > > > > > lot
> > > > > >> > > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > work,
> > > > > >> > > > > > > > > > > > > > might easier to just add a parameter
> to
> > > the
> > > > > >> current
> > > > > >> > > > > > snapshot
> > > > > >> > > > > > > > API
> > > > > >> > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > trigger an extraction of the storage
> > > > snapshot
> > > > > >> into
> > > > > >> > > the
> > > > > >> > > > > > > > secondary
> > > > > >> > > > > > > > > > > > storage?
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > PL
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > > > > Tutkowski <
> > > > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > I think that all sounds reasonable
> > then
> > > -
> > > > > >> thanks!
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > > > > Mushtaq <
> > > > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > >> <javascript:;>>
> > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >> You are correct Mike in terms of
> the
> > > > > >> > requirements.
> > > > > >> > > > One
> > > > > >> > > > > > of
> > > > > >> > > > > > > > our
> > > > > >> > > > > > > > > > > > earlier
> > > > > >> > > > > > > > > > > > > > >> iterations on this was to have an
> > > > argument
> > > > > to
> > > > > >> > the
> > > > > >> > > > > create
> > > > > >> > > > > > > > > > > > > > >> snapshot
> > > > > >> > > > > > > > > > > > API
> > > > > >> > > > > > > > > > > > > > >> which decides whether to backup the
> > > > volume
> > > > > to
> > > > > >> > sec
> > > > > >> > > > > > storage
> > > > > >> > > > > > > > but
> > > > > >> > > > > > > > > > > > > > >> we realized it would make
> management
> > of
> > > > > >> > snapshots
> > > > > >> > > > > quite
> > > > > >> > > > > > > > messy
> > > > > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > > > >> > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> > > > Tutkowski
> > > > > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > > > > <javascript:;>
> > > > > >> > > > > <javascript:;>>
> > > > > >> > > > > > > > > > > > > > >> wrote:
> > > > > >> > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >>> Hi,
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> Just to make sure I understand all
> > the
> > > > > >> > > requirements
> > > > > >> > > > > > here:
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 1) This relates only to managed
> > > storage
> > > > > (1:1
> > > > > >> > > > mapping
> > > > > >> > > > > > > > between
> > > > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> > > > volume).
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > > > (introduced
> > > > > >> in
> > > > > >> > > 4.6)
> > > > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> > > snapshot
> > > > on
> > > > > >> the
> > > > > >> > > SAN,
> > > > > >> > > > > and
> > > > > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > > > > >> > > > > > > > > > > > > > >>> something) to not only take the
> SAN
> > > > > >> snapshot,
> > > > > >> > but
> > > > > >> > > > to
> > > > > >> > > > > > copy
> > > > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer
> only)
> > to
> > > > > NFS.
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always
> taken.
> > > > It's
> > > > > >> the
> > > > > >> > > > backup
> > > > > >> > > > > to
> > > > > >> > > > > > > NFS
> > > > > >> > > > > > > > > > > > > > >>> that is optional.
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 4) Templates can be created from
> the
> > > > > >> snapshot
> > > > > >> > > > that's
> > > > > >> > > > > on
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be
> created
> > > > from
> > > > > >> the
> > > > > >> > > > > snapshot
> > > > > >> > > > > > > > > > > > > > >>> that's on
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>> SAN (already works as long as the
> > new
> > > > > >> > CloudStack
> > > > > >> > > > > volume
> > > > > >> > > > > > > > ends
> > > > > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > > > > snapshot
> > > > > >> API
> > > > > >> > > > then
> > > > > >> > > > > or
> > > > > >> > > > > > > > > > > > > > >>> would that just be the standard
> > volume
> > > > > >> snapshot
> > > > > >> > > > > without
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>> backup to
> > > > > >> > > > > > > > > > > NFS?
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> Thanks!
> > > > > >> > > > > > > > > > > > > > >>> Mike
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM,
> Syed
> > > > > Mushtaq
> > > > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> > > <javascript:;>
> > > > > >> > > > > <javascript:;>>
> > > > > >> > > > > > > > > > > > > > >>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > > > functionalities
> > > > > >> > > > > (snapshot
> > > > > >> > > > > > on
> > > > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because
> Ideally,
> > we
> > > > > would
> > > > > >> > like
> > > > > >> > > > to
> > > > > >> > > > > > have
> > > > > >> > > > > > > > > both.
> > > > > >> > > > > > > > > > > > > > >>>> For example, some of our
> customers
> > > want
> > > > > to
> > > > > >> > > > implement
> > > > > >> > > > > > > their
> > > > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> > > encryption
> > > > > to
> > > > > >> > their
> > > > > >> > > > > > backups
> > > > > >> > > > > > > > > > > > > > >>>> which is a perfect use case for
> > > Storage
> > > > > >> > Snapshot
> > > > > >> > > > > while
> > > > > >> > > > > > > our
> > > > > >> > > > > > > > > > > > > > >>>> other customers will still keep
> > using
> > > > the
> > > > > >> > > standard
> > > > > >> > > > > > > volume
> > > > > >> > > > > > > > > > > snapshot.
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>> To keep things backward
> compatible,
> > > we
> > > > > can
> > > > > >> > add a
> > > > > >> > > > > > setting
> > > > > >> > > > > > > > > > > > > > >>>> which
> > > > > >> > > > > > > > > > > > says
> > > > > >> > > > > > > > > > > > > > >>>> to not upload on secondary
> storage,
> > > > > >> because,
> > > > > >> > > after
> > > > > >> > > > > > all,
> > > > > >> > > > > > > > you
> > > > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first
> > when
> > > > > doing
> > > > > >> a
> > > > > >> > > > Volume
> > > > > >> > > > > > > > > Snapshot.
> > > > > >> > > > > > > > > > > > > > >>>> You could stop the process there
> > and
> > > > not
> > > > > do
> > > > > >> > the
> > > > > >> > > > > > upload.
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>> What do you think about this
> > > approach?
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > > > >> > > > > > > > > > > > > > >>>> -Syed
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM,
> > Mike
> > > > > >> > Tutkowski <
> > > > > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out
> > > load
> > > > > >> here,
> > > > > >> > but
> > > > > >> > > > if
> > > > > >> > > > > a
> > > > > >> > > > > > > > given
> > > > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't
> actually
> > > need
> > > > > to
> > > > > >> > > provide
> > > > > >> > > > > > both
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > ability
> > > > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and
> export
> > it
> > > > to
> > > > > >> NFS
> > > > > >> > (if
> > > > > >> > > > > just
> > > > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK),
> then
> > > we
> > > > > >> might
> > > > > >> > be
> > > > > >> > > > able
> > > > > >> > > > > > to
> > > > > >> > > > > > > > get
> > > > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > > > >> > > > > > > > > > > > API
> > > > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> > > > custom
> > > > > >> > > snapshot
> > > > > >> > > > > > > strategy
> > > > > >> > > > > > > > > > > > > > >>>>> and
> > > > > >> > > > > > > > > > > > data
> > > > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the
> case
> > > > where
> > > > > >> the
> > > > > >> > > > > > CloudStack
> > > > > >> > > > > > > > > > > > > > >>>>> cloud
> > > > > >> > > > > > > > > > > > does
> > > > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > > > >> exported-to-NFS
> > > > > >> > > > > backup.
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> In other words, the "default"
> > > behavior
> > > > > >> would
> > > > > >> > be
> > > > > >> > > > to
> > > > > >> > > > > > use
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data
> motion
> > > > > strategy
> > > > > >> > that
> > > > > >> > > > we
> > > > > >> > > > > > > > already
> > > > > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a
> > SAN
> > > > > >> > snapshot).
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud,
> however,
> > > > wants
> > > > > >> to
> > > > > >> > > take
> > > > > >> > > > a
> > > > > >> > > > > > SAN
> > > > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data
> > exported
> > > to
> > > > > >> NFS,
> > > > > >> > > then
> > > > > >> > > > we
> > > > > >> > > > > > > could
> > > > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing
> config
> > > > file
> > > > > to
> > > > > >> > make
> > > > > >> > > > use
> > > > > >> > > > > > of
> > > > > >> > > > > > > a
> > > > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data
> > > motion
> > > > > >> > strategy
> > > > > >> > > > that
> > > > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > > > >> > > > > > > > > > > > > > activities.
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is
> > still
> > > > the
> > > > > >> > default
> > > > > >> > > > for
> > > > > >> > > > > > > > users,
> > > > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change
> > > this
> > > > > >> > behavior
> > > > > >> > > > via
> > > > > >> > > > > > > > > > > configuration.
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM,
> > > Mike
> > > > > >> > > Tutkowski <
> > > > > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to
> > > come
> > > > up
> > > > > >> > with a
> > > > > >> > > > > > viable
> > > > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable
> > way
> > > > for
> > > > > >> them
> > > > > >> > > to
> > > > > >> > > > > move
> > > > > >> > > > > > > > from
> > > > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and
> > > some
> > > > > >> obvious
> > > > > >> > > way
> > > > > >> > > > > > that
> > > > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > > > >> > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > do this).
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45
> AM,
> > > Syed
> > > > > >> > Mushtaq
> > > > > >> > > <
> > > > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > > > <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> > > > upgrade
> > > > > >> path
> > > > > >> > > > > however,
> > > > > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6
> and
> > > are
> > > > > on a
> > > > > >> > > > managed
> > > > > >> > > > > > > > storage
> > > > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > > > > >> functionality
> > > > > >> > > > with
> > > > > >> > > > > > > Volume
> > > > > >> > > > > > > > > > > Snapshots.
> > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43
> PM,
> > > Syed
> > > > > >> > Mushtaq
> > > > > >> > > <
> > > > > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > > > > <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly,
> > > > > currently
> > > > > >> > > > taking a
> > > > > >> > > > > > > > Volume
> > > > > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a
> > > managed
> > > > > >> storage
> > > > > >> > > > keeps
> > > > > >> > > > > > it
> > > > > >> > > > > > > on
> > > > > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part
> of
> > > > this
> > > > > >> > > feature,
> > > > > >> > > > we
> > > > > >> > > > > > can
> > > > > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed
> > > storage
> > > > > are
> > > > > >> > > > uploaded
> > > > > >> > > > > > to
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would
> > > make
> > > > > the
> > > > > >> > > Volume
> > > > > >> > > > > > > Snapshot
> > > > > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same
> > > regardless
> > > > of
> > > > > >> the
> > > > > >> > > > > storage
> > > > > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > > > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for
> utilizing
> > > the
> > > > > >> > > efficient
> > > > > >> > > > > > > backend
> > > > > >> > > > > > > > > > > > > > >>>>>>>> storage
> > > > > >> > > > > > > > > > > > > > capabilities, we can use the new
> storage
> > > > > >> snapshots
> > > > > >> > > API.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36
> PM,
> > > > Mike
> > > > > >> > > > Tutkowski <
> > > > > >> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> > > > > >> > <javascript:;>
> > > > > >> > > > > <javascript:;>>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need
> > to
> > > > > have a
> > > > > >> > plan
> > > > > >> > > > to
> > > > > >> > > > > > deal
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we
> already
> > > > have a
> > > > > >> > > feature
> > > > > >> > > > > (in
> > > > > >> > > > > > > 4.6
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> and
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to
> use
> > > the
> > > > > >> > existing
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to
> > create a
> > > > > >> volume
> > > > > >> > > > > snapshot
> > > > > >> > > > > > > (for
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a
> > > backend
> > > > > SAN
> > > > > >> > > > (using a
> > > > > >> > > > > > > > custom
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a
> custom
> > > > data
> > > > > >> > motion
> > > > > >> > > > > > > strategy).
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in,
> then
> > > how
> > > > > >> should
> > > > > >> > > the
> > > > > >> > > > > > > original
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in
> 4.6
> > > and
> > > > > >> later)
> > > > > >> > > be
> > > > > >> > > > > > > changed?
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > > > > >> > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support
> > > > customers
> > > > > >> who
> > > > > >> > > are
> > > > > >> > > > > > > already
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> using
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API
> > to
> > > > take
> > > > > >> > > > snapshots
> > > > > >> > > > > > on a
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > > > > >> > > > > > > > > > > > > SAN?
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27
> > AM,
> > > > > Will
> > > > > >> > > > Stevens <
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > > > > <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create
> a
> > > > > Template
> > > > > >> > > from a
> > > > > >> > > > > > > > > > > > StorageSnapshot?
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template
> be
> > > > stored
> > > > > >> in
> > > > > >> > the
> > > > > >> > > > > > > secondary
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal
> templates
> > > or
> > > > > will
> > > > > >> > that
> > > > > >> > > > be
> > > > > >> > > > > > > > handled
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > >> > > > > > > > > > > > > > vendor side?
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud
> > Solutions
> > > > > >> Experts
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal
> *|*
> > > > Quebec
> > > > > >> *|*
> > > > > >> > > H3J
> > > > > >> > > > > 1S6
> > > > > >> > > > > > w
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw
> > @CloudOps_
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22
> > PM,
> > > > > Syed
> > > > > >> > > > Mushtaq <
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at
> 1:19
> > > PM,
> > > > > Will
> > > > > >> > > > Stevens
> > > > > >> > > > > <
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the
> > > Design
> > > > > >> Spec in
> > > > > >> > > the
> > > > > >> > > > > > Jira
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not
> > > clear
> > > > > in
> > > > > >> the
> > > > > >> > > > > > 'mention'
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows
> > as a
> > > > > page
> > > > > >> > 'you
> > > > > >> > > do
> > > > > >> > > > > not
> > > > > >> > > > > > > > have
> > > > > >> > > > > > > > > > > permission to'.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud
> > > Solutions
> > > > > >> Experts
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal
> > *|*
> > > > > Quebec
> > > > > >> > *|*
> > > > > >> > > > H3J
> > > > > >> > > > > > 1S6
> > > > > >> > > > > > > w
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw
> > > @CloudOps_
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at
> 1:02
> > > PM,
> > > > > >> Syed
> > > > > >> > > Ahmed
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > >
> https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new
> > set
> > > > of
> > > > > >> APIs
> > > > > >> > to
> > > > > >> > > > do
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed
> > storage
> > > > > >> backends
> > > > > >> > > > like
> > > > > >> > > > > > > > > SolidFire.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current
> > managed
> > > > > >> storage
> > > > > >> > > stay
> > > > > >> > > > > on
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary
> > to
> > > > what
> > > > > >> > > > CloudStack
> > > > > >> > > > > > > calls
> > > > > >> > > > > > > > > > > snpshots.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on
> > > > storage
> > > > > >> and
> > > > > >> > > > keeping
> > > > > >> > > > > > it
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own
> > advantages
> > > > > >> > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to
> > > have
> > > > > both
> > > > > >> > ways
> > > > > >> > > > of
> > > > > >> > > > > > > doing
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal
> > > adds
> > > > 4
> > > > > >> new
> > > > > >> > > APIs
> > > > > >> > > > to
> > > > > >> > > > > > > > create
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend
> > > storage.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think
> of
> > > this
> > > > > >> > > feature? I
> > > > > >> > > > > > would
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some
> > feedback.
> > > I
> > > > am
> > > > > >> > > working
> > > > > >> > > > on
> > > > > >> > > > > > > > making
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > >> > > > > > > > > > > > spec
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted
> > to
> > > > > have a
> > > > > >> > high
> > > > > >> > > > > level
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before
> > > starting
> > > > > to
> > > > > >> > work
> > > > > >> > > on
> > > > > >> > > > > it.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> --
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack
> Developer,
> > > > > >> SolidFire
> > > > > >> > > Inc.*
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> e:
> > mike.tutkowski@solidfire.com
> > > > > >> > > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world
> > uses
> > > > the
> > > > > >> > cloud
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> <
> > > > > >> > > > > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > > > > >*™
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> *
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>> --
> > > > > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer,
> > > > SolidFire
> > > > > >> > Inc.*
> > > > > >> > > > > > > > > > > > > > >>>>>> e:
> mike.tutkowski@solidfire.com
> > > > > >> > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > >>>>>> Advancing the way the world
> uses
> > > the
> > > > > >> cloud
> > > > > >> > > > > > > > > > > > > > >>>>>> <
> > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > > > >*™*
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> --
> > > > > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer,
> > > > SolidFire
> > > > > >> Inc.*
> > > > > >> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > > >> > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > >>>>> Advancing the way the world uses
> > the
> > > > > cloud
> > > > > >> > > > > > > > > > > > > > >>>>> <
> > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > > > >*™*
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> --
> > > > > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer,
> > > SolidFire
> > > > > >> Inc.*
> > > > > >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > >>> Advancing the way the world uses
> the
> > > > cloud
> > > > > >> > > > > > > > > > > > > > >>> <
> > > > > >> > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > > >*™*
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > --
> > > > > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > > *Senior CloudStack Developer,
> > SolidFire
> > > > > Inc.*
> > > > > >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > > o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > > Advancing the way the world uses the
> > > cloud
> > > > > >> > > > > > > > > > > > > > > <
> > > > > >> > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > >*™*
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Find out more about ShapeBlue and our
> > > range
> > > > of
> > > > > >> > > > CloudStack
> > > > > >> > > > > > > > > > > > > > related
> > > > > >> > > > > > > > > > > > > services:
> > > > > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > > >> > > > > > > > > > > > > > <
> > > > > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > > > >> > > |
> > > > > >> > > > > > > > CSForge –
> > > > > >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > > > > >> > > > > > > http://shapeblue.com/csforge/
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > > > > > CloudStack Consulting
> > > > > >> > > > > > > > > > > > > > <
> > > > http://shapeblue.com/cloudstack-consultancy/
> > > > > >
> > > > > >> |
> > > > > >> > > > > > > > > > > > > CloudStack
> > > > > >> > > > > > > > > > > > > > Software Engineering
> > > > > >> > > > > > > > > > > > > > <
> > > > > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > > > > >> > > > >
> > > > > >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > > > > >> > > > > > > > > > > > > > <
> > > > > >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > >> > > > > >
> > > > > >> > > > > > |
> > > > > >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > >> > > > > > > > > > > > > > <
> > > http://shapeblue.com/cloudstack-training/>
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > --
> > > > > >> > > > > > > > > > > > *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> > Inc.*
> > > > > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > o: 303.746.7302
> > > > > >> > > > > > > > > > > > Advancing the way the world uses the cloud
> > > > > >> > > > > > > > > > > > <
> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > >*™*
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > Find out more about ShapeBlue and our range
> of
> > > > > >> CloudStack
> > > > > >> > > > > related
> > > > > >> > > > > > > > > > services:
> > > > > >> > > > > > > > > > > IaaS Cloud Design & Build
> > > > > >> > > > > > > > > > > <
> > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > >
> > > > > >> |
> > > > > >> > > > > CSForge –
> > > > > >> > > > > > > > rapid
> > > > > >> > > > > > > > > > > IaaS deployment framework <
> > > > > >> http://shapeblue.com/csforge/
> > > > > >> > >
> > > > > >> > > > > > > CloudStack
> > > > > >> > > > > > > > > > > Consulting <
> > > > > >> http://shapeblue.com/cloudstack-consultancy/
> > > > > >> > >
> > > > > >> > > |
> > > > > >> > > > > > > > CloudStack
> > > > > >> > > > > > > > > > > Software Engineering
> > > > > >> > > > > > > > > > > <
> > > > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > > > >> > > > > > > > > > > CloudStack Infrastructure Support
> > > > > >> > > > > > > > > > > <
> > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > >> > >
> > > > > >> > > |
> > > > > >> > > > > > > > CloudStack
> > > > > >> > > > > > > > > > > Bootcamp Training Courses <
> > > > > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > Find out more about ShapeBlue and our range of
> > > > > >> CloudStack
> > > > > >> > > > related
> > > > > >> > > > > > > > > services:
> > > > > >> > > > > > > > > > IaaS Cloud Design & Build
> > > > > >> > > > > > > > > > <
> > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > >
> > > > > |
> > > > > >> > > > CSForge –
> > > > > >> > > > > > > rapid
> > > > > >> > > > > > > > > > IaaS deployment framework <
> > > > > >> http://shapeblue.com/csforge/>
> > > > > >> > > > > > > > > > CloudStack Consulting <
> > > > > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > > >> > > > > > > |
> > > > > >> > > > > > > > > CloudStack
> > > > > >> > > > > > > > > > Software Engineering
> > > > > >> > > > > > > > > > <
> > > > > http://shapeblue.com/cloudstack-software-engineering/>
> > > > > >> > > > > > > > > > CloudStack Infrastructure Support
> > > > > >> > > > > > > > > > <
> > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > > > >> > |
> > > > > >> > > > > > > CloudStack
> > > > > >> > > > > > > > > > Bootcamp Training Courses <
> > > > > >> > > > > > http://shapeblue.com/cloudstack-training/
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > --
> > > > > >> > > > > > > *Mike Tutkowski*
> > > > > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > <javascript:;>
> > > > > >> > > > > > > o: 303.746.7302
> > > > > >> > > > > > > Advancing the way the world uses the cloud
> > > > > >> > > > > > > <http://solidfire.com/solution/overview/?video=play
> > >*™*
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > --
> > > > > >> > > > > *Mike Tutkowski*
> > > > > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > >> > > > > o: 303.746.7302
> > > > > >> > > > > Advancing the way the world uses the cloud
> > > > > >> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > --
> > > > > >> > > > Ian Rae
> > > > > >> > > > CEO | PDG
> > > > > >> > > > c: 514.944.4008
> > > > > >> > > >
> > > > > >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> > > > > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J
> > 1S6
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > *Mike Tutkowski*
> > > > > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >> > e: mike.tutkowski@solidfire.com
> > > > > >> > o: 303.746.7302
> > > > > >> > Advancing the way the world uses the cloud
> > > > > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > e: mike.tutkowski@solidfire.com
> > > > > > o: 303.746.7302
> > > > > > Advancing the way the world uses the cloud
> > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack
> Software Engineering
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support
> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

RE: [Propose][New Feature] Storage Snapshots

[Paul Angus <pa...@shapeblue.com>]

Just to make sure I'm on the same page, are we talking about; https://issues.apache.org/jira/browse/CLOUDSTACK-9278 ?

The FS reads (to me) more like 1a + the possibility to export to secondary storage if required?
Have I understood correctly?
I have seen [1a] implemented for VMware by NetApp in their beta CloudStack plugin (pleased I can say that without Mike beating me up now). No changes to the CloudStack API were required. (nb it didn't export to secondary storage).



1. VM Snapshot (point-in-time hypervisor based snapshots)
1a. SAN assisted VM snapshots (point-in-time hypervisor snapshot takes place on transparently SAN to avoid performance issue in disk chains)
2. SAN Snapshot (Storage Snapshot) - NEW
3. Volume Snapshot (current old/slow transfer to secstorage)
4. Backup - JUST AN IDEAL.




[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology   ,       ShapeBlue


d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>     |      m:      +44 7711 418784<tel:+44%207711%20418784>

e:      paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus>      |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:image36de3e.png@6c252406.4da4f02d]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




-----Original Message-----
From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
Sent: Monday, February 8, 2016 7:16 PM
To: dev@cloudstack.apache.org
Subject: Re: [Propose][New Feature] Storage Snapshots

Correct, Will.

That Global Settings would only be for managed storage. Non-managed
(traditional) volume snapshots are completely un-impacted by this feature.

If we need to sometimes keep the snapshots on the SAN and sometimes push
them to secondary storage, we'll need a more robust solution than Global
Settings, though.

On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <ws...@cloudops.com> wrote:

> Sorry. I missed a bit of context when I responded. The global setting
> would be only for the managed storage case, currently being called Storage
> Snapshots, and is only to determine if a copy is pushed to secondary
> storage right? The global setting would not change the behavior of the
> Volume Snapshots right?
>
> I was referring to the need for Volume Snapshots and Storage Snapshots to
> exist together. Disregard my comment. I caught up on context after I
> posted. My bad...
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com
> > wrote:
>
> > Hey Will,
> >
> > Who's picking the behavior? Is it the cloud provider or the end user?
> >
> > Thanks
> >
> > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > I don't think a global setting is a good option because we need both
> > > functionality to be available at the same time and for different use
> > cases
> > > to be able to pick which they choose.
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > *CloudOps* *| *Cloud Solutions Experts
> > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > w cloudops.com *|* tw @CloudOps_
> > >
> > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com
> > > > wrote:
> > >
> > > > Now that I re-read your e-mail, it dawned on me: The end-user doesn't
> > > care
> > > > where the snapshot is.
> > > >
> > > > If that's true, then we should perhaps control this via Global
> Settings
> > > or
> > > > something.
> > > >
> > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com> wrote:
> > > >
> > > > > It's not ideal - true, but it does allow us to be backward
> > compatible.
> > > > >
> > > > > If you have other ideas, though, about how to maintain backward
> > > > > compatibility, I'm definitely open to hear them.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > syed1.mushtaq@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Mike,
> > > > >>
> > > > >> Adding a flag to createSnapshot was the first and the most obvious
> > > thing
> > > > >> that came to our minds. The problem that I had with this was that:
> > > > >>
> > > > >> 1) I feel it is exposing something to the end user that is
> internal
> > to
> > > > the
> > > > >> cloud.
> > > > >>
> > > > >> 2) We have to follow two different ways of restore/deletion in the
> > > same
> > > > >> code path depending on where the Snapshot resides which I find
> kind
> > > of a
> > > > >> bad design.
> > > > >>
> > > > >> But if exposing a archive flag to the end user is acceptable then
> we
> > > can
> > > > >> definitely use this instead of adding the StorageSnapshot API
> > > > >>
> > > > >> Thanks,
> > > > >> -Syed
> > > > >>
> > > > >>
> > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > >> mike.tutkowski@solidfire.com
> > > > >> > wrote:
> > > > >>
> > > > >> > Hi Pierre-Luc,
> > > > >> >
> > > > >> > My recommendation would be this:
> > > > >> >
> > > > >> > Add an "archive" flag to the current volume-snapshot API. Its
> > > default
> > > > >> would
> > > > >> > be "false" because that would be backward compatible with how
> 4.6
> > > has
> > > > >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6,
> > 4.7,
> > > > and
> > > > >> > 4.8).
> > > > >> >
> > > > >> > If you set archive=true, then we would perform a background
> > > migration
> > > > of
> > > > >> > the snapshot from the SAN to the secondary storage (then delete
> > the
> > > > SAN
> > > > >> > snapshot).
> > > > >> >
> > > > >> > That archive parameter would only be valid for managed storage.
> > > > >> >
> > > > >> > Sound reasonable?
> > > > >> >
> > > > >> > Also, a VM snapshot that includes disks provided by managed
> > storage
> > > > >> should
> > > > >> > work.
> > > > >> >
> > > > >> > Talk to you later,
> > > > >> > Mike
> > > > >> >
> > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > pdion@cloudops.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Mike,
> > > > >> > >
> > > > >> > > In terms of API's, would you prefer introducing a parameter to
> > the
> > > > >> > existing
> > > > >> > > VolumeSnapshot, example: extract={true|false} with a
> default
> > > > value
> > > > >> of
> > > > >> > > true which would extract snapshot into the secondary storage,
> > > which
> > > > >> is
> > > > >> > the
> > > > >> > > current default behavior. Then for SAN snapshot that remain on
> > the
> > > > >> SAN we
> > > > >> > > would just set "extract=false" ? as oppose to create a new
> > > > >> > > StorageSnapshot API ?
> > > > >> > >
> > > > >> > >
> > > > >> > > Paul,
> > > > >> > >
> > > > >> > > From what I'm seeing so far, we can't do a VM-snapshot when
> > using
> > > > >> managed
> > > > >> > > storage for VM having more than one Volume. For the reason
> that
> > > > >> snapshot
> > > > >> > > are performed outside of the hypervisor awareness and
> > > > asynchronously.
> > > > >> If
> > > > >> > > someone have a way to address that, it would make thinks much
> > more
> > > > >> > > attractive.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com>
> > > wrote:
> > > > >> > >
> > > > >> > > > I think a service provider backup scenario is more likely to
> > > take
> > > > >> > > advantage
> > > > >> > > > of SAN snapshot. There are a few reasons for this.
> Traditional
> > > > >> backups
> > > > >> > > > involve access to the file system, and there is an
> expectation
> > > > that
> > > > >> > this
> > > > >> > > > can be done with reasonably short time frames without
> > negatively
> > > > >> > > impacting
> > > > >> > > > VM performance, and that the backup orchestrator can apply
> > > various
> > > > >> > logic
> > > > >> > > > and or transformations to the data (compress, encrypt,
> deltas
> > > > >> etc...).
> > > > >> > > > While it is true that one could apply a backup process to a
> > > cloud
> > > > >> > > snapshot,
> > > > >> > > > this would be slow and inefficient requiring the data to be
> > > moved
> > > > >> > several
> > > > >> > > > times and there are some major bottlenecks with cloud
> > snapshots.
> > > > >> With a
> > > > >> > > > cloud snapshot - there seems to be no reasonable expectation
> > of
> > > > >> being
> > > > >> > > able
> > > > >> > > > to do differential snapshots (I think this depends on the
> > > > >> hypervisor)
> > > > >> > and
> > > > >> > > > if you do differential snapshots this will make file backups
> > > from
> > > > >> those
> > > > >> > > > snapshots even more complicated to orchestrate.
> > > > >> > > >
> > > > >> > > > Suspect there needs to be a different thread on how to
> better
> > > > enable
> > > > >> > > > backups, as a service. As per Paul's suggestion, but it is a
> > > > related
> > > > >> > > > workflow so relevant to this discussion.
> > > > >> > > >
> > > > >> > > > Ian
> > > > >> > > >
> > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > >> > > mike.tutkowski@solidfire.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > To me it sounds like number two and number three are
> > different
> > > > >> uses
> > > > >> > for
> > > > >> > > > the
> > > > >> > > > > same "thing"(which is totally fine).
> > > > >> > > > >
> > > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > > >> asynchronously, do
> > > > >> > > we
> > > > >> > > > > see the SSVM as performing the export?
> > > > >> > > > >
> > > > >> > > > > To be backwards compatible with what we have in 4.6 and
> > later
> > > > for
> > > > >> > > volume
> > > > >> > > > > snapshots for managed storage, I think it might be easier
> if
> > > we
> > > > >> pass
> > > > >> > > in a
> > > > >> > > > > flag that says whether or not to archive the SAN snapshot
> > > > (which,
> > > > >> I
> > > > >> > > > think,
> > > > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > > > >> > > > >
> > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > pdion@cloudops.com
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi Mike,
> > > > >> > > > > >
> > > > >> > > > > > The reason behind the creation of a SAN snapshot which
> is
> > > > >> exported
> > > > >> > > into
> > > > >> > > > > > secondary storage, is because creating a copy of the
> .VHD
> > > > >> directly
> > > > >> > > > would
> > > > >> > > > > > impact uptime of the VM as creating that copy take lots
> of
> > > > time.
> > > > >> > Has
> > > > >> > > > > oppose
> > > > >> > > > > > to a SAN snapshot that is close to instantaneous which
> can
> > > > >> > afterward
> > > > >> > > be
> > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > >> > > > > >
> > > > >> > > > > > I would suspect an extracted VolumeSnapshot taken from a
> > SAN
> > > > >> > snapshot
> > > > >> > > > > could
> > > > >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> > > > >> consumption
> > > > >> > > on
> > > > >> > > > > the
> > > > >> > > > > > Primary Storage side.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > I see 3 definitions in our current discussion regarding
> > the
> > > > term
> > > > >> > > > snapshot
> > > > >> > > > > > (these are not official terminology but by own
> > > interpretation
> > > > of
> > > > >> > > them):
> > > > >> > > > > >
> > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition
> > of
> > > a
> > > > >> > > > snapshot):
> > > > >> > > > > > it's a volume snapshot at the storage level, point in
> time
> > > of
> > > > >> your
> > > > >> > > > data.
> > > > >> > > > > it
> > > > >> > > > > > reside on the primary storage. Useful and efficient for
> > > > software
> > > > >> > side
> > > > >> > > > > > incident.
> > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/
> > cloud
> > > > >> backup
> > > > >> > > > aws-S3
> > > > >> > > > > > style ): Point in time copy of the Virtual Disk that
> > reside
> > > > on a
> > > > >> > > > > different
> > > > >> > > > > > storage array then the original Volume. Facilitate data
> > > > >> migration
> > > > >> > > > between
> > > > >> > > > > > clusters and, in case of primary storage incident,
> Volume
> > > > >> snapshots
> > > > >> > > are
> > > > >> > > > > not
> > > > >> > > > > > impacted and can be reuse.
> > > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that
> > > also
> > > > >> > > validate
> > > > >> > > > > data
> > > > >> > > > > > integrity, provide a storage efficient archiving method
> > and
> > > an
> > > > >> > > > > independent
> > > > >> > > > > > way to restore your data in case of an major
> > infrastructure
> > > > >> > disaster.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Regards,
> > > > >> > > > > >
> > > > >> > > > > > PL
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>
> > > > >> > > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > So, let's see if I currently follow the requirements:
> > > > >> > > > > > >
> > > > >> > > > > > > * Augment volume snapshots for managed storage to
> > > > >> conditionally
> > > > >> > > > export
> > > > >> > > > > > data
> > > > >> > > > > > > to NFS. The current process of taking a snapshot on
> the
> > > SAN
> > > > is
> > > > >> > > fine,
> > > > >> > > > > but
> > > > >> > > > > > > we'd like the option to export the data to NFS, as
> well.
> > > > >> > > > > > >
> > > > >> > > > > > > Questions:
> > > > >> > > > > > >
> > > > >> > > > > > > Once the data has been exported to NFS, do we keep the
> > SAN
> > > > >> > snapshot
> > > > >> > > > or
> > > > >> > > > > > > delete it?
> > > > >> > > > > > >
> > > > >> > > > > > > If we are deleting the SAN snapshot, then why don't we
> > > just
> > > > >> copy
> > > > >> > > the
> > > > >> > > > > VHD
> > > > >> > > > > > > from primary to secondary the way we do today for
> > > > non-managed
> > > > >> > (i.e.
> > > > >> > > > > > > traditional) storage? Why create a SAN snapshot in
> that
> > > > >> scenario?
> > > > >> > > > > Perhaps
> > > > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> > > secondary
> > > > >> > > storage
> > > > >> > > > > > > instead of a XenServer host?
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks for the clarification.
> > > > >> > > > > > >
> > > > >> > > > > > > By the way, to me a backup is when you copy data from
> > one
> > > > >> storage
> > > > >> > > > > system
> > > > >> > > > > > to
> > > > >> > > > > > > another (regardless of features, if any, to restore
> the
> > > data
> > > > >> in
> > > > >> > the
> > > > >> > > > > > > future). A snapshot is a point-in-time view of the
> data
> > > of a
> > > > >> > volume
> > > > >> > > > and
> > > > >> > > > > > > it's stored on the same storage system as the volume.
> > > > >> > > > > > >
> > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > > >> > > pdion@cloudops.com
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > <javascript:;>>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > That's fun to see that discussion happening. I 100%
> > > agree
> > > > >> with
> > > > >> > > > Paul's
> > > > >> > > > > > > > points of view. VolumeSnapshot are not a backup,
> but I
> > > do
> > > > >> > > consider
> > > > >> > > > > them
> > > > >> > > > > > > as
> > > > >> > > > > > > > a safety vest against Primary Storage failure,
> because
> > > > >> failure
> > > > >> > > > append
> > > > >> > > > > > > :-( .
> > > > >> > > > > > > >
> > > > >> > > > > > > > The current proposal around snapshots that reside on
> > the
> > > > >> > primary
> > > > >> > > > > > storage
> > > > >> > > > > > > or
> > > > >> > > > > > > > snapshots that end in the Secondary Storage is not
> to
> > > > >> address
> > > > >> > > any
> > > > >> > > > > kind
> > > > >> > > > > > > of
> > > > >> > > > > > > > backups requirement because a snapshot is not a
> > backup,
> > > > >> event
> > > > >> > an
> > > > >> > > > > > > extracted
> > > > >> > > > > > > > VM snapshot.
> > > > >> > > > > > > >
> > > > >> > > > > > > > The main idea, and again this is for managed
> storage;
> > > > >> > > > > > > >
> > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> > > > >> capability
> > > > >> > > for
> > > > >> > > > > > fast
> > > > >> > > > > > > > response time that support rollback to previous
> > > timestamp,
> > > > >> > create
> > > > >> > > > new
> > > > >> > > > > > > > volume and maybe create template.
> > > > >> > > > > > > > not required to be a new API if the work is
> > already
> > > > >> done, I
> > > > >> > > > think
> > > > >> > > > > > > this
> > > > >> > > > > > > > is a different behaviors than the user expectation
> of
> > a
> > > > >> > > > > > volume-snapshot.
> > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> > > behavior
> > > > >> that
> > > > >> > > > create
> > > > >> > > > > > an
> > > > >> > > > > > > > extraction of a volume into SecondaryStorage which
> can
> > > be
> > > > >> reuse
> > > > >> > > to
> > > > >> > > > > > > create a
> > > > >> > > > > > > > new volume into another Primary Storage. This type
> of
> > > > >> snapshot
> > > > >> > > is a
> > > > >> > > > > > slow
> > > > >> > > > > > > > job since yes it would have to copy the full volume
> > size
> > > > on
> > > > >> the
> > > > >> > > > > > Secondary
> > > > >> > > > > > > > storage.
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > PL
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > >> > > > > > <javascript:;>>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > I think I share you view on the 'Ideal world'.
> > Backup
> > > > (via
> > > > >> > > Volume
> > > > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack.
> This
> > is
> > > > >> > > amplified
> > > > >> > > > > > > > especially
> > > > >> > > > > > > > > when you have a object storage as your secondary
> > > storage
> > > > >> > > because
> > > > >> > > > it
> > > > >> > > > > > > > > requires two copies (one to an NFS staging area
> and
> > > from
> > > > >> > there
> > > > >> > > to
> > > > >> > > > > > > object
> > > > >> > > > > > > > > storage). And not to mention that all these copies
> > are
> > > > >> > > consuming
> > > > >> > > > > > > > hypervisor
> > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> > bottleneck
> > > as
> > > > >> all
> > > > >> > > the
> > > > >> > > > > > > Network
> > > > >> > > > > > > > > and I/O flow through it. So our intention of
> > proposing
> > > > the
> > > > >> > > > "Storage
> > > > >> > > > > > > > > Snapshots" is to give a better way of achiving
> > > snapshots
> > > > >> > while
> > > > >> > > > > still
> > > > >> > > > > > > > > keeping the original definition of volume
> snpashots
> > > (ie
> > > > >> > upload
> > > > >> > > to
> > > > >> > > > > sec
> > > > >> > > > > > > > > storage).
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > > > backups.
> > > > >> > They
> > > > >> > > > > don't
> > > > >> > > > > > > work
> > > > >> > > > > > > > > form multi-disk LVM volume groups and dynamic
> > disks. I
> > > > am
> > > > >> all
> > > > >> > > in
> > > > >> > > > > for
> > > > >> > > > > > a
> > > > >> > > > > > > > > better backup solution which handles these use
> cases
> > > and
> > > > >> > > utilizes
> > > > >> > > > > the
> > > > >> > > > > > > > > storage's advanced features.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > > >> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > In the beginning... there were CloudStack
> > snapshots
> > > > and
> > > > >> > they
> > > > >> > > > were
> > > > >> > > > > > > > > actually
> > > > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > > > snapshots.
> > > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > > point-in-time
> > > > >> > > > > hypervisor
> > > > >> > > > > > > > > > snapshots) and we started referring to the
> > original
> > > > >> > snapshots
> > > > >> > > > as
> > > > >> > > > > > > volume
> > > > >> > > > > > > > > > snapshots.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > CloudStack does not offer 'backups', but many
> > people
> > > > use
> > > > >> > > volume
> > > > >> > > > > > > > snapshots
> > > > >> > > > > > > > > > as backups. However you can't in-place restore
> > > volume
> > > > >> > > snapshots
> > > > >> > > > > and
> > > > >> > > > > > > if
> > > > >> > > > > > > > > you
> > > > >> > > > > > > > > > have a VM with multiple volumes, the volume
> > > snapshots
> > > > >> must
> > > > >> > be
> > > > >> > > > > done
> > > > >> > > > > > in
> > > > >> > > > > > > > > > series, meaning that the state across all of the
> > > > >> volumes is
> > > > >> > > > > > unlikely
> > > > >> > > > > > > to
> > > > >> > > > > > > > > be
> > > > >> > > > > > > > > > consistent.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> > > > options
> > > > >> > > which
> > > > >> > > > > > users
> > > > >> > > > > > > > > might
> > > > >> > > > > > > > > > expect as well options as to where they might be
> > > > >> stored. In
> > > > >> > > my
> > > > >> > > > > > ideal
> > > > >> > > > > > > > > world
> > > > >> > > > > > > > > > they would also be able to leverage back-end
> > > hardware
> > > > >> (such
> > > > >> > > as
> > > > >> > > > > > > > Solidfire,
> > > > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> > > Commvault
> > > > >> etc
> > > > >> > to
> > > > >> > > > > > > accelerate
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > process.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > >> > > > > > > > > > Paul Angus
> > > > >> > > > > > > > > > VP Technology , ShapeBlue
> > > > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > >> > > > > > > > > >
> > > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> |
> > > > >> > > > > m:
> > > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > >> > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > > >> > <javascript:;>
> > > > >> > > |
> > > > >> > > > > t: @cloudyangus*
> > > > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > >> > > > > <ja...@cloudyangus>
> > > > >> > > > > > | w:
> > > > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> > 4HS
> > > UK
> > > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in
> > England
> > > &
> > > > >> > Wales.
> > > > >> > > > > > > ShapeBlue
> > > > >> > > > > > > > > > Services India LLP is a company incorporated in
> > > India
> > > > >> and
> > > > >> > is
> > > > >> > > > > > operated
> > > > >> > > > > > > > > under
> > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > > >> Consultoria
> > > > >> > > Ltda
> > > > >> > > > > is
> > > > >> > > > > > a
> > > > >> > > > > > > > > > company incorporated in Brasil and is operated
> > under
> > > > >> > license
> > > > >> > > > from
> > > > >> > > > > > > Shape
> > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > registered
> > > > >> by
> > > > >> > The
> > > > >> > > > > > > Republic
> > > > >> > > > > > > > of
> > > > >> > > > > > > > > > South Africa and is traded under license from
> > Shape
> > > > Blue
> > > > >> > Ltd.
> > > > >> > > > > > > ShapeBlue
> > > > >> > > > > > > > > is
> > > > >> > > > > > > > > > a registered trademark.
> > > > >> > > > > > > > > > This email and any attachments to it may be
> > > > confidential
> > > > >> > and
> > > > >> > > > are
> > > > >> > > > > > > > intended
> > > > >> > > > > > > > > > solely for the use of the individual to whom it
> is
> > > > >> > addressed.
> > > > >> > > > Any
> > > > >> > > > > > > views
> > > > >> > > > > > > > > or
> > > > >> > > > > > > > > > opinions expressed are solely those of the
> author
> > > and
> > > > do
> > > > >> > not
> > > > >> > > > > > > > necessarily
> > > > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > > > companies.
> > > > >> If
> > > > >> > > you
> > > > >> > > > > are
> > > > >> > > > > > > not
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > intended recipient of this email, you must
> neither
> > > > take
> > > > >> any
> > > > >> > > > > action
> > > > >> > > > > > > > based
> > > > >> > > > > > > > > > upon its contents, nor copy or show it to
> anyone.
> > > > Please
> > > > >> > > > contact
> > > > >> > > > > > the
> > > > >> > > > > > > > > sender
> > > > >> > > > > > > > > > if you believe you have received this email in
> > > error.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > -----Original Message-----
> > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > syed1.mushtaq@gmail.com
> > > > >> > > > > <javascript:;> <javascript:;>]
> > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > >> > <javascript:;>
> > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > Snapshots
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Paul,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > When you say actual backups, how would it be
> > > different
> > > > >> from
> > > > >> > > the
> > > > >> > > > > > > Volume
> > > > >> > > > > > > > > > Snapshots that exist currently. My understanding
> > is
> > > > that
> > > > >> > > > Backups
> > > > >> > > > > > end
> > > > >> > > > > > > up
> > > > >> > > > > > > > > in
> > > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > > point-in-time
> > > > >> > state
> > > > >> > > of
> > > > >> > > > > > your
> > > > >> > > > > > > > > volume
> > > > >> > > > > > > > > > which can be restored back correct?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > -Syed
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> <javascript:;>>
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > Hi Syed,
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > As I understand it, the SolidFire plugin will
> > > export
> > > > >> the
> > > > >> > > > > snapshot
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > > secondary storage if the user requests a
> > template
> > > > from
> > > > >> > the
> > > > >> > > > > > snapshot
> > > > >> > > > > > > > or
> > > > >> > > > > > > > > > > wants to download the snapshot from the cloud.
> > > This
> > > > >> is a
> > > > >> > > > good,
> > > > >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> > > > storage
> > > > >> is
> > > > >> > > > super
> > > > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays
> take
> > up
> > > > >> next
> > > > >> > to
> > > > >> > > no
> > > > >> > > > > > > space.
> > > > >> > > > > > > > > > > BUT I think that we are talking about a more
> > > general
> > > > >> > > purpose
> > > > >> > > > > API,
> > > > >> > > > > > > and
> > > > >> > > > > > > > > > > other storage systems may not be as awesome as
> > > > Mike's.
> > > > >> > > That's
> > > > >> > > > > my
> > > > >> > > > > > > > > > > concern. Also, the time to transfer for say
> 1TB
> > to
> > > > >> move
> > > > >> > > from
> > > > >> > > > > > > primary
> > > > >> > > > > > > > > > > to sec storage and then create a VM template
> out
> > > of
> > > > it
> > > > >> > may
> > > > >> > > be
> > > > >> > > > > too
> > > > >> > > > > > > > long
> > > > >> > > > > > > > > > for users.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > > > snapshot
> > > > >> for
> > > > >> > > > > backup,
> > > > >> > > > > > > > it's
> > > > >> > > > > > > > > > > just that users want to do backups and a
> volume
> > > > >> snapshot
> > > > >> > is
> > > > >> > > > the
> > > > >> > > > > > > only
> > > > >> > > > > > > > > > > type of snapshot that copies the disk
> elsewhere
> > > and
> > > > >> can
> > > > >> > be
> > > > >> > > > > > > scheduled.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> > > actual
> > > > >> > backups
> > > > >> > > > > > > (through
> > > > >> > > > > > > > > > > recognised backup providers) and the user
> > > > requirements
> > > > >> > > around
> > > > >> > > > > > them
> > > > >> > > > > > > > > > > (particularly restoration use cases) as a
> > separate
> > > > >> thread
> > > > >> > > of
> > > > >> > > > > > work.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > Paul
> > > > >> Angus
> > > > >> > > VP
> > > > >> > > > > > > > Technology
> > > > >> > > > > > > > > > > , ShapeBlue
> > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > >> > > > > > > > > > >
> > > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > >> > > > |
> > > > >> > > > > m:
> > > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > > >> > > <javascript:;> |
> > > > >> > > > > t: @cloudyangus*
> > > > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > >> <javascript:;>
> > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com
> >
> > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> > 4HS
> > > > UK
> > > > >> > Shape
> > > > >> > > > > Blue
> > > > >> > > > > > > Ltd
> > > > >> > > > > > > > > > > is a company incorporated in England & Wales.
> > > > >> ShapeBlue
> > > > >> > > > > Services
> > > > >> > > > > > > > India
> > > > >> > > > > > > > > > > LLP is a company incorporated in India and is
> > > > operated
> > > > >> > > under
> > > > >> > > > > > > license
> > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> > Consultoria
> > > > >> Ltda
> > > > >> > is
> > > > >> > > a
> > > > >> > > > > > > company
> > > > >> > > > > > > > > > > incorporated in Brasil and is operated under
> > > license
> > > > >> from
> > > > >> > > > Shape
> > > > >> > > > > > > Blue
> > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> > registered
> > > by
> > > > >> The
> > > > >> > > > > Republic
> > > > >> > > > > > > of
> > > > >> > > > > > > > > > > South Africa and is traded under license from
> > > Shape
> > > > >> Blue
> > > > >> > > Ltd.
> > > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > >> > > > > > > > > > > This email and any attachments to it may be
> > > > >> confidential
> > > > >> > > and
> > > > >> > > > > are
> > > > >> > > > > > > > > > > intended solely for the use of the individual
> to
> > > > whom
> > > > >> it
> > > > >> > is
> > > > >> > > > > > > > addressed.
> > > > >> > > > > > > > > > > Any views or opinions expressed are solely
> those
> > > of
> > > > >> the
> > > > >> > > > author
> > > > >> > > > > > and
> > > > >> > > > > > > do
> > > > >> > > > > > > > > > > not necessarily represent those of Shape Blue
> > Ltd
> > > or
> > > > >> > > related
> > > > >> > > > > > > > > > > companies. If you are not the intended
> recipient
> > > of
> > > > >> this
> > > > >> > > > email,
> > > > >> > > > > > you
> > > > >> > > > > > > > > > > must neither take any action based upon its
> > > > contents,
> > > > >> nor
> > > > >> > > > copy
> > > > >> > > > > or
> > > > >> > > > > > > > show
> > > > >> > > > > > > > > > > it to anyone. Please contact the sender if you
> > > > believe
> > > > >> > you
> > > > >> > > > have
> > > > >> > > > > > > > > received
> > > > >> > > > > > > > > > this email in error.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > -----Original Message-----
> > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > syed1.mushtaq@gmail.com
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > <javascript:;>]
> > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > >> > > <javascript:;>
> > > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > > Snapshots
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > I think the terminology confusion comes from
> AWS
> > > > where
> > > > >> > they
> > > > >> > > > do
> > > > >> > > > > > EBS
> > > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort
> of
> > > > >> followed
> > > > >> > > > that.
> > > > >> > > > > > And
> > > > >> > > > > > > > as
> > > > >> > > > > > > > > > > an end user who is oblivious to the internals
> of
> > > my
> > > > >> > > provider,
> > > > >> > > > > my
> > > > >> > > > > > > > > > > expectation would be something similar to what
> > AWS
> > > > as
> > > > >> > that
> > > > >> > > is
> > > > >> > > > > my
> > > > >> > > > > > > > > > > biggest reference point.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > To your point Mike, I agree that a Primary
> > Storage
> > > > >> > failure
> > > > >> > > on
> > > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> > motivations
> > > > >> for us
> > > > >> > > to
> > > > >> > > > > push
> > > > >> > > > > > > > data
> > > > >> > > > > > > > > > > to secondary storage. Primary storage (atleast
> > for
> > > > us)
> > > > >> > > costs
> > > > >> > > > > > > around 3
> > > > >> > > > > > > > > > > times as much as secondary storage and
> snapshots
> > > on
> > > > >> > primary
> > > > >> > > > > > storage
> > > > >> > > > > > > > > > > are rarely used (especially for some of our
> > > > customers
> > > > >> who
> > > > >> > > do
> > > > >> > > > > > daily
> > > > >> > > > > > > > > > backups).
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike
> Tutkowski
> > <
> > > > >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > >> > > <javascript:;>>
> > > > >> > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > Some of the weirdness is around terminology.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > For most systems I've worked on, a snapshot
> > and
> > > a
> > > > >> > backup
> > > > >> > > > are
> > > > >> > > > > > two
> > > > >> > > > > > > > > > > > completely different things (but CloudStack
> > has
> > > > >> > > > traditionally
> > > > >> > > > > > > used
> > > > >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and
> say,
> > > > >> though,
> > > > >> > > that
> > > > >> > > > > if
> > > > >> > > > > > > your
> > > > >> > > > > > > > > > > > primary storage is running on SolidFire that
> > it
> > > is
> > > > >> > > unlikely
> > > > >> > > > > > > you'll
> > > > >> > > > > > > > > > > > encounter an issue where your primary
> storage
> > > goes
> > > > >> > > offline
> > > > >> > > > > (and
> > > > >> > > > > > > > > > > > you'll even maintain your performance
> > guarantees
> > > > >> during
> > > > >> > > > > failure
> > > > >> > > > > > > > > > > > scenarios and upgrades, as well). That being
> > the
> > > > >> case,
> > > > >> > it
> > > > >> > > > is
> > > > >> > > > > > less
> > > > >> > > > > > > > > > > > useful to require a backup to Swift (but
> it's
> > > > >> perfectly
> > > > >> > > OK
> > > > >> > > > if
> > > > >> > > > > > > > that's
> > > > >> > > > > > > > > > > > what we want to do
> > > > >> > > > > > > > > > > here).
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > > > >> > <javascript:;>>
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Paul,
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > I believe with the current implementation
> of
> > > > >> > Snapshots
> > > > >> > > on
> > > > >> > > > > > > managed
> > > > >> > > > > > > > > > > > > storage
> > > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never
> exported
> > > to
> > > > >> the
> > > > >> > > > > secondary
> > > > >> > > > > > > > > > storage.
> > > > >> > > > > > > > > > > > > While this solves the problem of having
> > > > snapshots
> > > > >> > > taking
> > > > >> > > > > > > forever
> > > > >> > > > > > > > > > > > > to get to sec storage, this leaves us
> with a
> > > > >> > > > > > > > > > > > huge
> > > > >> > > > > > > > > > > > > liability if our primary storage goes
> down.
> > We
> > > > see
> > > > >> > > > > snapshots
> > > > >> > > > > > as
> > > > >> > > > > > > > > > > > > our recovery path because we store them in
> > > Swift
> > > > >> > which
> > > > >> > > is
> > > > >> > > > > > > > reliable
> > > > >> > > > > > > > > > > > > and resilient to failures.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> > > > Volume
> > > > >> > > > snapshots
> > > > >> > > > > > > > always
> > > > >> > > > > > > > > > > > > backed up to secondary storage and Storage
> > > > >> Snapshots
> > > > >> > > stay
> > > > >> > > > > on
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > > primary
> > > > >> > > > > > > > > > > > storage.
> > > > >> > > > > > > > > > > > > A provider could potentially mix both
> these
> > > and
> > > > >> solve
> > > > >> > > the
> > > > >> > > > > > > problem
> > > > >> > > > > > > > > > > > > that you mentioned where you want to meet
> > > user's
> > > > >> > > > > expectation
> > > > >> > > > > > > of a
> > > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while
> > > having
> > > > >> an
> > > > >> > > > ability
> > > > >> > > > > > to
> > > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the
> > device)
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > > > > -Syed
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > >> > > <javascript:;>>
> > > > >> > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > HI guys,
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Could someone point me to the Jira bug
> of
> > FS
> > > > for
> > > > >> > the
> > > > >> > > > > > > > > > > > > > SAN-snapshot
> > > > >> > > > > > > > > > > > feature
> > > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > From my discussions with users and
> > operators
> > > > >> around
> > > > >> > > > > > snapshots
> > > > >> > > > > > > > > > > > > > I'd make
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > following observations:
> > > > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups
> (both
> > > > >> long-term
> > > > >> > > and
> > > > >> > > > > > short
> > > > >> > > > > > > > > > > > > > term)
> > > > >> > > > > > > > > > > > with
> > > > >> > > > > > > > > > > > > > the expectation that they can use them
> for
> > > > >> recovery
> > > > >> > > if
> > > > >> > > > > > > > required.
> > > > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > > > something
> > > > >> > has
> > > > >> > > > gone
> > > > >> > > > > > > wrong
> > > > >> > > > > > > > > > > > > > with primary storage.
> > > > >> > > > > > > > > > > > > > c. users sometimes want to be able to
> > export
> > > > >> > > snapshots
> > > > >> > > > as
> > > > >> > > > > > > well
> > > > >> > > > > > > > > > > > > > as
> > > > >> > > > > > > > > > > > create
> > > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > > >> > > > > > > > > > > > > > d. snapshots are a currently a massive
> > pain
> > > > for
> > > > >> > > > > operators,
> > > > >> > > > > > I
> > > > >> > > > > > > > > > > > > > know at
> > > > >> > > > > > > > > > > > > least
> > > > >> > > > > > > > > > > > > > one public cloud who have snapshots
> which
> > > > take 2
> > > > >> > days
> > > > >> > > > to
> > > > >> > > > > > > > > complete.
> > > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used
> > for
> > > > >> > multiple
> > > > >> > > > LVM
> > > > >> > > > > > > > disks.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > I think the process Mike has used in the
> > > > >> SolidFire
> > > > >> > > > plugin
> > > > >> > > > > > > (only
> > > > >> > > > > > > > > > > > > > moving
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > disk image to secondary storage when you
> > > > >> absolutely
> > > > >> > > > have
> > > > >> > > > > > to)
> > > > >> > > > > > > is
> > > > >> > > > > > > > > > > > > > a very
> > > > >> > > > > > > > > > > > > good
> > > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> > > problems
> > > > >> an
> > > > >> > > > > operator
> > > > >> > > > > > > > might
> > > > >> > > > > > > > > > > > > experience
> > > > >> > > > > > > > > > > > > > if they have an issue with a given
> primary
> > > > >> storage
> > > > >> > > pool
> > > > >> > > > > in
> > > > >> > > > > > a
> > > > >> > > > > > > > > > cluster.
> > > > >> > > > > > > > > > > > (I
> > > > >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> > > > >> SolidFire
> > > > >> > > case
> > > > >> > > > > > Mike
> > > > >> > > > > > > :)
> > > > >> > > > > > > > > > > > > > ) And
> > > > >> > > > > > > > > > > > if
> > > > >> > > > > > > > > > > > > > the transfer from primary to secondary
> is
> > > > slow,
> > > > >> the
> > > > >> > > > time
> > > > >> > > > > to
> > > > >> > > > > > > > > > > > > > being able
> > > > >> > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > create a template or export the volume
> > will
> > > be
> > > > >> > slow.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > So for me the issue is around making
> sure
> > > that
> > > > >> the
> > > > >> > > end
> > > > >> > > > > > users
> > > > >> > > > > > > > > > > > expectations
> > > > >> > > > > > > > > > > > > > are met (while improving the
> > > speed/efficiency
> > > > of
> > > > >> > the
> > > > >> > > > back
> > > > >> > > > > > > end)
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > > http://www.shapeblue.com>
> > > > >> Paul
> > > > >> > > > Angus
> > > > >> > > > > > VP
> > > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> > 0540*
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > >
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > >> > > > > > > |
> > > > >> > > > > > > > m:
> > > > >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> > <javascript:;>
> > > > >> > > > > <javascript:;> | t:
> > > > >> > > > > > @cloudyangus*
> > > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com
> <javascript:;>
> > > > >> > > > <javascript:;>
> > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > > http://www.shapeblue.com
> > > > >
> > > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> London
> > > WC2N
> > > > >> 4HS
> > > > >> > UK
> > > > >> > > > > Shape
> > > > >> > > > > > > > Blue
> > > > >> > > > > > > > > > > > > > Ltd is a company incorporated in
> England &
> > > > >> Wales.
> > > > >> > > > > ShapeBlue
> > > > >> > > > > > > > > > > > > > Services India LLP is a company
> > incorporated
> > > > in
> > > > >> > India
> > > > >> > > > and
> > > > >> > > > > > is
> > > > >> > > > > > > > > > > > > > operated
> > > > >> > > > > > > > > > > > > under
> > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> > > Brasil
> > > > >> > > > > Consultoria
> > > > >> > > > > > > Ltda
> > > > >> > > > > > > > > > > > > > is a company incorporated in Brasil and
> is
> > > > >> operated
> > > > >> > > > under
> > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue
> SA
> > > Pty
> > > > >> Ltd
> > > > >> > is
> > > > >> > > a
> > > > >> > > > > > > company
> > > > >> > > > > > > > > > > > > > registered by The Republic
> > > > >> > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > South Africa and is traded under license
> > > from
> > > > >> Shape
> > > > >> > > > Blue
> > > > >> > > > > > Ltd.
> > > > >> > > > > > > > > > > > > > ShapeBlue
> > > > >> > > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > a registered trademark.
> > > > >> > > > > > > > > > > > > > This email and any attachments to it may
> > be
> > > > >> > > > confidential
> > > > >> > > > > > and
> > > > >> > > > > > > > are
> > > > >> > > > > > > > > > > > intended
> > > > >> > > > > > > > > > > > > > solely for the use of the individual to
> > whom
> > > > it
> > > > >> is
> > > > >> > > > > > addressed.
> > > > >> > > > > > > > > > > > > > Any views
> > > > >> > > > > > > > > > > > > or
> > > > >> > > > > > > > > > > > > > opinions expressed are solely those of
> the
> > > > >> author
> > > > >> > and
> > > > >> > > > do
> > > > >> > > > > > not
> > > > >> > > > > > > > > > > > necessarily
> > > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or
> > related
> > > > >> > > companies.
> > > > >> > > > > If
> > > > >> > > > > > > you
> > > > >> > > > > > > > > > > > > > are not
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > intended recipient of this email, you
> must
> > > > >> neither
> > > > >> > > take
> > > > >> > > > > any
> > > > >> > > > > > > > > > > > > > action
> > > > >> > > > > > > > > > > > based
> > > > >> > > > > > > > > > > > > > upon its contents, nor copy or show it
> to
> > > > >> anyone.
> > > > >> > > > Please
> > > > >> > > > > > > > contact
> > > > >> > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > sender
> > > > >> > > > > > > > > > > > > > if you believe you have received this
> > email
> > > in
> > > > >> > error.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > -----Original Message-----
> > > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > > >> pdion@cloudops.com
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > <javascript:;>]
> > > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature]
> > Storage
> > > > >> > Snapshots
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Hi Mike,
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > > >> StorageSnapshot
> > > > >> > > for
> > > > >> > > > > > > managed
> > > > >> > > > > > > > > > > > > > storage
> > > > >> > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > > > expected,
> > > > >> > > > behavior
> > > > >> > > > > > is
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> > > Storage.
> > > > >> So a
> > > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > > >> > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > snapshot that remain on the managed
> > storage
> > > > >> > > appliance.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Quickly looking at the API doc and I
> don't
> > > > see a
> > > > >> > > strong
> > > > >> > > > > > > > > > > > > > requirement for volume snapshots to be
> > moved
> > > > >> into
> > > > >> > > > > secondary
> > > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API
> is
> > > not
> > > > >> > useful,
> > > > >> > > > but
> > > > >> > > > > > > both
> > > > >> > > > > > > > > > > > > > use cases are required. A snapshot that
> > > remain
> > > > >> on
> > > > >> > the
> > > > >> > > > > > managed
> > > > >> > > > > > > > > > > > > > storage, and another type of
> > > > >> > > > > > > > > > > > snapshot
> > > > >> > > > > > > > > > > > > > that end up into the secondary storage.
> > > Since
> > > > >> > you've
> > > > >> > > > > done a
> > > > >> > > > > > > lot
> > > > >> > > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > work,
> > > > >> > > > > > > > > > > > > > might easier to just add a parameter to
> > the
> > > > >> current
> > > > >> > > > > > snapshot
> > > > >> > > > > > > > API
> > > > >> > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > > trigger an extraction of the storage
> > > snapshot
> > > > >> into
> > > > >> > > the
> > > > >> > > > > > > > secondary
> > > > >> > > > > > > > > > > > storage?
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > PL
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > > > Tutkowski <
> > > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > I think that all sounds reasonable
> then
> > -
> > > > >> thanks!
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > > > Mushtaq <
> > > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > >> <javascript:;>>
> > > > >> > > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> > > > >> > requirements.
> > > > >> > > > One
> > > > >> > > > > > of
> > > > >> > > > > > > > our
> > > > >> > > > > > > > > > > > earlier
> > > > >> > > > > > > > > > > > > > >> iterations on this was to have an
> > > argument
> > > > to
> > > > >> > the
> > > > >> > > > > create
> > > > >> > > > > > > > > > > > > > >> snapshot
> > > > >> > > > > > > > > > > > API
> > > > >> > > > > > > > > > > > > > >> which decides whether to backup the
> > > volume
> > > > to
> > > > >> > sec
> > > > >> > > > > > storage
> > > > >> > > > > > > > but
> > > > >> > > > > > > > > > > > > > >> we realized it would make management
> of
> > > > >> > snapshots
> > > > >> > > > > quite
> > > > >> > > > > > > > messy
> > > > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > > >> > > > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> > > Tutkowski
> > > > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > > > <javascript:;>
> > > > >> > > > > <javascript:;>>
> > > > >> > > > > > > > > > > > > > >> wrote:
> > > > >> > > > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > > > >>> Hi,
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> Just to make sure I understand all
> the
> > > > >> > > requirements
> > > > >> > > > > > here:
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 1) This relates only to managed
> > storage
> > > > (1:1
> > > > >> > > > mapping
> > > > >> > > > > > > > between
> > > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> > > volume).
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > > (introduced
> > > > >> in
> > > > >> > > 4.6)
> > > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> > snapshot
> > > on
> > > > >> the
> > > > >> > > SAN,
> > > > >> > > > > and
> > > > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > > > >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> > > > >> snapshot,
> > > > >> > but
> > > > >> > > > to
> > > > >> > > > > > copy
> > > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only)
> to
> > > > NFS.
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken.
> > > It's
> > > > >> the
> > > > >> > > > backup
> > > > >> > > > > to
> > > > >> > > > > > > NFS
> > > > >> > > > > > > > > > > > > > >>> that is optional.
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> > > > >> snapshot
> > > > >> > > > that's
> > > > >> > > > > on
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created
> > > from
> > > > >> the
> > > > >> > > > > snapshot
> > > > >> > > > > > > > > > > > > > >>> that's on
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > >>> SAN (already works as long as the
> new
> > > > >> > CloudStack
> > > > >> > > > > volume
> > > > >> > > > > > > > ends
> > > > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > > > snapshot
> > > > >> API
> > > > >> > > > then
> > > > >> > > > > or
> > > > >> > > > > > > > > > > > > > >>> would that just be the standard
> volume
> > > > >> snapshot
> > > > >> > > > > without
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > > > >>> backup to
> > > > >> > > > > > > > > > > NFS?
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> Thanks!
> > > > >> > > > > > > > > > > > > > >>> Mike
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> > > > Mushtaq
> > > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> > <javascript:;>
> > > > >> > > > > <javascript:;>>
> > > > >> > > > > > > > > > > > > > >>> wrote:
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > > functionalities
> > > > >> > > > > (snapshot
> > > > >> > > > > > on
> > > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally,
> we
> > > > would
> > > > >> > like
> > > > >> > > > to
> > > > >> > > > > > have
> > > > >> > > > > > > > > both.
> > > > >> > > > > > > > > > > > > > >>>> For example, some of our customers
> > want
> > > > to
> > > > >> > > > implement
> > > > >> > > > > > > their
> > > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> > encryption
> > > > to
> > > > >> > their
> > > > >> > > > > > backups
> > > > >> > > > > > > > > > > > > > >>>> which is a perfect use case for
> > Storage
> > > > >> > Snapshot
> > > > >> > > > > while
> > > > >> > > > > > > our
> > > > >> > > > > > > > > > > > > > >>>> other customers will still keep
> using
> > > the
> > > > >> > > standard
> > > > >> > > > > > > volume
> > > > >> > > > > > > > > > > snapshot.
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>> To keep things backward compatible,
> > we
> > > > can
> > > > >> > add a
> > > > >> > > > > > setting
> > > > >> > > > > > > > > > > > > > >>>> which
> > > > >> > > > > > > > > > > > says
> > > > >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> > > > >> because,
> > > > >> > > after
> > > > >> > > > > > all,
> > > > >> > > > > > > > you
> > > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first
> when
> > > > doing
> > > > >> a
> > > > >> > > > Volume
> > > > >> > > > > > > > > Snapshot.
> > > > >> > > > > > > > > > > > > > >>>> You could stop the process there
> and
> > > not
> > > > do
> > > > >> > the
> > > > >> > > > > > upload.
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>> What do you think about this
> > approach?
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > > >> > > > > > > > > > > > > > >>>> -Syed
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM,
> Mike
> > > > >> > Tutkowski <
> > > > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out
> > load
> > > > >> here,
> > > > >> > but
> > > > >> > > > if
> > > > >> > > > > a
> > > > >> > > > > > > > given
> > > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually
> > need
> > > > to
> > > > >> > > provide
> > > > >> > > > > > both
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > ability
> > > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export
> it
> > > to
> > > > >> NFS
> > > > >> > (if
> > > > >> > > > > just
> > > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then
> > we
> > > > >> might
> > > > >> > be
> > > > >> > > > able
> > > > >> > > > > > to
> > > > >> > > > > > > > get
> > > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > > >> > > > > > > > > > > > API
> > > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> > > custom
> > > > >> > > snapshot
> > > > >> > > > > > > strategy
> > > > >> > > > > > > > > > > > > > >>>>> and
> > > > >> > > > > > > > > > > > data
> > > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case
> > > where
> > > > >> the
> > > > >> > > > > > CloudStack
> > > > >> > > > > > > > > > > > > > >>>>> cloud
> > > > >> > > > > > > > > > > > does
> > > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > > >> exported-to-NFS
> > > > >> > > > > backup.
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> In other words, the "default"
> > behavior
> > > > >> would
> > > > >> > be
> > > > >> > > > to
> > > > >> > > > > > use
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> > > > strategy
> > > > >> > that
> > > > >> > > > we
> > > > >> > > > > > > > already
> > > > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a
> SAN
> > > > >> > snapshot).
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however,
> > > wants
> > > > >> to
> > > > >> > > take
> > > > >> > > > a
> > > > >> > > > > > SAN
> > > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data
> exported
> > to
> > > > >> NFS,
> > > > >> > > then
> > > > >> > > > we
> > > > >> > > > > > > could
> > > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config
> > > file
> > > > to
> > > > >> > make
> > > > >> > > > use
> > > > >> > > > > > of
> > > > >> > > > > > > a
> > > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data
> > motion
> > > > >> > strategy
> > > > >> > > > that
> > > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > > >> > > > > > > > > > > > > > activities.
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is
> still
> > > the
> > > > >> > default
> > > > >> > > > for
> > > > >> > > > > > > > users,
> > > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change
> > this
> > > > >> > behavior
> > > > >> > > > via
> > > > >> > > > > > > > > > > configuration.
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM,
> > Mike
> > > > >> > > Tutkowski <
> > > > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to
> > come
> > > up
> > > > >> > with a
> > > > >> > > > > > viable
> > > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable
> way
> > > for
> > > > >> them
> > > > >> > > to
> > > > >> > > > > move
> > > > >> > > > > > > > from
> > > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and
> > some
> > > > >> obvious
> > > > >> > > way
> > > > >> > > > > > that
> > > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > > >> > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > do this).
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM,
> > Syed
> > > > >> > Mushtaq
> > > > >> > > <
> > > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > > <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> > > upgrade
> > > > >> path
> > > > >> > > > > however,
> > > > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and
> > are
> > > > on a
> > > > >> > > > managed
> > > > >> > > > > > > > storage
> > > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > > > >> functionality
> > > > >> > > > with
> > > > >> > > > > > > Volume
> > > > >> > > > > > > > > > > Snapshots.
> > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM,
> > Syed
> > > > >> > Mushtaq
> > > > >> > > <
> > > > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > > > <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly,
> > > > currently
> > > > >> > > > taking a
> > > > >> > > > > > > > Volume
> > > > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a
> > managed
> > > > >> storage
> > > > >> > > > keeps
> > > > >> > > > > > it
> > > > >> > > > > > > on
> > > > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part of
> > > this
> > > > >> > > feature,
> > > > >> > > > we
> > > > >> > > > > > can
> > > > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > > > >> > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed
> > storage
> > > > are
> > > > >> > > > uploaded
> > > > >> > > > > > to
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would
> > make
> > > > the
> > > > >> > > Volume
> > > > >> > > > > > > Snapshot
> > > > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same
> > regardless
> > > of
> > > > >> the
> > > > >> > > > > storage
> > > > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing
> > the
> > > > >> > > efficient
> > > > >> > > > > > > backend
> > > > >> > > > > > > > > > > > > > >>>>>>>> storage
> > > > >> > > > > > > > > > > > > > capabilities, we can use the new storage
> > > > >> snapshots
> > > > >> > > API.
> > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM,
> > > Mike
> > > > >> > > > Tutkowski <
> > > > >> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> > > > >> > <javascript:;>
> > > > >> > > > > <javascript:;>>
> > > > >> > > > > > wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need
> to
> > > > have a
> > > > >> > plan
> > > > >> > > > to
> > > > >> > > > > > deal
> > > > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we already
> > > have a
> > > > >> > > feature
> > > > >> > > > > (in
> > > > >> > > > > > > 4.6
> > > > >> > > > > > > > > > > > > > >>>>>>>>> and
> > > > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to use
> > the
> > > > >> > existing
> > > > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to
> create a
> > > > >> volume
> > > > >> > > > > snapshot
> > > > >> > > > > > > (for
> > > > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > > > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a
> > backend
> > > > SAN
> > > > >> > > > (using a
> > > > >> > > > > > > > custom
> > > > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom
> > > data
> > > > >> > motion
> > > > >> > > > > > > strategy).
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then
> > how
> > > > >> should
> > > > >> > > the
> > > > >> > > > > > > original
> > > > >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6
> > and
> > > > >> later)
> > > > >> > > be
> > > > >> > > > > > > changed?
> > > > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > > > >> > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support
> > > customers
> > > > >> who
> > > > >> > > are
> > > > >> > > > > > > already
> > > > >> > > > > > > > > > > > > > >>>>>>>>> using
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API
> to
> > > take
> > > > >> > > > snapshots
> > > > >> > > > > > on a
> > > > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > > > >> > > > > > > > > > > > > SAN?
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27
> AM,
> > > > Will
> > > > >> > > > Stevens <
> > > > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > > > <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create a
> > > > Template
> > > > >> > > from a
> > > > >> > > > > > > > > > > > StorageSnapshot?
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template be
> > > stored
> > > > >> in
> > > > >> > the
> > > > >> > > > > > > secondary
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal templates
> > or
> > > > will
> > > > >> > that
> > > > >> > > > be
> > > > >> > > > > > > > handled
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > >> > > > > > > > > > > > > > vendor side?
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud
> Solutions
> > > > >> Experts
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|*
> > > Quebec
> > > > >> *|*
> > > > >> > > H3J
> > > > >> > > > > 1S6
> > > > >> > > > > > w
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw
> @CloudOps_
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22
> PM,
> > > > Syed
> > > > >> > > > Mushtaq <
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19
> > PM,
> > > > Will
> > > > >> > > > Stevens
> > > > >> > > > > <
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the
> > Design
> > > > >> Spec in
> > > > >> > > the
> > > > >> > > > > > Jira
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not
> > clear
> > > > in
> > > > >> the
> > > > >> > > > > > 'mention'
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows
> as a
> > > > page
> > > > >> > 'you
> > > > >> > > do
> > > > >> > > > > not
> > > > >> > > > > > > > have
> > > > >> > > > > > > > > > > permission to'.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud
> > Solutions
> > > > >> Experts
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal
> *|*
> > > > Quebec
> > > > >> > *|*
> > > > >> > > > H3J
> > > > >> > > > > > 1S6
> > > > >> > > > > > > w
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw
> > @CloudOps_
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02
> > PM,
> > > > >> Syed
> > > > >> > > Ahmed
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > >
> > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new
> set
> > > of
> > > > >> APIs
> > > > >> > to
> > > > >> > > > do
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed
> storage
> > > > >> backends
> > > > >> > > > like
> > > > >> > > > > > > > > SolidFire.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current
> managed
> > > > >> storage
> > > > >> > > stay
> > > > >> > > > > on
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary
> to
> > > what
> > > > >> > > > CloudStack
> > > > >> > > > > > > calls
> > > > >> > > > > > > > > > > snpshots.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on
> > > storage
> > > > >> and
> > > > >> > > > keeping
> > > > >> > > > > > it
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own
> advantages
> > > > >> > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to
> > have
> > > > both
> > > > >> > ways
> > > > >> > > > of
> > > > >> > > > > > > doing
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal
> > adds
> > > 4
> > > > >> new
> > > > >> > > APIs
> > > > >> > > > to
> > > > >> > > > > > > > create
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend
> > storage.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of
> > this
> > > > >> > > feature? I
> > > > >> > > > > > would
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some
> feedback.
> > I
> > > am
> > > > >> > > working
> > > > >> > > > on
> > > > >> > > > > > > > making
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > >> > > > > > > > > > > > spec
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted
> to
> > > > have a
> > > > >> > high
> > > > >> > > > > level
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before
> > starting
> > > > to
> > > > >> > work
> > > > >> > > on
> > > > >> > > > > it.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> --
> > > > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer,
> > > > >> SolidFire
> > > > >> > > Inc.*
> > > > >> > > > > > > > > > > > > > >>>>>>>>> e:
> mike.tutkowski@solidfire.com
> > > > >> > > > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world
> uses
> > > the
> > > > >> > cloud
> > > > >> > > > > > > > > > > > > > >>>>>>>>> <
> > > > >> > > > > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > > > > >*™
> > > > >> > > > > > > > > > > > > > >>>>>>>>> *
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>> --
> > > > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer,
> > > SolidFire
> > > > >> > Inc.*
> > > > >> > > > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > > >> > > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > >> > > > > > > > > > > > > > >>>>>> Advancing the way the world uses
> > the
> > > > >> cloud
> > > > >> > > > > > > > > > > > > > >>>>>> <
> > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > > > >*™*
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> --
> > > > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer,
> > > SolidFire
> > > > >> Inc.*
> > > > >> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > >> > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > >> > > > > > > > > > > > > > >>>>> Advancing the way the world uses
> the
> > > > cloud
> > > > >> > > > > > > > > > > > > > >>>>> <
> > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > > > >*™*
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> --
> > > > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer,
> > SolidFire
> > > > >> Inc.*
> > > > >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > > >> > > > > > > > > > > > > > >>> Advancing the way the world uses the
> > > cloud
> > > > >> > > > > > > > > > > > > > >>> <
> > > > >> > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > > >*™*
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > --
> > > > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > > *Senior CloudStack Developer,
> SolidFire
> > > > Inc.*
> > > > >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > > o: 303.746.7302
> > > > >> > > > > > > > > > > > > > > Advancing the way the world uses the
> > cloud
> > > > >> > > > > > > > > > > > > > > <
> > > > >> > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > >*™*
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Find out more about ShapeBlue and our
> > range
> > > of
> > > > >> > > > CloudStack
> > > > >> > > > > > > > > > > > > > related
> > > > >> > > > > > > > > > > > > services:
> > > > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > >> > > > > > > > > > > > > > <
> > > > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > > >> > > |
> > > > >> > > > > > > > CSForge –
> > > > >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > > > >> > > > > > > http://shapeblue.com/csforge/
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > > > > > CloudStack Consulting
> > > > >> > > > > > > > > > > > > > <
> > > http://shapeblue.com/cloudstack-consultancy/
> > > > >
> > > > >> |
> > > > >> > > > > > > > > > > > > CloudStack
> > > > >> > > > > > > > > > > > > > Software Engineering
> > > > >> > > > > > > > > > > > > > <
> > > > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > > > >> > > > >
> > > > >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > > > >> > > > > > > > > > > > > > <
> > > > >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> > > > >> > > > > >
> > > > >> > > > > > |
> > > > >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > >> > > > > > > > > > > > > > <
> > http://shapeblue.com/cloudstack-training/>
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > --
> > > > >> > > > > > > > > > > > *Mike Tutkowski*
> > > > >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> Inc.*
> > > > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > o: 303.746.7302
> > > > >> > > > > > > > > > > > Advancing the way the world uses the cloud
> > > > >> > > > > > > > > > > > <
> > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > >*™*
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > Find out more about ShapeBlue and our range of
> > > > >> CloudStack
> > > > >> > > > > related
> > > > >> > > > > > > > > > services:
> > > > >> > > > > > > > > > > IaaS Cloud Design & Build
> > > > >> > > > > > > > > > > <
> > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > >
> > > > >> |
> > > > >> > > > > CSForge –
> > > > >> > > > > > > > rapid
> > > > >> > > > > > > > > > > IaaS deployment framework <
> > > > >> http://shapeblue.com/csforge/
> > > > >> > >
> > > > >> > > > > > > CloudStack
> > > > >> > > > > > > > > > > Consulting <
> > > > >> http://shapeblue.com/cloudstack-consultancy/
> > > > >> > >
> > > > >> > > |
> > > > >> > > > > > > > CloudStack
> > > > >> > > > > > > > > > > Software Engineering
> > > > >> > > > > > > > > > > <
> > > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > > >> > > > > > > > > > > CloudStack Infrastructure Support
> > > > >> > > > > > > > > > > <
> > > > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > > > >> > >
> > > > >> > > |
> > > > >> > > > > > > > CloudStack
> > > > >> > > > > > > > > > > Bootcamp Training Courses <
> > > > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > Find out more about ShapeBlue and our range of
> > > > >> CloudStack
> > > > >> > > > related
> > > > >> > > > > > > > > services:
> > > > >> > > > > > > > > > IaaS Cloud Design & Build
> > > > >> > > > > > > > > > <
> > http://shapeblue.com/iaas-cloud-design-and-build//
> > > >
> > > > |
> > > > >> > > > CSForge –
> > > > >> > > > > > > rapid
> > > > >> > > > > > > > > > IaaS deployment framework <
> > > > >> http://shapeblue.com/csforge/>
> > > > >> > > > > > > > > > CloudStack Consulting <
> > > > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > >> > > > > > > |
> > > > >> > > > > > > > > CloudStack
> > > > >> > > > > > > > > > Software Engineering
> > > > >> > > > > > > > > > <
> > > > http://shapeblue.com/cloudstack-software-engineering/>
> > > > >> > > > > > > > > > CloudStack Infrastructure Support
> > > > >> > > > > > > > > > <
> > > > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > > >> > |
> > > > >> > > > > > > CloudStack
> > > > >> > > > > > > > > > Bootcamp Training Courses <
> > > > >> > > > > > http://shapeblue.com/cloudstack-training/
> > > > >> > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > --
> > > > >> > > > > > > *Mike Tutkowski*
> > > > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>
> > > > >> > > > > > > o: 303.746.7302
> > > > >> > > > > > > Advancing the way the world uses the cloud
> > > > >> > > > > > > <http://solidfire.com/solution/overview/?video=play
> >*™*
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > --
> > > > >> > > > > *Mike Tutkowski*
> > > > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > >> > > > > o: 303.746.7302
> > > > >> > > > > Advancing the way the world uses the cloud
> > > > >> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > > Ian Rae
> > > > >> > > > CEO | PDG
> > > > >> > > > c: 514.944.4008
> > > > >> > > >
> > > > >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> > > > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J
> 1S6
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > *Mike Tutkowski*
> > > > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > > > >> > e: mike.tutkowski@solidfire.com
> > > > >> > o: 303.746.7302
> > > > >> > Advancing the way the world uses the cloud
> > > > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>



--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Right, we'd want to keep track of what kind of a volume snapshot was taken
and not assume all in the past were using the same Global Settings value.

On Mon, Feb 8, 2016 at 12:33 PM, Will Stevens <ws...@cloudops.com> wrote:

> A global setting would probably be fine.  How would the case be handled if
> the global setting is changed?  Would it only affect the snapshots created
> after the change was made?  We would also code defensively so if the global
> setting changes that we don't assume all the snapshots in the past had the
> same global setting when they were created.
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Mon, Feb 8, 2016 at 2:16 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com
> > wrote:
>
> > Correct, Will.
> >
> > That Global Settings would only be for managed storage. Non-managed
> > (traditional) volume snapshots are completely un-impacted by this
> feature.
> >
> > If we need to sometimes keep the snapshots on the SAN and sometimes push
> > them to secondary storage, we'll need a more robust solution than Global
> > Settings, though.
> >
> > On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > Sorry.  I missed a bit of context when I responded.  The global setting
> > > would be only for the managed storage case, currently being called
> > Storage
> > > Snapshots, and is only to determine if a copy is pushed to secondary
> > > storage right?  The global setting would not change the behavior of the
> > > Volume Snapshots right?
> > >
> > > I was referring to the need for Volume Snapshots and Storage Snapshots
> to
> > > exist together.  Disregard my comment.  I caught up on context after I
> > > posted.  My bad...
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > *CloudOps* *| *Cloud Solutions Experts
> > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > w cloudops.com *|* tw @CloudOps_
> > >
> > > On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com
> > > > wrote:
> > >
> > > > Hey Will,
> > > >
> > > > Who's picking the behavior? Is it the cloud provider or the end user?
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <wstevens@cloudops.com
> >
> > > > wrote:
> > > >
> > > > > I don't think a global setting is a good option because we need
> both
> > > > > functionality to be available at the same time and for different
> use
> > > > cases
> > > > > to be able to pick which they choose.
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > *CloudOps* *| *Cloud Solutions Experts
> > > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > > w cloudops.com *|* tw @CloudOps_
> > > > >
> > > > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > Now that I re-read your e-mail, it dawned on me: The end-user
> > doesn't
> > > > > care
> > > > > > where the snapshot is.
> > > > > >
> > > > > > If that's true, then we should perhaps control this via Global
> > > Settings
> > > > > or
> > > > > > something.
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > > > mike.tutkowski@solidfire.com> wrote:
> > > > > >
> > > > > > > It's not ideal - true, but it does allow us to be backward
> > > > compatible.
> > > > > > >
> > > > > > > If you have other ideas, though, about how to maintain backward
> > > > > > > compatibility, I'm definitely open to hear them.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > > > syed1.mushtaq@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Mike,
> > > > > > >>
> > > > > > >> Adding a flag to createSnapshot was the first and the most
> > obvious
> > > > > thing
> > > > > > >> that came to our minds. The problem that I had with this was
> > that:
> > > > > > >>
> > > > > > >> 1) I feel it is exposing something to the end user that is
> > > internal
> > > > to
> > > > > > the
> > > > > > >> cloud.
> > > > > > >>
> > > > > > >> 2) We have to follow two different ways of restore/deletion in
> > the
> > > > > same
> > > > > > >> code path depending on where the Snapshot resides which I find
> > > kind
> > > > > of a
> > > > > > >> bad design.
> > > > > > >>
> > > > > > >> But if exposing a archive flag to the end user is acceptable
> > then
> > > we
> > > > > can
> > > > > > >> definitely use this instead of adding the StorageSnapshot API
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> -Syed
> > > > > > >>
> > > > > > >>
> > > > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > > > >> mike.tutkowski@solidfire.com
> > > > > > >> > wrote:
> > > > > > >>
> > > > > > >> > Hi Pierre-Luc,
> > > > > > >> >
> > > > > > >> > My recommendation would be this:
> > > > > > >> >
> > > > > > >> > Add an "archive" flag to the current volume-snapshot API.
> Its
> > > > > default
> > > > > > >> would
> > > > > > >> > be "false" because that would be backward compatible with
> how
> > > 4.6
> > > > > has
> > > > > > >> > volume snapshots implemented (i.e. they stay on the SAN in
> > 4.6,
> > > > 4.7,
> > > > > > and
> > > > > > >> > 4.8).
> > > > > > >> >
> > > > > > >> > If you set archive=true, then we would perform a background
> > > > > migration
> > > > > > of
> > > > > > >> > the snapshot from the SAN to the secondary storage (then
> > delete
> > > > the
> > > > > > SAN
> > > > > > >> > snapshot).
> > > > > > >> >
> > > > > > >> > That archive parameter would only be valid for managed
> > storage.
> > > > > > >> >
> > > > > > >> > Sound reasonable?
> > > > > > >> >
> > > > > > >> > Also, a VM snapshot that includes disks provided by managed
> > > > storage
> > > > > > >> should
> > > > > > >> > work.
> > > > > > >> >
> > > > > > >> > Talk to you later,
> > > > > > >> > Mike
> > > > > > >> >
> > > > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > > > pdion@cloudops.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Mike,
> > > > > > >> > >
> > > > > > >> > > In terms of API's, would you prefer introducing a
> parameter
> > to
> > > > the
> > > > > > >> > existing
> > > > > > >> > > VolumeSnapshot, example:   extract={true|false}  with a
> > > default
> > > > > > value
> > > > > > >> of
> > > > > > >> > > true  which would extract snapshot into the secondary
> > storage,
> > > > > which
> > > > > > >> is
> > > > > > >> > the
> > > > > > >> > > current default behavior. Then for SAN snapshot that
> remain
> > on
> > > > the
> > > > > > >> SAN we
> > > > > > >> > > would just set "extract=false" ?  as oppose to create a
> new
> > > > > > >> > >  StorageSnapshot API ?
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Paul,
> > > > > > >> > >
> > > > > > >> > > From what I'm seeing so far, we can't do a VM-snapshot
> when
> > > > using
> > > > > > >> managed
> > > > > > >> > > storage for VM having more than one Volume. For the reason
> > > that
> > > > > > >> snapshot
> > > > > > >> > > are performed outside of the hypervisor awareness and
> > > > > > asynchronously.
> > > > > > >> If
> > > > > > >> > > someone have a way to address that, it would make thinks
> > much
> > > > more
> > > > > > >> > > attractive.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <
> irae@cloudops.com
> > >
> > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > I think a service provider backup scenario is more
> likely
> > to
> > > > > take
> > > > > > >> > > advantage
> > > > > > >> > > > of SAN snapshot. There are a few reasons for this.
> > > Traditional
> > > > > > >> backups
> > > > > > >> > > > involve access to the file system, and there is an
> > > expectation
> > > > > > that
> > > > > > >> > this
> > > > > > >> > > > can be done with reasonably short time frames without
> > > > negatively
> > > > > > >> > > impacting
> > > > > > >> > > > VM performance, and that the backup orchestrator can
> apply
> > > > > various
> > > > > > >> > logic
> > > > > > >> > > > and or transformations to the data (compress, encrypt,
> > > deltas
> > > > > > >> etc...).
> > > > > > >> > > > While it is true that one could apply a backup process
> to
> > a
> > > > > cloud
> > > > > > >> > > snapshot,
> > > > > > >> > > > this would be slow and inefficient requiring the data to
> > be
> > > > > moved
> > > > > > >> > several
> > > > > > >> > > > times and there are some major bottlenecks with cloud
> > > > snapshots.
> > > > > > >> With a
> > > > > > >> > > > cloud snapshot - there seems to be no reasonable
> > expectation
> > > > of
> > > > > > >> being
> > > > > > >> > > able
> > > > > > >> > > > to do differential snapshots (I think this depends on
> the
> > > > > > >> hypervisor)
> > > > > > >> > and
> > > > > > >> > > > if you do differential snapshots this will make file
> > backups
> > > > > from
> > > > > > >> those
> > > > > > >> > > > snapshots even more complicated to orchestrate.
> > > > > > >> > > >
> > > > > > >> > > > Suspect there needs to be a different thread on how to
> > > better
> > > > > > enable
> > > > > > >> > > > backups, as a service. As per Paul's suggestion, but it
> > is a
> > > > > > related
> > > > > > >> > > > workflow so relevant to this discussion.
> > > > > > >> > > >
> > > > > > >> > > > Ian
> > > > > > >> > > >
> > > > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > > > >> > > mike.tutkowski@solidfire.com>
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > To me it sounds like number two and number three are
> > > > different
> > > > > > >> uses
> > > > > > >> > for
> > > > > > >> > > > the
> > > > > > >> > > > > same "thing"(which is totally fine).
> > > > > > >> > > > >
> > > > > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > > > > >> asynchronously, do
> > > > > > >> > > we
> > > > > > >> > > > > see the SSVM as performing the export?
> > > > > > >> > > > >
> > > > > > >> > > > > To be backwards compatible with what we have in 4.6
> and
> > > > later
> > > > > > for
> > > > > > >> > > volume
> > > > > > >> > > > > snapshots for managed storage, I think it might be
> > easier
> > > if
> > > > > we
> > > > > > >> pass
> > > > > > >> > > in a
> > > > > > >> > > > > flag that says whether or not to archive the SAN
> > snapshot
> > > > > > (which,
> > > > > > >> I
> > > > > > >> > > > think,
> > > > > > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > > > > > >> > > > >
> > > > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > > > pdion@cloudops.com
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hi Mike,
> > > > > > >> > > > > >
> > > > > > >> > > > > > The reason behind the creation of a SAN snapshot
> which
> > > is
> > > > > > >> exported
> > > > > > >> > > into
> > > > > > >> > > > > > secondary storage, is because creating a copy of the
> > > .VHD
> > > > > > >> directly
> > > > > > >> > > > would
> > > > > > >> > > > > > impact uptime of the VM as creating that copy take
> > lots
> > > of
> > > > > > time.
> > > > > > >> > Has
> > > > > > >> > > > > oppose
> > > > > > >> > > > > > to a SAN snapshot that is close to instantaneous
> which
> > > can
> > > > > > >> > afterward
> > > > > > >> > > be
> > > > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > > > >> > > > > >
> > > > > > >> > > > > > I would suspect an extracted VolumeSnapshot taken
> > from a
> > > > SAN
> > > > > > >> > snapshot
> > > > > > >> > > > > could
> > > > > > >> > > > > > have is SAN snapshot deleted to avoid duplica and
> > space
> > > > > > >> consumption
> > > > > > >> > > on
> > > > > > >> > > > > the
> > > > > > >> > > > > > Primary Storage side.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > I see 3 definitions in our current discussion
> > regarding
> > > > the
> > > > > > term
> > > > > > >> > > > snapshot
> > > > > > >> > > > > > (these are not official terminology but by own
> > > > > interpretation
> > > > > > of
> > > > > > >> > > them):
> > > > > > >> > > > > >
> > > > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's
> > definition
> > > > of
> > > > > a
> > > > > > >> > > > snapshot):
> > > > > > >> > > > > > it's a volume snapshot at the storage level, point
> in
> > > time
> > > > > of
> > > > > > >> your
> > > > > > >> > > > data.
> > > > > > >> > > > > it
> > > > > > >> > > > > > reside on the primary storage. Useful and efficient
> > for
> > > > > > software
> > > > > > >> > side
> > > > > > >> > > > > > incident.
> > > > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack
> VolumeSnapshot/
> > > > cloud
> > > > > > >> backup
> > > > > > >> > > > aws-S3
> > > > > > >> > > > > > style ): Point in time copy of the Virtual Disk that
> > > > reside
> > > > > > on a
> > > > > > >> > > > > different
> > > > > > >> > > > > > storage array then the original Volume. Facilitate
> > data
> > > > > > >> migration
> > > > > > >> > > > between
> > > > > > >> > > > > > clusters and, in case of primary storage incident,
> > > Volume
> > > > > > >> snapshots
> > > > > > >> > > are
> > > > > > >> > > > > not
> > > > > > >> > > > > > impacted and can be reuse.
> > > > > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data
> > that
> > > > > also
> > > > > > >> > > validate
> > > > > > >> > > > > data
> > > > > > >> > > > > > integrity, provide a storage efficient archiving
> > method
> > > > and
> > > > > an
> > > > > > >> > > > > independent
> > > > > > >> > > > > > way to restore your data in case of an major
> > > > infrastructure
> > > > > > >> > disaster.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Regards,
> > > > > > >> > > > > >
> > > > > > >> > > > > > PL
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > So, let's see if I currently follow the
> > requirements:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > * Augment volume snapshots for managed storage to
> > > > > > >> conditionally
> > > > > > >> > > > export
> > > > > > >> > > > > > data
> > > > > > >> > > > > > > to NFS. The current process of taking a snapshot
> on
> > > the
> > > > > SAN
> > > > > > is
> > > > > > >> > > fine,
> > > > > > >> > > > > but
> > > > > > >> > > > > > > we'd like the option to export the data to NFS, as
> > > well.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Questions:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Once the data has been exported to NFS, do we keep
> > the
> > > > SAN
> > > > > > >> > snapshot
> > > > > > >> > > > or
> > > > > > >> > > > > > > delete it?
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > If we are deleting the SAN snapshot, then why
> don't
> > we
> > > > > just
> > > > > > >> copy
> > > > > > >> > > the
> > > > > > >> > > > > VHD
> > > > > > >> > > > > > > from primary to secondary the way we do today for
> > > > > > non-managed
> > > > > > >> > (i.e.
> > > > > > >> > > > > > > traditional) storage? Why create a SAN snapshot in
> > > that
> > > > > > >> scenario?
> > > > > > >> > > > > Perhaps
> > > > > > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> > > > > secondary
> > > > > > >> > > storage
> > > > > > >> > > > > > > instead of a XenServer host?
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks for the clarification.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > By the way, to me a backup is when you copy data
> > from
> > > > one
> > > > > > >> storage
> > > > > > >> > > > > system
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > another (regardless of features, if any, to
> restore
> > > the
> > > > > data
> > > > > > >> in
> > > > > > >> > the
> > > > > > >> > > > > > > future). A snapshot is a point-in-time view of the
> > > data
> > > > > of a
> > > > > > >> > volume
> > > > > > >> > > > and
> > > > > > >> > > > > > > it's stored on the same storage system as the
> > volume.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > > > > >> > > pdion@cloudops.com
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > <javascript:;>>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > That's fun to see that discussion happening. I
> > 100%
> > > > > agree
> > > > > > >> with
> > > > > > >> > > > Paul's
> > > > > > >> > > > > > > > points of view. VolumeSnapshot are not a backup,
> > > but I
> > > > > do
> > > > > > >> > > consider
> > > > > > >> > > > > them
> > > > > > >> > > > > > > as
> > > > > > >> > > > > > > > a safety vest against Primary Storage failure,
> > > because
> > > > > > >> failure
> > > > > > >> > > > append
> > > > > > >> > > > > > > :-( .
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > The current proposal around snapshots that
> reside
> > on
> > > > the
> > > > > > >> > primary
> > > > > > >> > > > > > storage
> > > > > > >> > > > > > > or
> > > > > > >> > > > > > > > snapshots that end in the Secondary Storage  is
> > not
> > > to
> > > > > > >> address
> > > > > > >> > > any
> > > > > > >> > > > > kind
> > > > > > >> > > > > > > of
> > > > > > >> > > > > > > > backups requirement because a snapshot is not a
> > > > backup,
> > > > > > >> event
> > > > > > >> > an
> > > > > > >> > > > > > > extracted
> > > > > > >> > > > > > > > VM snapshot.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > The main idea, and again this is for managed
> > > storage;
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side
> > snapshot
> > > > > > >> capability
> > > > > > >> > > for
> > > > > > >> > > > > > fast
> > > > > > >> > > > > > > > response time that support rollback to previous
> > > > > timestamp,
> > > > > > >> > create
> > > > > > >> > > > new
> > > > > > >> > > > > > > > volume and maybe create template.
> > > > > > >> > > > > > > >     not required to be a new API if the work is
> > > > already
> > > > > > >> done, I
> > > > > > >> > > > think
> > > > > > >> > > > > > > this
> > > > > > >> > > > > > > > is a different behaviors than the user
> expectation
> > > of
> > > > a
> > > > > > >> > > > > > volume-snapshot.
> > > > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> > > > > behavior
> > > > > > >> that
> > > > > > >> > > > create
> > > > > > >> > > > > > an
> > > > > > >> > > > > > > > extraction of a volume into SecondaryStorage
> which
> > > can
> > > > > be
> > > > > > >> reuse
> > > > > > >> > > to
> > > > > > >> > > > > > > create a
> > > > > > >> > > > > > > > new volume into another Primary Storage. This
> type
> > > of
> > > > > > >> snapshot
> > > > > > >> > > is a
> > > > > > >> > > > > > slow
> > > > > > >> > > > > > > > job since yes it would have to copy the full
> > volume
> > > > size
> > > > > > on
> > > > > > >> the
> > > > > > >> > > > > > Secondary
> > > > > > >> > > > > > > > storage.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > PL
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > >> > > > > > <javascript:;>>
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > I think I share you view on the 'Ideal world'.
> > > > Backup
> > > > > > (via
> > > > > > >> > > Volume
> > > > > > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack.
> > > This
> > > > is
> > > > > > >> > > amplified
> > > > > > >> > > > > > > > especially
> > > > > > >> > > > > > > > > when you have a object storage as your
> secondary
> > > > > storage
> > > > > > >> > > because
> > > > > > >> > > > it
> > > > > > >> > > > > > > > > requires two copies (one to an NFS staging
> area
> > > and
> > > > > from
> > > > > > >> > there
> > > > > > >> > > to
> > > > > > >> > > > > > > object
> > > > > > >> > > > > > > > > storage). And not to mention that all these
> > copies
> > > > are
> > > > > > >> > > consuming
> > > > > > >> > > > > > > > hypervisor
> > > > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> > > > bottleneck
> > > > > as
> > > > > > >> all
> > > > > > >> > > the
> > > > > > >> > > > > > > Network
> > > > > > >> > > > > > > > > and I/O flow through it. So our intention of
> > > > proposing
> > > > > > the
> > > > > > >> > > > "Storage
> > > > > > >> > > > > > > > > Snapshots" is to give a better way of achiving
> > > > > snapshots
> > > > > > >> > while
> > > > > > >> > > > > still
> > > > > > >> > > > > > > > > keeping the original definition of volume
> > > snpashots
> > > > > (ie
> > > > > > >> > upload
> > > > > > >> > > to
> > > > > > >> > > > > sec
> > > > > > >> > > > > > > > > storage).
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > But as Erik pointed out volume snapshots are
> not
> > > > > > backups.
> > > > > > >> > They
> > > > > > >> > > > > don't
> > > > > > >> > > > > > > work
> > > > > > >> > > > > > > > > form multi-disk LVM volume groups and dynamic
> > > > disks. I
> > > > > > am
> > > > > > >> all
> > > > > > >> > > in
> > > > > > >> > > > > for
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > > > better backup solution which handles these use
> > > cases
> > > > > and
> > > > > > >> > > utilizes
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > > storage's advanced features.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > > > > >> > > > > > paul.angus@shapeblue.com <javascript:;>
> > <javascript:;>>
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > In the beginning... there were CloudStack
> > > > snapshots
> > > > > > and
> > > > > > >> > they
> > > > > > >> > > > were
> > > > > > >> > > > > > > > > actually
> > > > > > >> > > > > > > > > > volume snapshots not hypervisor
> point-in-time
> > > > > > snapshots.
> > > > > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > > > > point-in-time
> > > > > > >> > > > > hypervisor
> > > > > > >> > > > > > > > > > snapshots) and we started referring to the
> > > > original
> > > > > > >> > snapshots
> > > > > > >> > > > as
> > > > > > >> > > > > > > volume
> > > > > > >> > > > > > > > > > snapshots.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > CloudStack does not offer 'backups', but
> many
> > > > people
> > > > > > use
> > > > > > >> > > volume
> > > > > > >> > > > > > > > snapshots
> > > > > > >> > > > > > > > > > as backups. However you can't in-place
> restore
> > > > > volume
> > > > > > >> > > snapshots
> > > > > > >> > > > > and
> > > > > > >> > > > > > > if
> > > > > > >> > > > > > > > > you
> > > > > > >> > > > > > > > > > have a VM with multiple volumes, the volume
> > > > > snapshots
> > > > > > >> must
> > > > > > >> > be
> > > > > > >> > > > > done
> > > > > > >> > > > > > in
> > > > > > >> > > > > > > > > > series, meaning that the state across all of
> > the
> > > > > > >> volumes is
> > > > > > >> > > > > > unlikely
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > be
> > > > > > >> > > > > > > > > > consistent.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > 'Actual Backups' would enable all of the
> > restore
> > > > > > options
> > > > > > >> > > which
> > > > > > >> > > > > > users
> > > > > > >> > > > > > > > > might
> > > > > > >> > > > > > > > > > expect as well options as to where they
> might
> > be
> > > > > > >> stored. In
> > > > > > >> > > my
> > > > > > >> > > > > > ideal
> > > > > > >> > > > > > > > > world
> > > > > > >> > > > > > > > > > they would also be able to leverage back-end
> > > > > hardware
> > > > > > >> (such
> > > > > > >> > > as
> > > > > > >> > > > > > > > Solidfire,
> > > > > > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> > > > > Commvault
> > > > > > >> etc
> > > > > > >> > to
> > > > > > >> > > > > > > accelerate
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > process.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > [image: ShapeBlue] <
> http://www.shapeblue.com>
> > > > > > >> > > > > > > > > > Paul Angus
> > > > > > >> > > > > > > > > > VP Technology ,  ShapeBlue
> > > > > > >> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540
> *
> > > > > > >> > > > > > > > > >
> > > > > > >> > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > |
> > > > > > >> > > > > m:
> > > > > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > >> > > > > > > > > > e:  *paul.angus@shapeblue.com
> <javascript:;>
> > > > > > >> > <javascript:;>
> > > > > > >> > > |
> > > > > > >> > > > > t: @cloudyangus*
> > > > > > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > > >> > > > > <ja...@cloudyangus>
> > > > > > >> > > > > > |  w:
> > > > > > >> > > > > > > > > > *www.shapeblue.com* <
> http://www.shapeblue.com
> > >
> > > > > > >> > > > > > > > > > a:  53 Chandos Place, Covent Garden London
> > WC2N
> > > > 4HS
> > > > > UK
> > > > > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in
> > > > England
> > > > > &
> > > > > > >> > Wales.
> > > > > > >> > > > > > > ShapeBlue
> > > > > > >> > > > > > > > > > Services India LLP is a company incorporated
> > in
> > > > > India
> > > > > > >> and
> > > > > > >> > is
> > > > > > >> > > > > > operated
> > > > > > >> > > > > > > > > under
> > > > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> Brasil
> > > > > > >> Consultoria
> > > > > > >> > > Ltda
> > > > > > >> > > > > is
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > > > > company incorporated in Brasil and is
> operated
> > > > under
> > > > > > >> > license
> > > > > > >> > > > from
> > > > > > >> > > > > > > Shape
> > > > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > > > registered
> > > > > > >> by
> > > > > > >> > The
> > > > > > >> > > > > > > Republic
> > > > > > >> > > > > > > > of
> > > > > > >> > > > > > > > > > South Africa and is traded under license
> from
> > > > Shape
> > > > > > Blue
> > > > > > >> > Ltd.
> > > > > > >> > > > > > > ShapeBlue
> > > > > > >> > > > > > > > > is
> > > > > > >> > > > > > > > > > a registered trademark.
> > > > > > >> > > > > > > > > > This email and any attachments to it may be
> > > > > > confidential
> > > > > > >> > and
> > > > > > >> > > > are
> > > > > > >> > > > > > > > intended
> > > > > > >> > > > > > > > > > solely for the use of the individual to whom
> > it
> > > is
> > > > > > >> > addressed.
> > > > > > >> > > > Any
> > > > > > >> > > > > > > views
> > > > > > >> > > > > > > > > or
> > > > > > >> > > > > > > > > > opinions expressed are solely those of the
> > > author
> > > > > and
> > > > > > do
> > > > > > >> > not
> > > > > > >> > > > > > > > necessarily
> > > > > > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > > > > > companies.
> > > > > > >> If
> > > > > > >> > > you
> > > > > > >> > > > > are
> > > > > > >> > > > > > > not
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > intended recipient of this email, you must
> > > neither
> > > > > > take
> > > > > > >> any
> > > > > > >> > > > > action
> > > > > > >> > > > > > > > based
> > > > > > >> > > > > > > > > > upon its contents, nor copy or show it to
> > > anyone.
> > > > > > Please
> > > > > > >> > > > contact
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > sender
> > > > > > >> > > > > > > > > > if you believe you have received this email
> in
> > > > > error.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > -----Original Message-----
> > > > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > syed1.mushtaq@gmail.com
> > > > > > >> > > > > <javascript:;> <javascript:;>]
> > > > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > >> > > > > > > > > > To: dev@cloudstack.apache.org
> <javascript:;>
> > > > > > >> > <javascript:;>
> > > > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > > > Snapshots
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Paul,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > When you say actual backups, how would it be
> > > > > different
> > > > > > >> from
> > > > > > >> > > the
> > > > > > >> > > > > > > Volume
> > > > > > >> > > > > > > > > > Snapshots that exist currently. My
> > understanding
> > > > is
> > > > > > that
> > > > > > >> > > > Backups
> > > > > > >> > > > > > end
> > > > > > >> > > > > > > up
> > > > > > >> > > > > > > > > in
> > > > > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > > > > point-in-time
> > > > > > >> > state
> > > > > > >> > > of
> > > > > > >> > > > > > your
> > > > > > >> > > > > > > > > volume
> > > > > > >> > > > > > > > > > which can be restored back correct?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > -Syed
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus
> <
> > > > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> > > <javascript:;>>
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > Hi Syed,
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > As I understand it, the SolidFire plugin
> > will
> > > > > export
> > > > > > >> the
> > > > > > >> > > > > snapshot
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > > > secondary storage if the user requests a
> > > > template
> > > > > > from
> > > > > > >> > the
> > > > > > >> > > > > > snapshot
> > > > > > >> > > > > > > > or
> > > > > > >> > > > > > > > > > > wants to download the snapshot from the
> > cloud.
> > > > > This
> > > > > > >> is a
> > > > > > >> > > > good,
> > > > > > >> > > > > > > > > > > pragmatic approach and yes Mike the
> > SolidFire
> > > > > > storage
> > > > > > >> is
> > > > > > >> > > > super
> > > > > > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays
> > > take
> > > > up
> > > > > > >> next
> > > > > > >> > to
> > > > > > >> > > no
> > > > > > >> > > > > > > space.
> > > > > > >> > > > > > > > > > > BUT I think that we are talking about a
> more
> > > > > general
> > > > > > >> > > purpose
> > > > > > >> > > > > API,
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > > > > other storage systems may not be as
> awesome
> > as
> > > > > > Mike's.
> > > > > > >> > > That's
> > > > > > >> > > > > my
> > > > > > >> > > > > > > > > > > concern. Also, the time to transfer for
> say
> > > 1TB
> > > > to
> > > > > > >> move
> > > > > > >> > > from
> > > > > > >> > > > > > > primary
> > > > > > >> > > > > > > > > > > to sec storage and then create a VM
> template
> > > out
> > > > > of
> > > > > > it
> > > > > > >> > may
> > > > > > >> > > be
> > > > > > >> > > > > too
> > > > > > >> > > > > > > > long
> > > > > > >> > > > > > > > > > for users.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term
> volume
> > > > > > snapshot
> > > > > > >> for
> > > > > > >> > > > > backup,
> > > > > > >> > > > > > > > it's
> > > > > > >> > > > > > > > > > > just that users want to do backups and a
> > > volume
> > > > > > >> snapshot
> > > > > > >> > is
> > > > > > >> > > > the
> > > > > > >> > > > > > > only
> > > > > > >> > > > > > > > > > > type of snapshot that copies the disk
> > > elsewhere
> > > > > and
> > > > > > >> can
> > > > > > >> > be
> > > > > > >> > > > > > > scheduled.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > I'm 'pondering' the implications of
> enabling
> > > > > actual
> > > > > > >> > backups
> > > > > > >> > > > > > > (through
> > > > > > >> > > > > > > > > > > recognised backup providers) and the user
> > > > > > requirements
> > > > > > >> > > around
> > > > > > >> > > > > > them
> > > > > > >> > > > > > > > > > > (particularly restoration use cases) as a
> > > > separate
> > > > > > >> thread
> > > > > > >> > > of
> > > > > > >> > > > > > work.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > [image: ShapeBlue] <
> > http://www.shapeblue.com>
> > > > > Paul
> > > > > > >> Angus
> > > > > > >> > > VP
> > > > > > >> > > > > > > > Technology
> > > > > > >> > > > > > > > > > > , ShapeBlue
> > > > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> 0540*
> > > > > > >> > > > > > > > > > >
> > > > > > >> > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > >> > > > |
> > > > > > >> > > > > m:
> > > > > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com
> <javascript:;>
> > > > > > >> > > <javascript:;> |
> > > > > > >> > > > > t: @cloudyangus*
> > > > > > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > > >> <javascript:;>
> > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > >> > > > > > > > > > > *www.shapeblue.com* <
> > http://www.shapeblue.com
> > > >
> > > > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> > WC2N
> > > > 4HS
> > > > > > UK
> > > > > > >> > Shape
> > > > > > >> > > > > Blue
> > > > > > >> > > > > > > Ltd
> > > > > > >> > > > > > > > > > > is a company incorporated in England &
> > Wales.
> > > > > > >> ShapeBlue
> > > > > > >> > > > > Services
> > > > > > >> > > > > > > > India
> > > > > > >> > > > > > > > > > > LLP is a company incorporated in India and
> > is
> > > > > > operated
> > > > > > >> > > under
> > > > > > >> > > > > > > license
> > > > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> > > > Consultoria
> > > > > > >> Ltda
> > > > > > >> > is
> > > > > > >> > > a
> > > > > > >> > > > > > > company
> > > > > > >> > > > > > > > > > > incorporated in Brasil and is operated
> under
> > > > > license
> > > > > > >> from
> > > > > > >> > > > Shape
> > > > > > >> > > > > > > Blue
> > > > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> > > > registered
> > > > > by
> > > > > > >> The
> > > > > > >> > > > > Republic
> > > > > > >> > > > > > > of
> > > > > > >> > > > > > > > > > > South Africa and is traded under license
> > from
> > > > > Shape
> > > > > > >> Blue
> > > > > > >> > > Ltd.
> > > > > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > > > >> > > > > > > > > > > This email and any attachments to it may
> be
> > > > > > >> confidential
> > > > > > >> > > and
> > > > > > >> > > > > are
> > > > > > >> > > > > > > > > > > intended solely for the use of the
> > individual
> > > to
> > > > > > whom
> > > > > > >> it
> > > > > > >> > is
> > > > > > >> > > > > > > > addressed.
> > > > > > >> > > > > > > > > > > Any views or opinions expressed are solely
> > > those
> > > > > of
> > > > > > >> the
> > > > > > >> > > > author
> > > > > > >> > > > > > and
> > > > > > >> > > > > > > do
> > > > > > >> > > > > > > > > > > not necessarily represent those of Shape
> > Blue
> > > > Ltd
> > > > > or
> > > > > > >> > > related
> > > > > > >> > > > > > > > > > > companies. If you are not the intended
> > > recipient
> > > > > of
> > > > > > >> this
> > > > > > >> > > > email,
> > > > > > >> > > > > > you
> > > > > > >> > > > > > > > > > > must neither take any action based upon
> its
> > > > > > contents,
> > > > > > >> nor
> > > > > > >> > > > copy
> > > > > > >> > > > > or
> > > > > > >> > > > > > > > show
> > > > > > >> > > > > > > > > > > it to anyone. Please contact the sender if
> > you
> > > > > > believe
> > > > > > >> > you
> > > > > > >> > > > have
> > > > > > >> > > > > > > > > received
> > > > > > >> > > > > > > > > > this email in error.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > -----Original Message-----
> > > > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > > syed1.mushtaq@gmail.com
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > <javascript:;>]
> > > > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org
> > <javascript:;>
> > > > > > >> > > <javascript:;>
> > > > > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature]
> Storage
> > > > > > Snapshots
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > I think the terminology confusion comes
> from
> > > AWS
> > > > > > where
> > > > > > >> > they
> > > > > > >> > > > do
> > > > > > >> > > > > > EBS
> > > > > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack
> > sort
> > > of
> > > > > > >> followed
> > > > > > >> > > > that.
> > > > > > >> > > > > > And
> > > > > > >> > > > > > > > as
> > > > > > >> > > > > > > > > > > an end user who is oblivious to the
> > internals
> > > of
> > > > > my
> > > > > > >> > > provider,
> > > > > > >> > > > > my
> > > > > > >> > > > > > > > > > > expectation would be something similar to
> > what
> > > > AWS
> > > > > > as
> > > > > > >> > that
> > > > > > >> > > is
> > > > > > >> > > > > my
> > > > > > >> > > > > > > > > > > biggest reference point.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > To your point Mike, I agree that a Primary
> > > > Storage
> > > > > > >> > failure
> > > > > > >> > > on
> > > > > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> > > > motivations
> > > > > > >> for us
> > > > > > >> > > to
> > > > > > >> > > > > push
> > > > > > >> > > > > > > > data
> > > > > > >> > > > > > > > > > > to secondary storage. Primary storage
> > (atleast
> > > > for
> > > > > > us)
> > > > > > >> > > costs
> > > > > > >> > > > > > > around 3
> > > > > > >> > > > > > > > > > > times as much as secondary storage and
> > > snapshots
> > > > > on
> > > > > > >> > primary
> > > > > > >> > > > > > storage
> > > > > > >> > > > > > > > > > > are rarely used (especially for some of
> our
> > > > > > customers
> > > > > > >> who
> > > > > > >> > > do
> > > > > > >> > > > > > daily
> > > > > > >> > > > > > > > > > backups).
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike
> > > Tutkowski
> > > > <
> > > > > > >> > > > > > > > > > > mike.tutkowski@solidfire.com
> <javascript:;>
> > > > > > >> > > <javascript:;>>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Some of the weirdness is around
> > terminology.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > For most systems I've worked on, a
> > snapshot
> > > > and
> > > > > a
> > > > > > >> > backup
> > > > > > >> > > > are
> > > > > > >> > > > > > two
> > > > > > >> > > > > > > > > > > > completely different things (but
> > CloudStack
> > > > has
> > > > > > >> > > > traditionally
> > > > > > >> > > > > > > used
> > > > > > >> > > > > > > > > > > > the term "volume snapshot" to mean
> > backup).
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here
> and
> > > say,
> > > > > > >> though,
> > > > > > >> > > that
> > > > > > >> > > > > if
> > > > > > >> > > > > > > your
> > > > > > >> > > > > > > > > > > > primary storage is running on SolidFire
> > that
> > > > it
> > > > > is
> > > > > > >> > > unlikely
> > > > > > >> > > > > > > you'll
> > > > > > >> > > > > > > > > > > > encounter an issue where your primary
> > > storage
> > > > > goes
> > > > > > >> > > offline
> > > > > > >> > > > > (and
> > > > > > >> > > > > > > > > > > > you'll even maintain your performance
> > > > guarantees
> > > > > > >> during
> > > > > > >> > > > > failure
> > > > > > >> > > > > > > > > > > > scenarios and upgrades, as well). That
> > being
> > > > the
> > > > > > >> case,
> > > > > > >> > it
> > > > > > >> > > > is
> > > > > > >> > > > > > less
> > > > > > >> > > > > > > > > > > > useful to require a backup to Swift (but
> > > it's
> > > > > > >> perfectly
> > > > > > >> > > OK
> > > > > > >> > > > if
> > > > > > >> > > > > > > > that's
> > > > > > >> > > > > > > > > > > > what we want to do
> > > > > > >> > > > > > > > > > > here).
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed
> > Mushtaq
> > > > > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > > > > > >> > <javascript:;>>
> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Paul,
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > I believe with the current
> > implementation
> > > of
> > > > > > >> > Snapshots
> > > > > > >> > > on
> > > > > > >> > > > > > > managed
> > > > > > >> > > > > > > > > > > > > storage
> > > > > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never
> > > exported
> > > > > to
> > > > > > >> the
> > > > > > >> > > > > secondary
> > > > > > >> > > > > > > > > > storage.
> > > > > > >> > > > > > > > > > > > > While this solves the problem of
> having
> > > > > > snapshots
> > > > > > >> > > taking
> > > > > > >> > > > > > > forever
> > > > > > >> > > > > > > > > > > > > to get to sec storage, this leaves us
> > > with a
> > > > > > >> > > > > > > > > > > > huge
> > > > > > >> > > > > > > > > > > > > liability if our primary storage goes
> > > down.
> > > > We
> > > > > > see
> > > > > > >> > > > > snapshots
> > > > > > >> > > > > > as
> > > > > > >> > > > > > > > > > > > > our recovery path because we store
> them
> > in
> > > > > Swift
> > > > > > >> > which
> > > > > > >> > > is
> > > > > > >> > > > > > > > reliable
> > > > > > >> > > > > > > > > > > > > and resilient to failures.
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > With Storage snpashots our goal is to
> > have
> > > > > > Volume
> > > > > > >> > > > snapshots
> > > > > > >> > > > > > > > always
> > > > > > >> > > > > > > > > > > > > backed up to secondary storage and
> > Storage
> > > > > > >> Snapshots
> > > > > > >> > > stay
> > > > > > >> > > > > on
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > > > > primary
> > > > > > >> > > > > > > > > > > > storage.
> > > > > > >> > > > > > > > > > > > > A provider could potentially mix both
> > > these
> > > > > and
> > > > > > >> solve
> > > > > > >> > > the
> > > > > > >> > > > > > > problem
> > > > > > >> > > > > > > > > > > > > that you mentioned where you want to
> > meet
> > > > > user's
> > > > > > >> > > > > expectation
> > > > > > >> > > > > > > of a
> > > > > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage)
> > while
> > > > > having
> > > > > > >> an
> > > > > > >> > > > ability
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the
> > > > device)
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > > > > -Syed
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul
> > Angus
> > > > > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com
> > <javascript:;>
> > > > > > >> > > <javascript:;>>
> > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > HI guys,
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Could someone point me to the Jira
> bug
> > > of
> > > > FS
> > > > > > for
> > > > > > >> > the
> > > > > > >> > > > > > > > > > > > > > SAN-snapshot
> > > > > > >> > > > > > > > > > > > feature
> > > > > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > From my discussions with users and
> > > > operators
> > > > > > >> around
> > > > > > >> > > > > > snapshots
> > > > > > >> > > > > > > > > > > > > > I'd make
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > following observations:
> > > > > > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups
> > > (both
> > > > > > >> long-term
> > > > > > >> > > and
> > > > > > >> > > > > > short
> > > > > > >> > > > > > > > > > > > > > term)
> > > > > > >> > > > > > > > > > > > with
> > > > > > >> > > > > > > > > > > > > > the expectation that they can use
> them
> > > for
> > > > > > >> recovery
> > > > > > >> > > if
> > > > > > >> > > > > > > > required.
> > > > > > >> > > > > > > > > > > > > > b. operators fall back to snapshots
> if
> > > > > > something
> > > > > > >> > has
> > > > > > >> > > > gone
> > > > > > >> > > > > > > wrong
> > > > > > >> > > > > > > > > > > > > > with primary storage.
> > > > > > >> > > > > > > > > > > > > > c. users sometimes want to be able
> to
> > > > export
> > > > > > >> > > snapshots
> > > > > > >> > > > as
> > > > > > >> > > > > > > well
> > > > > > >> > > > > > > > > > > > > > as
> > > > > > >> > > > > > > > > > > > create
> > > > > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > > > > >> > > > > > > > > > > > > > d. snapshots are a currently a
> massive
> > > > pain
> > > > > > for
> > > > > > >> > > > > operators,
> > > > > > >> > > > > > I
> > > > > > >> > > > > > > > > > > > > > know at
> > > > > > >> > > > > > > > > > > > > least
> > > > > > >> > > > > > > > > > > > > > one public cloud who have snapshots
> > > which
> > > > > > take 2
> > > > > > >> > days
> > > > > > >> > > > to
> > > > > > >> > > > > > > > > complete.
> > > > > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be
> > used
> > > > for
> > > > > > >> > multiple
> > > > > > >> > > > LVM
> > > > > > >> > > > > > > > disks.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > I think the process Mike has used in
> > the
> > > > > > >> SolidFire
> > > > > > >> > > > plugin
> > > > > > >> > > > > > > (only
> > > > > > >> > > > > > > > > > > > > > moving
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > disk image to secondary storage when
> > you
> > > > > > >> absolutely
> > > > > > >> > > > have
> > > > > > >> > > > > > to)
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > > > > > > > a very
> > > > > > >> > > > > > > > > > > > > good
> > > > > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder
> what
> > > > > problems
> > > > > > >> an
> > > > > > >> > > > > operator
> > > > > > >> > > > > > > > might
> > > > > > >> > > > > > > > > > > > > experience
> > > > > > >> > > > > > > > > > > > > > if they have an issue with a given
> > > primary
> > > > > > >> storage
> > > > > > >> > > pool
> > > > > > >> > > > > in
> > > > > > >> > > > > > a
> > > > > > >> > > > > > > > > > cluster.
> > > > > > >> > > > > > > > > > > > (I
> > > > > > >> > > > > > > > > > > > > > know that that is REALLY unlikely in
> > the
> > > > > > >> SolidFire
> > > > > > >> > > case
> > > > > > >> > > > > > Mike
> > > > > > >> > > > > > > :)
> > > > > > >> > > > > > > > > > > > > > ) And
> > > > > > >> > > > > > > > > > > > if
> > > > > > >> > > > > > > > > > > > > > the transfer from primary to
> secondary
> > > is
> > > > > > slow,
> > > > > > >> the
> > > > > > >> > > > time
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > > > > > > > being able
> > > > > > >> > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > create a template or export the
> volume
> > > > will
> > > > > be
> > > > > > >> > slow.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > So for me the issue is around making
> > > sure
> > > > > that
> > > > > > >> the
> > > > > > >> > > end
> > > > > > >> > > > > > users
> > > > > > >> > > > > > > > > > > > expectations
> > > > > > >> > > > > > > > > > > > > > are met (while improving the
> > > > > speed/efficiency
> > > > > > of
> > > > > > >> > the
> > > > > > >> > > > back
> > > > > > >> > > > > > > end)
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > > > > http://www.shapeblue.com>
> > > > > > >> Paul
> > > > > > >> > > > Angus
> > > > > > >> > > > > > VP
> > > > > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > > > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203
> 603
> > > > 0540*
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > >
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > >> > > > > > > |
> > > > > > >> > > > > > > > m:
> > > > > > >> > > > > > > > > > > > > > *+44 7711 418784*
> > <+44%207711%20418784>
> > > > > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;> | t:
> > > > > > >> > > > > > @cloudyangus*
> > > > > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com
> > > <javascript:;>
> > > > > > >> > > > <javascript:;>
> > > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > > > > http://www.shapeblue.com
> > > > > > >
> > > > > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > > London
> > > > > WC2N
> > > > > > >> 4HS
> > > > > > >> > UK
> > > > > > >> > > > > Shape
> > > > > > >> > > > > > > > Blue
> > > > > > >> > > > > > > > > > > > > > Ltd is a company incorporated in
> > > England &
> > > > > > >> Wales.
> > > > > > >> > > > > ShapeBlue
> > > > > > >> > > > > > > > > > > > > > Services India LLP is a company
> > > > incorporated
> > > > > > in
> > > > > > >> > India
> > > > > > >> > > > and
> > > > > > >> > > > > > is
> > > > > > >> > > > > > > > > > > > > > operated
> > > > > > >> > > > > > > > > > > > > under
> > > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape
> > Blue
> > > > > Brasil
> > > > > > >> > > > > Consultoria
> > > > > > >> > > > > > > Ltda
> > > > > > >> > > > > > > > > > > > > > is a company incorporated in Brasil
> > and
> > > is
> > > > > > >> operated
> > > > > > >> > > > under
> > > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd.
> ShapeBlue
> > > SA
> > > > > Pty
> > > > > > >> Ltd
> > > > > > >> > is
> > > > > > >> > > a
> > > > > > >> > > > > > > company
> > > > > > >> > > > > > > > > > > > > > registered by The Republic
> > > > > > >> > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > > > South Africa and is traded under
> > license
> > > > > from
> > > > > > >> Shape
> > > > > > >> > > > Blue
> > > > > > >> > > > > > Ltd.
> > > > > > >> > > > > > > > > > > > > > ShapeBlue
> > > > > > >> > > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > a registered trademark.
> > > > > > >> > > > > > > > > > > > > > This email and any attachments to it
> > may
> > > > be
> > > > > > >> > > > confidential
> > > > > > >> > > > > > and
> > > > > > >> > > > > > > > are
> > > > > > >> > > > > > > > > > > > intended
> > > > > > >> > > > > > > > > > > > > > solely for the use of the individual
> > to
> > > > whom
> > > > > > it
> > > > > > >> is
> > > > > > >> > > > > > addressed.
> > > > > > >> > > > > > > > > > > > > > Any views
> > > > > > >> > > > > > > > > > > > > or
> > > > > > >> > > > > > > > > > > > > > opinions expressed are solely those
> of
> > > the
> > > > > > >> author
> > > > > > >> > and
> > > > > > >> > > > do
> > > > > > >> > > > > > not
> > > > > > >> > > > > > > > > > > > necessarily
> > > > > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or
> > > > related
> > > > > > >> > > companies.
> > > > > > >> > > > > If
> > > > > > >> > > > > > > you
> > > > > > >> > > > > > > > > > > > > > are not
> > > > > > >> > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > intended recipient of this email,
> you
> > > must
> > > > > > >> neither
> > > > > > >> > > take
> > > > > > >> > > > > any
> > > > > > >> > > > > > > > > > > > > > action
> > > > > > >> > > > > > > > > > > > based
> > > > > > >> > > > > > > > > > > > > > upon its contents, nor copy or show
> it
> > > to
> > > > > > >> anyone.
> > > > > > >> > > > Please
> > > > > > >> > > > > > > > contact
> > > > > > >> > > > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > sender
> > > > > > >> > > > > > > > > > > > > > if you believe you have received
> this
> > > > email
> > > > > in
> > > > > > >> > error.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > -----Original Message-----
> > > > > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > > > > >> pdion@cloudops.com
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > <javascript:;>]
> > > > > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56
> > PM
> > > > > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature]
> > > > Storage
> > > > > > >> > Snapshots
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Hi Mike,
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > > > > >> StorageSnapshot
> > > > > > >> > > for
> > > > > > >> > > > > > > managed
> > > > > > >> > > > > > > > > > > > > > storage
> > > > > > >> > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > because the VolumeSnapshot default,
> or
> > > > > > expected,
> > > > > > >> > > > behavior
> > > > > > >> > > > > > is
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> > > > > Storage.
> > > > > > >> So a
> > > > > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > > > > >> > > > > > > > > > > > > for
> > > > > > >> > > > > > > > > > > > > > snapshot that remain on the managed
> > > > storage
> > > > > > >> > > appliance.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Quickly looking at the API doc and I
> > > don't
> > > > > > see a
> > > > > > >> > > strong
> > > > > > >> > > > > > > > > > > > > > requirement for volume snapshots to
> be
> > > > moved
> > > > > > >> into
> > > > > > >> > > > > secondary
> > > > > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot
> API
> > > is
> > > > > not
> > > > > > >> > useful,
> > > > > > >> > > > but
> > > > > > >> > > > > > > both
> > > > > > >> > > > > > > > > > > > > > use cases are required. A snapshot
> > that
> > > > > remain
> > > > > > >> on
> > > > > > >> > the
> > > > > > >> > > > > > managed
> > > > > > >> > > > > > > > > > > > > > storage, and another type of
> > > > > > >> > > > > > > > > > > > snapshot
> > > > > > >> > > > > > > > > > > > > > that end up into the secondary
> > storage.
> > > > > Since
> > > > > > >> > you've
> > > > > > >> > > > > done a
> > > > > > >> > > > > > > lot
> > > > > > >> > > > > > > > > > > > > > of
> > > > > > >> > > > > > > > > > > > work,
> > > > > > >> > > > > > > > > > > > > > might easier to just add a parameter
> > to
> > > > the
> > > > > > >> current
> > > > > > >> > > > > > snapshot
> > > > > > >> > > > > > > > API
> > > > > > >> > > > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > > > trigger an extraction of the storage
> > > > > snapshot
> > > > > > >> into
> > > > > > >> > > the
> > > > > > >> > > > > > > > secondary
> > > > > > >> > > > > > > > > > > > storage?
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > PL
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > > > > > Tutkowski <
> > > > > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > I think that all sounds reasonable
> > > then
> > > > -
> > > > > > >> thanks!
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM,
> Syed
> > > > > > Mushtaq <
> > > > > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > >> <javascript:;>>
> > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >> You are correct Mike in terms of
> > the
> > > > > > >> > requirements.
> > > > > > >> > > > One
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > our
> > > > > > >> > > > > > > > > > > > earlier
> > > > > > >> > > > > > > > > > > > > > >> iterations on this was to have an
> > > > > argument
> > > > > > to
> > > > > > >> > the
> > > > > > >> > > > > create
> > > > > > >> > > > > > > > > > > > > > >> snapshot
> > > > > > >> > > > > > > > > > > > API
> > > > > > >> > > > > > > > > > > > > > >> which decides whether to backup
> the
> > > > > volume
> > > > > > to
> > > > > > >> > sec
> > > > > > >> > > > > > storage
> > > > > > >> > > > > > > > but
> > > > > > >> > > > > > > > > > > > > > >> we realized it would make
> > management
> > > of
> > > > > > >> > snapshots
> > > > > > >> > > > > quite
> > > > > > >> > > > > > > > messy
> > > > > > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > > > > >> > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> > > > > Tutkowski
> > > > > > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>>
> > > > > > >> > > > > > > > > > > > > > >> wrote:
> > > > > > >> > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > >>> Hi,
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> Just to make sure I understand
> all
> > > the
> > > > > > >> > > requirements
> > > > > > >> > > > > > here:
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 1) This relates only to managed
> > > > storage
> > > > > > (1:1
> > > > > > >> > > > mapping
> > > > > > >> > > > > > > > between
> > > > > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> > > > > volume).
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > > > > (introduced
> > > > > > >> in
> > > > > > >> > > 4.6)
> > > > > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> > > > snapshot
> > > > > on
> > > > > > >> the
> > > > > > >> > > SAN,
> > > > > > >> > > > > and
> > > > > > >> > > > > > > > > > > > > > >>> extend it via a config option
> (or
> > > > > > >> > > > > > > > > > > > > > >>> something) to not only take the
> > SAN
> > > > > > >> snapshot,
> > > > > > >> > but
> > > > > > >> > > > to
> > > > > > >> > > > > > copy
> > > > > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer
> > only)
> > > to
> > > > > > NFS.
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always
> > taken.
> > > > > It's
> > > > > > >> the
> > > > > > >> > > > backup
> > > > > > >> > > > > to
> > > > > > >> > > > > > > NFS
> > > > > > >> > > > > > > > > > > > > > >>> that is optional.
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 4) Templates can be created from
> > the
> > > > > > >> snapshot
> > > > > > >> > > > that's
> > > > > > >> > > > > on
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be
> > created
> > > > > from
> > > > > > >> the
> > > > > > >> > > > > snapshot
> > > > > > >> > > > > > > > > > > > > > >>> that's on
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>> SAN (already works as long as
> the
> > > new
> > > > > > >> > CloudStack
> > > > > > >> > > > > volume
> > > > > > >> > > > > > > > ends
> > > > > > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> Would we have a need for a
> storage
> > > > > > snapshot
> > > > > > >> API
> > > > > > >> > > > then
> > > > > > >> > > > > or
> > > > > > >> > > > > > > > > > > > > > >>> would that just be the standard
> > > volume
> > > > > > >> snapshot
> > > > > > >> > > > > without
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>> backup to
> > > > > > >> > > > > > > > > > > NFS?
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> Thanks!
> > > > > > >> > > > > > > > > > > > > > >>> Mike
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM,
> > Syed
> > > > > > Mushtaq
> > > > > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>>
> > > > > > >> > > > > > > > > > > > > > >>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > > > > functionalities
> > > > > > >> > > > > (snapshot
> > > > > > >> > > > > > on
> > > > > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > > > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because
> > Ideally,
> > > we
> > > > > > would
> > > > > > >> > like
> > > > > > >> > > > to
> > > > > > >> > > > > > have
> > > > > > >> > > > > > > > > both.
> > > > > > >> > > > > > > > > > > > > > >>>> For example, some of our
> > customers
> > > > want
> > > > > > to
> > > > > > >> > > > implement
> > > > > > >> > > > > > > their
> > > > > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> > > > encryption
> > > > > > to
> > > > > > >> > their
> > > > > > >> > > > > > backups
> > > > > > >> > > > > > > > > > > > > > >>>> which is a perfect use case for
> > > > Storage
> > > > > > >> > Snapshot
> > > > > > >> > > > > while
> > > > > > >> > > > > > > our
> > > > > > >> > > > > > > > > > > > > > >>>> other customers will still keep
> > > using
> > > > > the
> > > > > > >> > > standard
> > > > > > >> > > > > > > volume
> > > > > > >> > > > > > > > > > > snapshot.
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>> To keep things backward
> > compatible,
> > > > we
> > > > > > can
> > > > > > >> > add a
> > > > > > >> > > > > > setting
> > > > > > >> > > > > > > > > > > > > > >>>> which
> > > > > > >> > > > > > > > > > > > says
> > > > > > >> > > > > > > > > > > > > > >>>> to not upload on secondary
> > storage,
> > > > > > >> because,
> > > > > > >> > > after
> > > > > > >> > > > > > all,
> > > > > > >> > > > > > > > you
> > > > > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first
> > > when
> > > > > > doing
> > > > > > >> a
> > > > > > >> > > > Volume
> > > > > > >> > > > > > > > > Snapshot.
> > > > > > >> > > > > > > > > > > > > > >>>> You could stop the process
> there
> > > and
> > > > > not
> > > > > > do
> > > > > > >> > the
> > > > > > >> > > > > > upload.
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>> What do you think about this
> > > > approach?
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > > > > >> > > > > > > > > > > > > > >>>> -Syed
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM,
> > > Mike
> > > > > > >> > Tutkowski <
> > > > > > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking
> out
> > > > load
> > > > > > >> here,
> > > > > > >> > but
> > > > > > >> > > > if
> > > > > > >> > > > > a
> > > > > > >> > > > > > > > given
> > > > > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't
> > actually
> > > > need
> > > > > > to
> > > > > > >> > > provide
> > > > > > >> > > > > > both
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > ability
> > > > > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and
> > export
> > > it
> > > > > to
> > > > > > >> NFS
> > > > > > >> > (if
> > > > > > >> > > > > just
> > > > > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK),
> > then
> > > > we
> > > > > > >> might
> > > > > > >> > be
> > > > > > >> > > > able
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > get
> > > > > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > > > > >> > > > > > > > > > > > API
> > > > > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a
> new
> > > > > custom
> > > > > > >> > > snapshot
> > > > > > >> > > > > > > strategy
> > > > > > >> > > > > > > > > > > > > > >>>>> and
> > > > > > >> > > > > > > > > > > > data
> > > > > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the
> > case
> > > > > where
> > > > > > >> the
> > > > > > >> > > > > > CloudStack
> > > > > > >> > > > > > > > > > > > > > >>>>> cloud
> > > > > > >> > > > > > > > > > > > does
> > > > > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > > > > >> exported-to-NFS
> > > > > > >> > > > > backup.
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> In other words, the "default"
> > > > behavior
> > > > > > >> would
> > > > > > >> > be
> > > > > > >> > > > to
> > > > > > >> > > > > > use
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data
> > motion
> > > > > > strategy
> > > > > > >> > that
> > > > > > >> > > > we
> > > > > > >> > > > > > > > already
> > > > > > >> > > > > > > > > > > > > > >>>>> have (the one that only takes
> a
> > > SAN
> > > > > > >> > snapshot).
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud,
> > however,
> > > > > wants
> > > > > > >> to
> > > > > > >> > > take
> > > > > > >> > > > a
> > > > > > >> > > > > > SAN
> > > > > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data
> > > exported
> > > > to
> > > > > > >> NFS,
> > > > > > >> > > then
> > > > > > >> > > > we
> > > > > > >> > > > > > > could
> > > > > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing
> > config
> > > > > file
> > > > > > to
> > > > > > >> > make
> > > > > > >> > > > use
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > a
> > > > > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data
> > > > motion
> > > > > > >> > strategy
> > > > > > >> > > > that
> > > > > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > > > > >> > > > > > > > > > > > > > activities.
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is
> > > still
> > > > > the
> > > > > > >> > default
> > > > > > >> > > > for
> > > > > > >> > > > > > > > users,
> > > > > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can
> change
> > > > this
> > > > > > >> > behavior
> > > > > > >> > > > via
> > > > > > >> > > > > > > > > > > configuration.
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55
> AM,
> > > > Mike
> > > > > > >> > > Tutkowski <
> > > > > > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need
> to
> > > > come
> > > > > up
> > > > > > >> > with a
> > > > > > >> > > > > > viable
> > > > > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some
> reasonable
> > > way
> > > > > for
> > > > > > >> them
> > > > > > >> > > to
> > > > > > >> > > > > move
> > > > > > >> > > > > > > > from
> > > > > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way
> (and
> > > > some
> > > > > > >> obvious
> > > > > > >> > > way
> > > > > > >> > > > > > that
> > > > > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > > > > >> > > > > > > > > > > > to
> > > > > > >> > > > > > > > > > > > > > do this).
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45
> > AM,
> > > > Syed
> > > > > > >> > Mushtaq
> > > > > > >> > > <
> > > > > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about
> the
> > > > > upgrade
> > > > > > >> path
> > > > > > >> > > > > however,
> > > > > > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6
> > and
> > > > are
> > > > > > on a
> > > > > > >> > > > managed
> > > > > > >> > > > > > > > storage
> > > > > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the
> same
> > > > > > >> functionality
> > > > > > >> > > > with
> > > > > > >> > > > > > > Volume
> > > > > > >> > > > > > > > > > > Snapshots.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43
> > PM,
> > > > Syed
> > > > > > >> > Mushtaq
> > > > > > >> > > <
> > > > > > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> So if I understand
> correctly,
> > > > > > currently
> > > > > > >> > > > taking a
> > > > > > >> > > > > > > > Volume
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a
> > > > managed
> > > > > > >> storage
> > > > > > >> > > > keeps
> > > > > > >> > > > > > it
> > > > > > >> > > > > > > on
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a
> part
> > of
> > > > > this
> > > > > > >> > > feature,
> > > > > > >> > > > we
> > > > > > >> > > > > > can
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > > > > > >> > > > > > > > > > > > that
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed
> > > > storage
> > > > > > are
> > > > > > >> > > > uploaded
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This
> would
> > > > make
> > > > > > the
> > > > > > >> > > Volume
> > > > > > >> > > > > > > Snapshot
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same
> > > > regardless
> > > > > of
> > > > > > >> the
> > > > > > >> > > > > storage
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for
> > utilizing
> > > > the
> > > > > > >> > > efficient
> > > > > > >> > > > > > > backend
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> storage
> > > > > > >> > > > > > > > > > > > > > capabilities, we can use the new
> > storage
> > > > > > >> snapshots
> > > > > > >> > > API.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36
> > PM,
> > > > > Mike
> > > > > > >> > > > Tutkowski <
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> mike.tutkowski@solidfire.com
> > > > > > >> > <javascript:;>
> > > > > > >> > > > > <javascript:;>>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we
> need
> > > to
> > > > > > have a
> > > > > > >> > plan
> > > > > > >> > > > to
> > > > > > >> > > > > > deal
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we
> > already
> > > > > have a
> > > > > > >> > > feature
> > > > > > >> > > > > (in
> > > > > > >> > > > > > > 4.6
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> and
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to
> > use
> > > > the
> > > > > > >> > existing
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to
> > > create a
> > > > > > >> volume
> > > > > > >> > > > > snapshot
> > > > > > >> > > > > > > (for
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a
> > > > backend
> > > > > > SAN
> > > > > > >> > > > (using a
> > > > > > >> > > > > > > > custom
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a
> > custom
> > > > > data
> > > > > > >> > motion
> > > > > > >> > > > > > > strategy).
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in,
> > then
> > > > how
> > > > > > >> should
> > > > > > >> > > the
> > > > > > >> > > > > > > original
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in
> > 4.6
> > > > and
> > > > > > >> later)
> > > > > > >> > > be
> > > > > > >> > > > > > > changed?
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > > > > > >> > > > > > > > > > > > is
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support
> > > > > customers
> > > > > > >> who
> > > > > > >> > > are
> > > > > > >> > > > > > > already
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> using
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot
> API
> > > to
> > > > > take
> > > > > > >> > > > snapshots
> > > > > > >> > > > > > on a
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > > > > > >> > > > > > > > > > > > > SAN?
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at
> 11:27
> > > AM,
> > > > > > Will
> > > > > > >> > > > Stevens <
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > > > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to
> create
> > a
> > > > > > Template
> > > > > > >> > > from a
> > > > > > >> > > > > > > > > > > > StorageSnapshot?
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template
> > be
> > > > > stored
> > > > > > >> in
> > > > > > >> > the
> > > > > > >> > > > > > > secondary
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal
> > templates
> > > > or
> > > > > > will
> > > > > > >> > that
> > > > > > >> > > > be
> > > > > > >> > > > > > > > handled
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > >> > > > > > > > > > > > > > vendor side?
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud
> > > Solutions
> > > > > > >> Experts
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal
> > *|*
> > > > > Quebec
> > > > > > >> *|*
> > > > > > >> > > H3J
> > > > > > >> > > > > 1S6
> > > > > > >> > > > > > w
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw
> > > @CloudOps_
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at
> 1:22
> > > PM,
> > > > > > Syed
> > > > > > >> > > > Mushtaq <
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at
> > 1:19
> > > > PM,
> > > > > > Will
> > > > > > >> > > > Stevens
> > > > > > >> > > > > <
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>> wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the
> > > > Design
> > > > > > >> Spec in
> > > > > > >> > > the
> > > > > > >> > > > > > Jira
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was
> not
> > > > clear
> > > > > > in
> > > > > > >> the
> > > > > > >> > > > > > 'mention'
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it
> shows
> > > as a
> > > > > > page
> > > > > > >> > 'you
> > > > > > >> > > do
> > > > > > >> > > > > not
> > > > > > >> > > > > > > > have
> > > > > > >> > > > > > > > > > > permission to'.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud
> > > > Solutions
> > > > > > >> Experts
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|*
> Montreal
> > > *|*
> > > > > > Quebec
> > > > > > >> > *|*
> > > > > > >> > > > H3J
> > > > > > >> > > > > > 1S6
> > > > > > >> > > > > > > w
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw
> > > > @CloudOps_
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at
> > 1:02
> > > > PM,
> > > > > > >> Syed
> > > > > > >> > > Ahmed
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > >
> > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a
> new
> > > set
> > > > > of
> > > > > > >> APIs
> > > > > > >> > to
> > > > > > >> > > > do
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed
> > > storage
> > > > > > >> backends
> > > > > > >> > > > like
> > > > > > >> > > > > > > > > SolidFire.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current
> > > managed
> > > > > > >> storage
> > > > > > >> > > stay
> > > > > > >> > > > > on
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is
> contrary
> > > to
> > > > > what
> > > > > > >> > > > CloudStack
> > > > > > >> > > > > > > calls
> > > > > > >> > > > > > > > > > > snpshots.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots
> on
> > > > > storage
> > > > > > >> and
> > > > > > >> > > > keeping
> > > > > > >> > > > > > it
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own
> > > advantages
> > > > > > >> > > > > > > > > > > > and
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like
> to
> > > > have
> > > > > > both
> > > > > > >> > ways
> > > > > > >> > > > of
> > > > > > >> > > > > > > doing
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This
> proposal
> > > > adds
> > > > > 4
> > > > > > >> new
> > > > > > >> > > APIs
> > > > > > >> > > > to
> > > > > > >> > > > > > > > create
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend
> > > > storage.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think
> > of
> > > > this
> > > > > > >> > > feature? I
> > > > > > >> > > > > > would
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some
> > > feedback.
> > > > I
> > > > > am
> > > > > > >> > > working
> > > > > > >> > > > on
> > > > > > >> > > > > > > > making
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > >> > > > > > > > > > > > spec
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but
> wanted
> > > to
> > > > > > have a
> > > > > > >> > high
> > > > > > >> > > > > level
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before
> > > > starting
> > > > > > to
> > > > > > >> > work
> > > > > > >> > > on
> > > > > > >> > > > > it.
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> --
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack
> > Developer,
> > > > > > >> SolidFire
> > > > > > >> > > Inc.*
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> e:
> > > mike.tutkowski@solidfire.com
> > > > > > >> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the
> world
> > > uses
> > > > > the
> > > > > > >> > cloud
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> <
> > > > > > >> > > > > > >
> http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > > > > >*™
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>> *
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>> --
> > > > > > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer,
> > > > > SolidFire
> > > > > > >> > Inc.*
> > > > > > >> > > > > > > > > > > > > > >>>>>> e:
> > mike.tutkowski@solidfire.com
> > > > > > >> > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > >>>>>> Advancing the way the world
> > uses
> > > > the
> > > > > > >> cloud
> > > > > > >> > > > > > > > > > > > > > >>>>>> <
> > > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > > > >*™*
> > > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>> --
> > > > > > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer,
> > > > > SolidFire
> > > > > > >> Inc.*
> > > > > > >> > > > > > > > > > > > > > >>>>> e:
> mike.tutkowski@solidfire.com
> > > > > > >> > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > >>>>> Advancing the way the world
> uses
> > > the
> > > > > > cloud
> > > > > > >> > > > > > > > > > > > > > >>>>> <
> > > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > > > >*™*
> > > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>>
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>> --
> > > > > > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer,
> > > > SolidFire
> > > > > > >> Inc.*
> > > > > > >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > >>> Advancing the way the world uses
> > the
> > > > > cloud
> > > > > > >> > > > > > > > > > > > > > >>> <
> > > > > > >> > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > > >*™*
> > > > > > >> > > > > > > > > > > > > > >>>
> > > > > > >> > > > > > > > > > > > > > >>
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > > --
> > > > > > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > > > > *Senior CloudStack Developer,
> > > SolidFire
> > > > > > Inc.*
> > > > > > >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > >> <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > > > > o: 303.746.7302
> > > > > > >> > > > > > > > > > > > > > > Advancing the way the world uses
> the
> > > > cloud
> > > > > > >> > > > > > > > > > > > > > > <
> > > > > > >> > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > > > > >*™*
> > > > > > >> > > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Find out more about ShapeBlue and
> our
> > > > range
> > > > > of
> > > > > > >> > > > CloudStack
> > > > > > >> > > > > > > > > > > > > > related
> > > > > > >> > > > > > > > > > > > > services:
> > > > > > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > >> > > > > > > > > > > > > > <
> > > > > > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > > > > >> > > |
> > > > > > >> > > > > > > > CSForge –
> > > > > > >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > > > > > >> > > > > > > http://shapeblue.com/csforge/
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > CloudStack Consulting
> > > > > > >> > > > > > > > > > > > > > <
> > > > > http://shapeblue.com/cloudstack-consultancy/
> > > > > > >
> > > > > > >> |
> > > > > > >> > > > > > > > > > > > > CloudStack
> > > > > > >> > > > > > > > > > > > > > Software Engineering
> > > > > > >> > > > > > > > > > > > > > <
> > > > > > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > > > > > >> > > > >
> > > > > > >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > >> > > > > > > > > > > > > > <
> > > > > > >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > > >> > > > > >
> > > > > > >> > > > > > |
> > > > > > >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > >> > > > > > > > > > > > > > <
> > > > http://shapeblue.com/cloudstack-training/>
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > --
> > > > > > >> > > > > > > > > > > > *Mike Tutkowski*
> > > > > > >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> > > Inc.*
> > > > > > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > <javascript:;>
> > > > > > >> > > > > <javascript:;>
> > > > > > >> > > > > > > > > > > > o: 303.746.7302
> > > > > > >> > > > > > > > > > > > Advancing the way the world uses the
> cloud
> > > > > > >> > > > > > > > > > > > <
> > > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >> > >*™*
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Find out more about ShapeBlue and our
> range
> > of
> > > > > > >> CloudStack
> > > > > > >> > > > > related
> > > > > > >> > > > > > > > > > services:
> > > > > > >> > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > >> > > > > > > > > > > <
> > > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > > >
> > > > > > >> |
> > > > > > >> > > > > CSForge –
> > > > > > >> > > > > > > > rapid
> > > > > > >> > > > > > > > > > > IaaS deployment framework <
> > > > > > >> http://shapeblue.com/csforge/
> > > > > > >> > >
> > > > > > >> > > > > > > CloudStack
> > > > > > >> > > > > > > > > > > Consulting <
> > > > > > >> http://shapeblue.com/cloudstack-consultancy/
> > > > > > >> > >
> > > > > > >> > > |
> > > > > > >> > > > > > > > CloudStack
> > > > > > >> > > > > > > > > > > Software Engineering
> > > > > > >> > > > > > > > > > > <
> > > > > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > >> > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > >> > > > > > > > > > > <
> > > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > > >> > >
> > > > > > >> > > |
> > > > > > >> > > > > > > > CloudStack
> > > > > > >> > > > > > > > > > > Bootcamp Training Courses <
> > > > > > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > Find out more about ShapeBlue and our range
> of
> > > > > > >> CloudStack
> > > > > > >> > > > related
> > > > > > >> > > > > > > > > services:
> > > > > > >> > > > > > > > > > IaaS Cloud Design & Build
> > > > > > >> > > > > > > > > > <
> > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > >
> > > > > > |
> > > > > > >> > > > CSForge –
> > > > > > >> > > > > > > rapid
> > > > > > >> > > > > > > > > > IaaS deployment framework <
> > > > > > >> http://shapeblue.com/csforge/>
> > > > > > >> > > > > > > > > > CloudStack Consulting <
> > > > > > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > > > >> > > > > > > |
> > > > > > >> > > > > > > > > CloudStack
> > > > > > >> > > > > > > > > > Software Engineering
> > > > > > >> > > > > > > > > > <
> > > > > > http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > >> > > > > > > > > > CloudStack Infrastructure Support
> > > > > > >> > > > > > > > > > <
> > > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > > > > >> > |
> > > > > > >> > > > > > > CloudStack
> > > > > > >> > > > > > > > > > Bootcamp Training Courses <
> > > > > > >> > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > --
> > > > > > >> > > > > > > *Mike Tutkowski*
> > > > > > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > <javascript:;>
> > > > > > >> > > > > > > o: 303.746.7302
> > > > > > >> > > > > > > Advancing the way the world uses the cloud
> > > > > > >> > > > > > > <
> http://solidfire.com/solution/overview/?video=play
> > > >*™*
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > --
> > > > > > >> > > > > *Mike Tutkowski*
> > > > > > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > >> > > > > o: 303.746.7302
> > > > > > >> > > > > Advancing the way the world uses the cloud
> > > > > > >> > > > > <http://solidfire.com/solution/overview/?video=play
> >*™*
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > --
> > > > > > >> > > > Ian Rae
> > > > > > >> > > > CEO | PDG
> > > > > > >> > > > c: 514.944.4008
> > > > > > >> > > >
> > > > > > >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> > > > > > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada |
> H3J
> > > 1S6
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > *Mike Tutkowski*
> > > > > > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >> > e: mike.tutkowski@solidfire.com
> > > > > > >> > o: 303.746.7302
> > > > > > >> > Advancing the way the world uses the cloud
> > > > > > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Mike Tutkowski*
> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > o: 303.746.7302
> > > > > > > Advancing the way the world uses the cloud
> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > e: mike.tutkowski@solidfire.com
> > > > > > o: 303.746.7302
> > > > > > Advancing the way the world uses the cloud
> > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Will Stevens <ws...@cloudops.com>]

A global setting would probably be fine.  How would the case be handled if
the global setting is changed?  Would it only affect the snapshots created
after the change was made?  We would also code defensively so if the global
setting changes that we don't assume all the snapshots in the past had the
same global setting when they were created.

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Mon, Feb 8, 2016 at 2:16 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> Correct, Will.
>
> That Global Settings would only be for managed storage. Non-managed
> (traditional) volume snapshots are completely un-impacted by this feature.
>
> If we need to sometimes keep the snapshots on the SAN and sometimes push
> them to secondary storage, we'll need a more robust solution than Global
> Settings, though.
>
> On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > Sorry.  I missed a bit of context when I responded.  The global setting
> > would be only for the managed storage case, currently being called
> Storage
> > Snapshots, and is only to determine if a copy is pushed to secondary
> > storage right?  The global setting would not change the behavior of the
> > Volume Snapshots right?
> >
> > I was referring to the need for Volume Snapshots and Storage Snapshots to
> > exist together.  Disregard my comment.  I caught up on context after I
> > posted.  My bad...
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com
> > > wrote:
> >
> > > Hey Will,
> > >
> > > Who's picking the behavior? Is it the cloud provider or the end user?
> > >
> > > Thanks
> > >
> > > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > > > I don't think a global setting is a good option because we need both
> > > > functionality to be available at the same time and for different use
> > > cases
> > > > to be able to pick which they choose.
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > *CloudOps* *| *Cloud Solutions Experts
> > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > w cloudops.com *|* tw @CloudOps_
> > > >
> > > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com
> > > > > wrote:
> > > >
> > > > > Now that I re-read your e-mail, it dawned on me: The end-user
> doesn't
> > > > care
> > > > > where the snapshot is.
> > > > >
> > > > > If that's true, then we should perhaps control this via Global
> > Settings
> > > > or
> > > > > something.
> > > > >
> > > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com> wrote:
> > > > >
> > > > > > It's not ideal - true, but it does allow us to be backward
> > > compatible.
> > > > > >
> > > > > > If you have other ideas, though, about how to maintain backward
> > > > > > compatibility, I'm definitely open to hear them.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > > syed1.mushtaq@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Mike,
> > > > > >>
> > > > > >> Adding a flag to createSnapshot was the first and the most
> obvious
> > > > thing
> > > > > >> that came to our minds. The problem that I had with this was
> that:
> > > > > >>
> > > > > >> 1) I feel it is exposing something to the end user that is
> > internal
> > > to
> > > > > the
> > > > > >> cloud.
> > > > > >>
> > > > > >> 2) We have to follow two different ways of restore/deletion in
> the
> > > > same
> > > > > >> code path depending on where the Snapshot resides which I find
> > kind
> > > > of a
> > > > > >> bad design.
> > > > > >>
> > > > > >> But if exposing a archive flag to the end user is acceptable
> then
> > we
> > > > can
> > > > > >> definitely use this instead of adding the StorageSnapshot API
> > > > > >>
> > > > > >> Thanks,
> > > > > >> -Syed
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > > >> mike.tutkowski@solidfire.com
> > > > > >> > wrote:
> > > > > >>
> > > > > >> > Hi Pierre-Luc,
> > > > > >> >
> > > > > >> > My recommendation would be this:
> > > > > >> >
> > > > > >> > Add an "archive" flag to the current volume-snapshot API. Its
> > > > default
> > > > > >> would
> > > > > >> > be "false" because that would be backward compatible with how
> > 4.6
> > > > has
> > > > > >> > volume snapshots implemented (i.e. they stay on the SAN in
> 4.6,
> > > 4.7,
> > > > > and
> > > > > >> > 4.8).
> > > > > >> >
> > > > > >> > If you set archive=true, then we would perform a background
> > > > migration
> > > > > of
> > > > > >> > the snapshot from the SAN to the secondary storage (then
> delete
> > > the
> > > > > SAN
> > > > > >> > snapshot).
> > > > > >> >
> > > > > >> > That archive parameter would only be valid for managed
> storage.
> > > > > >> >
> > > > > >> > Sound reasonable?
> > > > > >> >
> > > > > >> > Also, a VM snapshot that includes disks provided by managed
> > > storage
> > > > > >> should
> > > > > >> > work.
> > > > > >> >
> > > > > >> > Talk to you later,
> > > > > >> > Mike
> > > > > >> >
> > > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > > pdion@cloudops.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Mike,
> > > > > >> > >
> > > > > >> > > In terms of API's, would you prefer introducing a parameter
> to
> > > the
> > > > > >> > existing
> > > > > >> > > VolumeSnapshot, example:   extract={true|false}  with a
> > default
> > > > > value
> > > > > >> of
> > > > > >> > > true  which would extract snapshot into the secondary
> storage,
> > > > which
> > > > > >> is
> > > > > >> > the
> > > > > >> > > current default behavior. Then for SAN snapshot that remain
> on
> > > the
> > > > > >> SAN we
> > > > > >> > > would just set "extract=false" ?  as oppose to create a new
> > > > > >> > >  StorageSnapshot API ?
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Paul,
> > > > > >> > >
> > > > > >> > > From what I'm seeing so far, we can't do a VM-snapshot when
> > > using
> > > > > >> managed
> > > > > >> > > storage for VM having more than one Volume. For the reason
> > that
> > > > > >> snapshot
> > > > > >> > > are performed outside of the hypervisor awareness and
> > > > > asynchronously.
> > > > > >> If
> > > > > >> > > someone have a way to address that, it would make thinks
> much
> > > more
> > > > > >> > > attractive.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <irae@cloudops.com
> >
> > > > wrote:
> > > > > >> > >
> > > > > >> > > > I think a service provider backup scenario is more likely
> to
> > > > take
> > > > > >> > > advantage
> > > > > >> > > > of SAN snapshot. There are a few reasons for this.
> > Traditional
> > > > > >> backups
> > > > > >> > > > involve access to the file system, and there is an
> > expectation
> > > > > that
> > > > > >> > this
> > > > > >> > > > can be done with reasonably short time frames without
> > > negatively
> > > > > >> > > impacting
> > > > > >> > > > VM performance, and that the backup orchestrator can apply
> > > > various
> > > > > >> > logic
> > > > > >> > > > and or transformations to the data (compress, encrypt,
> > deltas
> > > > > >> etc...).
> > > > > >> > > > While it is true that one could apply a backup process to
> a
> > > > cloud
> > > > > >> > > snapshot,
> > > > > >> > > > this would be slow and inefficient requiring the data to
> be
> > > > moved
> > > > > >> > several
> > > > > >> > > > times and there are some major bottlenecks with cloud
> > > snapshots.
> > > > > >> With a
> > > > > >> > > > cloud snapshot - there seems to be no reasonable
> expectation
> > > of
> > > > > >> being
> > > > > >> > > able
> > > > > >> > > > to do differential snapshots (I think this depends on the
> > > > > >> hypervisor)
> > > > > >> > and
> > > > > >> > > > if you do differential snapshots this will make file
> backups
> > > > from
> > > > > >> those
> > > > > >> > > > snapshots even more complicated to orchestrate.
> > > > > >> > > >
> > > > > >> > > > Suspect there needs to be a different thread on how to
> > better
> > > > > enable
> > > > > >> > > > backups, as a service. As per Paul's suggestion, but it
> is a
> > > > > related
> > > > > >> > > > workflow so relevant to this discussion.
> > > > > >> > > >
> > > > > >> > > > Ian
> > > > > >> > > >
> > > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > > >> > > mike.tutkowski@solidfire.com>
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > To me it sounds like number two and number three are
> > > different
> > > > > >> uses
> > > > > >> > for
> > > > > >> > > > the
> > > > > >> > > > > same "thing"(which is totally fine).
> > > > > >> > > > >
> > > > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > > > >> asynchronously, do
> > > > > >> > > we
> > > > > >> > > > > see the SSVM as performing the export?
> > > > > >> > > > >
> > > > > >> > > > > To be backwards compatible with what we have in 4.6 and
> > > later
> > > > > for
> > > > > >> > > volume
> > > > > >> > > > > snapshots for managed storage, I think it might be
> easier
> > if
> > > > we
> > > > > >> pass
> > > > > >> > > in a
> > > > > >> > > > > flag that says whether or not to archive the SAN
> snapshot
> > > > > (which,
> > > > > >> I
> > > > > >> > > > think,
> > > > > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > > > > >> > > > >
> > > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > > pdion@cloudops.com
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi Mike,
> > > > > >> > > > > >
> > > > > >> > > > > > The reason behind the creation of a SAN snapshot which
> > is
> > > > > >> exported
> > > > > >> > > into
> > > > > >> > > > > > secondary storage, is because creating a copy of the
> > .VHD
> > > > > >> directly
> > > > > >> > > > would
> > > > > >> > > > > > impact uptime of the VM as creating that copy take
> lots
> > of
> > > > > time.
> > > > > >> > Has
> > > > > >> > > > > oppose
> > > > > >> > > > > > to a SAN snapshot that is close to instantaneous which
> > can
> > > > > >> > afterward
> > > > > >> > > be
> > > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > > >> > > > > >
> > > > > >> > > > > > I would suspect an extracted VolumeSnapshot taken
> from a
> > > SAN
> > > > > >> > snapshot
> > > > > >> > > > > could
> > > > > >> > > > > > have is SAN snapshot deleted to avoid duplica and
> space
> > > > > >> consumption
> > > > > >> > > on
> > > > > >> > > > > the
> > > > > >> > > > > > Primary Storage side.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > I see 3 definitions in our current discussion
> regarding
> > > the
> > > > > term
> > > > > >> > > > snapshot
> > > > > >> > > > > > (these are not official terminology but by own
> > > > interpretation
> > > > > of
> > > > > >> > > them):
> > > > > >> > > > > >
> > > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's
> definition
> > > of
> > > > a
> > > > > >> > > > snapshot):
> > > > > >> > > > > > it's a volume snapshot at the storage level, point in
> > time
> > > > of
> > > > > >> your
> > > > > >> > > > data.
> > > > > >> > > > > it
> > > > > >> > > > > > reside on the primary storage. Useful and efficient
> for
> > > > > software
> > > > > >> > side
> > > > > >> > > > > > incident.
> > > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/
> > > cloud
> > > > > >> backup
> > > > > >> > > > aws-S3
> > > > > >> > > > > > style ): Point in time copy of the Virtual Disk that
> > > reside
> > > > > on a
> > > > > >> > > > > different
> > > > > >> > > > > > storage array then the original Volume. Facilitate
> data
> > > > > >> migration
> > > > > >> > > > between
> > > > > >> > > > > > clusters and, in case of primary storage incident,
> > Volume
> > > > > >> snapshots
> > > > > >> > > are
> > > > > >> > > > > not
> > > > > >> > > > > > impacted and can be reuse.
> > > > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data
> that
> > > > also
> > > > > >> > > validate
> > > > > >> > > > > data
> > > > > >> > > > > > integrity, provide a storage efficient archiving
> method
> > > and
> > > > an
> > > > > >> > > > > independent
> > > > > >> > > > > > way to restore your data in case of an major
> > > infrastructure
> > > > > >> > disaster.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Regards,
> > > > > >> > > > > >
> > > > > >> > > > > > PL
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > So, let's see if I currently follow the
> requirements:
> > > > > >> > > > > > >
> > > > > >> > > > > > > * Augment volume snapshots for managed storage to
> > > > > >> conditionally
> > > > > >> > > > export
> > > > > >> > > > > > data
> > > > > >> > > > > > > to NFS. The current process of taking a snapshot on
> > the
> > > > SAN
> > > > > is
> > > > > >> > > fine,
> > > > > >> > > > > but
> > > > > >> > > > > > > we'd like the option to export the data to NFS, as
> > well.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Questions:
> > > > > >> > > > > > >
> > > > > >> > > > > > > Once the data has been exported to NFS, do we keep
> the
> > > SAN
> > > > > >> > snapshot
> > > > > >> > > > or
> > > > > >> > > > > > > delete it?
> > > > > >> > > > > > >
> > > > > >> > > > > > > If we are deleting the SAN snapshot, then why don't
> we
> > > > just
> > > > > >> copy
> > > > > >> > > the
> > > > > >> > > > > VHD
> > > > > >> > > > > > > from primary to secondary the way we do today for
> > > > > non-managed
> > > > > >> > (i.e.
> > > > > >> > > > > > > traditional) storage? Why create a SAN snapshot in
> > that
> > > > > >> scenario?
> > > > > >> > > > > Perhaps
> > > > > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> > > > secondary
> > > > > >> > > storage
> > > > > >> > > > > > > instead of a XenServer host?
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks for the clarification.
> > > > > >> > > > > > >
> > > > > >> > > > > > > By the way, to me a backup is when you copy data
> from
> > > one
> > > > > >> storage
> > > > > >> > > > > system
> > > > > >> > > > > > to
> > > > > >> > > > > > > another (regardless of features, if any, to restore
> > the
> > > > data
> > > > > >> in
> > > > > >> > the
> > > > > >> > > > > > > future). A snapshot is a point-in-time view of the
> > data
> > > > of a
> > > > > >> > volume
> > > > > >> > > > and
> > > > > >> > > > > > > it's stored on the same storage system as the
> volume.
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > > > >> > > pdion@cloudops.com
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > <javascript:;>>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > That's fun to see that discussion happening. I
> 100%
> > > > agree
> > > > > >> with
> > > > > >> > > > Paul's
> > > > > >> > > > > > > > points of view. VolumeSnapshot are not a backup,
> > but I
> > > > do
> > > > > >> > > consider
> > > > > >> > > > > them
> > > > > >> > > > > > > as
> > > > > >> > > > > > > > a safety vest against Primary Storage failure,
> > because
> > > > > >> failure
> > > > > >> > > > append
> > > > > >> > > > > > > :-( .
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > The current proposal around snapshots that reside
> on
> > > the
> > > > > >> > primary
> > > > > >> > > > > > storage
> > > > > >> > > > > > > or
> > > > > >> > > > > > > > snapshots that end in the Secondary Storage  is
> not
> > to
> > > > > >> address
> > > > > >> > > any
> > > > > >> > > > > kind
> > > > > >> > > > > > > of
> > > > > >> > > > > > > > backups requirement because a snapshot is not a
> > > backup,
> > > > > >> event
> > > > > >> > an
> > > > > >> > > > > > > extracted
> > > > > >> > > > > > > > VM snapshot.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > The main idea, and again this is for managed
> > storage;
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side
> snapshot
> > > > > >> capability
> > > > > >> > > for
> > > > > >> > > > > > fast
> > > > > >> > > > > > > > response time that support rollback to previous
> > > > timestamp,
> > > > > >> > create
> > > > > >> > > > new
> > > > > >> > > > > > > > volume and maybe create template.
> > > > > >> > > > > > > >     not required to be a new API if the work is
> > > already
> > > > > >> done, I
> > > > > >> > > > think
> > > > > >> > > > > > > this
> > > > > >> > > > > > > > is a different behaviors than the user expectation
> > of
> > > a
> > > > > >> > > > > > volume-snapshot.
> > > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> > > > behavior
> > > > > >> that
> > > > > >> > > > create
> > > > > >> > > > > > an
> > > > > >> > > > > > > > extraction of a volume into SecondaryStorage which
> > can
> > > > be
> > > > > >> reuse
> > > > > >> > > to
> > > > > >> > > > > > > create a
> > > > > >> > > > > > > > new volume into another Primary Storage. This type
> > of
> > > > > >> snapshot
> > > > > >> > > is a
> > > > > >> > > > > > slow
> > > > > >> > > > > > > > job since yes it would have to copy the full
> volume
> > > size
> > > > > on
> > > > > >> the
> > > > > >> > > > > > Secondary
> > > > > >> > > > > > > > storage.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > PL
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > >> > > > > > <javascript:;>>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > I think I share you view on the 'Ideal world'.
> > > Backup
> > > > > (via
> > > > > >> > > Volume
> > > > > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack.
> > This
> > > is
> > > > > >> > > amplified
> > > > > >> > > > > > > > especially
> > > > > >> > > > > > > > > when you have a object storage as your secondary
> > > > storage
> > > > > >> > > because
> > > > > >> > > > it
> > > > > >> > > > > > > > > requires two copies (one to an NFS staging area
> > and
> > > > from
> > > > > >> > there
> > > > > >> > > to
> > > > > >> > > > > > > object
> > > > > >> > > > > > > > > storage). And not to mention that all these
> copies
> > > are
> > > > > >> > > consuming
> > > > > >> > > > > > > > hypervisor
> > > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> > > bottleneck
> > > > as
> > > > > >> all
> > > > > >> > > the
> > > > > >> > > > > > > Network
> > > > > >> > > > > > > > > and I/O flow through it. So our intention of
> > > proposing
> > > > > the
> > > > > >> > > > "Storage
> > > > > >> > > > > > > > > Snapshots" is to give a better way of achiving
> > > > snapshots
> > > > > >> > while
> > > > > >> > > > > still
> > > > > >> > > > > > > > > keeping the original definition of volume
> > snpashots
> > > > (ie
> > > > > >> > upload
> > > > > >> > > to
> > > > > >> > > > > sec
> > > > > >> > > > > > > > > storage).
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > > > > backups.
> > > > > >> > They
> > > > > >> > > > > don't
> > > > > >> > > > > > > work
> > > > > >> > > > > > > > > form multi-disk LVM volume groups and dynamic
> > > disks. I
> > > > > am
> > > > > >> all
> > > > > >> > > in
> > > > > >> > > > > for
> > > > > >> > > > > > a
> > > > > >> > > > > > > > > better backup solution which handles these use
> > cases
> > > > and
> > > > > >> > > utilizes
> > > > > >> > > > > the
> > > > > >> > > > > > > > > storage's advanced features.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > > > >> > > > > > paul.angus@shapeblue.com <javascript:;>
> <javascript:;>>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > In the beginning... there were CloudStack
> > > snapshots
> > > > > and
> > > > > >> > they
> > > > > >> > > > were
> > > > > >> > > > > > > > > actually
> > > > > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > > > > snapshots.
> > > > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > > > point-in-time
> > > > > >> > > > > hypervisor
> > > > > >> > > > > > > > > > snapshots) and we started referring to the
> > > original
> > > > > >> > snapshots
> > > > > >> > > > as
> > > > > >> > > > > > > volume
> > > > > >> > > > > > > > > > snapshots.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > CloudStack does not offer 'backups', but many
> > > people
> > > > > use
> > > > > >> > > volume
> > > > > >> > > > > > > > snapshots
> > > > > >> > > > > > > > > > as backups. However you can't in-place restore
> > > > volume
> > > > > >> > > snapshots
> > > > > >> > > > > and
> > > > > >> > > > > > > if
> > > > > >> > > > > > > > > you
> > > > > >> > > > > > > > > > have a VM with multiple volumes, the volume
> > > > snapshots
> > > > > >> must
> > > > > >> > be
> > > > > >> > > > > done
> > > > > >> > > > > > in
> > > > > >> > > > > > > > > > series, meaning that the state across all of
> the
> > > > > >> volumes is
> > > > > >> > > > > > unlikely
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > be
> > > > > >> > > > > > > > > > consistent.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > 'Actual Backups' would enable all of the
> restore
> > > > > options
> > > > > >> > > which
> > > > > >> > > > > > users
> > > > > >> > > > > > > > > might
> > > > > >> > > > > > > > > > expect as well options as to where they might
> be
> > > > > >> stored. In
> > > > > >> > > my
> > > > > >> > > > > > ideal
> > > > > >> > > > > > > > > world
> > > > > >> > > > > > > > > > they would also be able to leverage back-end
> > > > hardware
> > > > > >> (such
> > > > > >> > > as
> > > > > >> > > > > > > > Solidfire,
> > > > > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> > > > Commvault
> > > > > >> etc
> > > > > >> > to
> > > > > >> > > > > > > accelerate
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > process.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > >> > > > > > > > > > Paul Angus
> > > > > >> > > > > > > > > > VP Technology ,  ShapeBlue
> > > > > >> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > >> > > > > > > > > >
> > > > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > |
> > > > > >> > > > > m:
> > > > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > >> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> > > > > >> > <javascript:;>
> > > > > >> > > |
> > > > > >> > > > > t: @cloudyangus*
> > > > > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > >> > > > > <ja...@cloudyangus>
> > > > > >> > > > > > |  w:
> > > > > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com
> >
> > > > > >> > > > > > > > > > a:  53 Chandos Place, Covent Garden London
> WC2N
> > > 4HS
> > > > UK
> > > > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in
> > > England
> > > > &
> > > > > >> > Wales.
> > > > > >> > > > > > > ShapeBlue
> > > > > >> > > > > > > > > > Services India LLP is a company incorporated
> in
> > > > India
> > > > > >> and
> > > > > >> > is
> > > > > >> > > > > > operated
> > > > > >> > > > > > > > > under
> > > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > > > >> Consultoria
> > > > > >> > > Ltda
> > > > > >> > > > > is
> > > > > >> > > > > > a
> > > > > >> > > > > > > > > > company incorporated in Brasil and is operated
> > > under
> > > > > >> > license
> > > > > >> > > > from
> > > > > >> > > > > > > Shape
> > > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > > registered
> > > > > >> by
> > > > > >> > The
> > > > > >> > > > > > > Republic
> > > > > >> > > > > > > > of
> > > > > >> > > > > > > > > > South Africa and is traded under license from
> > > Shape
> > > > > Blue
> > > > > >> > Ltd.
> > > > > >> > > > > > > ShapeBlue
> > > > > >> > > > > > > > > is
> > > > > >> > > > > > > > > > a registered trademark.
> > > > > >> > > > > > > > > > This email and any attachments to it may be
> > > > > confidential
> > > > > >> > and
> > > > > >> > > > are
> > > > > >> > > > > > > > intended
> > > > > >> > > > > > > > > > solely for the use of the individual to whom
> it
> > is
> > > > > >> > addressed.
> > > > > >> > > > Any
> > > > > >> > > > > > > views
> > > > > >> > > > > > > > > or
> > > > > >> > > > > > > > > > opinions expressed are solely those of the
> > author
> > > > and
> > > > > do
> > > > > >> > not
> > > > > >> > > > > > > > necessarily
> > > > > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > > > > companies.
> > > > > >> If
> > > > > >> > > you
> > > > > >> > > > > are
> > > > > >> > > > > > > not
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > intended recipient of this email, you must
> > neither
> > > > > take
> > > > > >> any
> > > > > >> > > > > action
> > > > > >> > > > > > > > based
> > > > > >> > > > > > > > > > upon its contents, nor copy or show it to
> > anyone.
> > > > > Please
> > > > > >> > > > contact
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > sender
> > > > > >> > > > > > > > > > if you believe you have received this email in
> > > > error.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > -----Original Message-----
> > > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > > syed1.mushtaq@gmail.com
> > > > > >> > > > > <javascript:;> <javascript:;>]
> > > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > > >> > <javascript:;>
> > > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > > Snapshots
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Paul,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > When you say actual backups, how would it be
> > > > different
> > > > > >> from
> > > > > >> > > the
> > > > > >> > > > > > > Volume
> > > > > >> > > > > > > > > > Snapshots that exist currently. My
> understanding
> > > is
> > > > > that
> > > > > >> > > > Backups
> > > > > >> > > > > > end
> > > > > >> > > > > > > up
> > > > > >> > > > > > > > > in
> > > > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > > > point-in-time
> > > > > >> > state
> > > > > >> > > of
> > > > > >> > > > > > your
> > > > > >> > > > > > > > > volume
> > > > > >> > > > > > > > > > which can be restored back correct?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > -Syed
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> > <javascript:;>>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > Hi Syed,
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > As I understand it, the SolidFire plugin
> will
> > > > export
> > > > > >> the
> > > > > >> > > > > snapshot
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > > secondary storage if the user requests a
> > > template
> > > > > from
> > > > > >> > the
> > > > > >> > > > > > snapshot
> > > > > >> > > > > > > > or
> > > > > >> > > > > > > > > > > wants to download the snapshot from the
> cloud.
> > > > This
> > > > > >> is a
> > > > > >> > > > good,
> > > > > >> > > > > > > > > > > pragmatic approach and yes Mike the
> SolidFire
> > > > > storage
> > > > > >> is
> > > > > >> > > > super
> > > > > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays
> > take
> > > up
> > > > > >> next
> > > > > >> > to
> > > > > >> > > no
> > > > > >> > > > > > > space.
> > > > > >> > > > > > > > > > > BUT I think that we are talking about a more
> > > > general
> > > > > >> > > purpose
> > > > > >> > > > > API,
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > > > > other storage systems may not be as awesome
> as
> > > > > Mike's.
> > > > > >> > > That's
> > > > > >> > > > > my
> > > > > >> > > > > > > > > > > concern. Also, the time to transfer for say
> > 1TB
> > > to
> > > > > >> move
> > > > > >> > > from
> > > > > >> > > > > > > primary
> > > > > >> > > > > > > > > > > to sec storage and then create a VM template
> > out
> > > > of
> > > > > it
> > > > > >> > may
> > > > > >> > > be
> > > > > >> > > > > too
> > > > > >> > > > > > > > long
> > > > > >> > > > > > > > > > for users.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > > > > snapshot
> > > > > >> for
> > > > > >> > > > > backup,
> > > > > >> > > > > > > > it's
> > > > > >> > > > > > > > > > > just that users want to do backups and a
> > volume
> > > > > >> snapshot
> > > > > >> > is
> > > > > >> > > > the
> > > > > >> > > > > > > only
> > > > > >> > > > > > > > > > > type of snapshot that copies the disk
> > elsewhere
> > > > and
> > > > > >> can
> > > > > >> > be
> > > > > >> > > > > > > scheduled.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> > > > actual
> > > > > >> > backups
> > > > > >> > > > > > > (through
> > > > > >> > > > > > > > > > > recognised backup providers) and the user
> > > > > requirements
> > > > > >> > > around
> > > > > >> > > > > > them
> > > > > >> > > > > > > > > > > (particularly restoration use cases) as a
> > > separate
> > > > > >> thread
> > > > > >> > > of
> > > > > >> > > > > > work.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > [image: ShapeBlue] <
> http://www.shapeblue.com>
> > > > Paul
> > > > > >> Angus
> > > > > >> > > VP
> > > > > >> > > > > > > > Technology
> > > > > >> > > > > > > > > > > , ShapeBlue
> > > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > >> > > > > > > > > > >
> > > > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > >> > > > |
> > > > > >> > > > > m:
> > > > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > > > >> > > <javascript:;> |
> > > > > >> > > > > t: @cloudyangus*
> > > > > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > >> <javascript:;>
> > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > >> > > > > > > > > > > *www.shapeblue.com* <
> http://www.shapeblue.com
> > >
> > > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> WC2N
> > > 4HS
> > > > > UK
> > > > > >> > Shape
> > > > > >> > > > > Blue
> > > > > >> > > > > > > Ltd
> > > > > >> > > > > > > > > > > is a company incorporated in England &
> Wales.
> > > > > >> ShapeBlue
> > > > > >> > > > > Services
> > > > > >> > > > > > > > India
> > > > > >> > > > > > > > > > > LLP is a company incorporated in India and
> is
> > > > > operated
> > > > > >> > > under
> > > > > >> > > > > > > license
> > > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> > > Consultoria
> > > > > >> Ltda
> > > > > >> > is
> > > > > >> > > a
> > > > > >> > > > > > > company
> > > > > >> > > > > > > > > > > incorporated in Brasil and is operated under
> > > > license
> > > > > >> from
> > > > > >> > > > Shape
> > > > > >> > > > > > > Blue
> > > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> > > registered
> > > > by
> > > > > >> The
> > > > > >> > > > > Republic
> > > > > >> > > > > > > of
> > > > > >> > > > > > > > > > > South Africa and is traded under license
> from
> > > > Shape
> > > > > >> Blue
> > > > > >> > > Ltd.
> > > > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > > >> > > > > > > > > > > This email and any attachments to it may be
> > > > > >> confidential
> > > > > >> > > and
> > > > > >> > > > > are
> > > > > >> > > > > > > > > > > intended solely for the use of the
> individual
> > to
> > > > > whom
> > > > > >> it
> > > > > >> > is
> > > > > >> > > > > > > > addressed.
> > > > > >> > > > > > > > > > > Any views or opinions expressed are solely
> > those
> > > > of
> > > > > >> the
> > > > > >> > > > author
> > > > > >> > > > > > and
> > > > > >> > > > > > > do
> > > > > >> > > > > > > > > > > not necessarily represent those of Shape
> Blue
> > > Ltd
> > > > or
> > > > > >> > > related
> > > > > >> > > > > > > > > > > companies. If you are not the intended
> > recipient
> > > > of
> > > > > >> this
> > > > > >> > > > email,
> > > > > >> > > > > > you
> > > > > >> > > > > > > > > > > must neither take any action based upon its
> > > > > contents,
> > > > > >> nor
> > > > > >> > > > copy
> > > > > >> > > > > or
> > > > > >> > > > > > > > show
> > > > > >> > > > > > > > > > > it to anyone. Please contact the sender if
> you
> > > > > believe
> > > > > >> > you
> > > > > >> > > > have
> > > > > >> > > > > > > > > received
> > > > > >> > > > > > > > > > this email in error.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > -----Original Message-----
> > > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > > syed1.mushtaq@gmail.com
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > <javascript:;>]
> > > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org
> <javascript:;>
> > > > > >> > > <javascript:;>
> > > > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > > > Snapshots
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > I think the terminology confusion comes from
> > AWS
> > > > > where
> > > > > >> > they
> > > > > >> > > > do
> > > > > >> > > > > > EBS
> > > > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack
> sort
> > of
> > > > > >> followed
> > > > > >> > > > that.
> > > > > >> > > > > > And
> > > > > >> > > > > > > > as
> > > > > >> > > > > > > > > > > an end user who is oblivious to the
> internals
> > of
> > > > my
> > > > > >> > > provider,
> > > > > >> > > > > my
> > > > > >> > > > > > > > > > > expectation would be something similar to
> what
> > > AWS
> > > > > as
> > > > > >> > that
> > > > > >> > > is
> > > > > >> > > > > my
> > > > > >> > > > > > > > > > > biggest reference point.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > To your point Mike, I agree that a Primary
> > > Storage
> > > > > >> > failure
> > > > > >> > > on
> > > > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> > > motivations
> > > > > >> for us
> > > > > >> > > to
> > > > > >> > > > > push
> > > > > >> > > > > > > > data
> > > > > >> > > > > > > > > > > to secondary storage. Primary storage
> (atleast
> > > for
> > > > > us)
> > > > > >> > > costs
> > > > > >> > > > > > > around 3
> > > > > >> > > > > > > > > > > times as much as secondary storage and
> > snapshots
> > > > on
> > > > > >> > primary
> > > > > >> > > > > > storage
> > > > > >> > > > > > > > > > > are rarely used (especially for some of our
> > > > > customers
> > > > > >> who
> > > > > >> > > do
> > > > > >> > > > > > daily
> > > > > >> > > > > > > > > > backups).
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike
> > Tutkowski
> > > <
> > > > > >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > > >> > > <javascript:;>>
> > > > > >> > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Some of the weirdness is around
> terminology.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > For most systems I've worked on, a
> snapshot
> > > and
> > > > a
> > > > > >> > backup
> > > > > >> > > > are
> > > > > >> > > > > > two
> > > > > >> > > > > > > > > > > > completely different things (but
> CloudStack
> > > has
> > > > > >> > > > traditionally
> > > > > >> > > > > > > used
> > > > > >> > > > > > > > > > > > the term "volume snapshot" to mean
> backup).
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and
> > say,
> > > > > >> though,
> > > > > >> > > that
> > > > > >> > > > > if
> > > > > >> > > > > > > your
> > > > > >> > > > > > > > > > > > primary storage is running on SolidFire
> that
> > > it
> > > > is
> > > > > >> > > unlikely
> > > > > >> > > > > > > you'll
> > > > > >> > > > > > > > > > > > encounter an issue where your primary
> > storage
> > > > goes
> > > > > >> > > offline
> > > > > >> > > > > (and
> > > > > >> > > > > > > > > > > > you'll even maintain your performance
> > > guarantees
> > > > > >> during
> > > > > >> > > > > failure
> > > > > >> > > > > > > > > > > > scenarios and upgrades, as well). That
> being
> > > the
> > > > > >> case,
> > > > > >> > it
> > > > > >> > > > is
> > > > > >> > > > > > less
> > > > > >> > > > > > > > > > > > useful to require a backup to Swift (but
> > it's
> > > > > >> perfectly
> > > > > >> > > OK
> > > > > >> > > > if
> > > > > >> > > > > > > > that's
> > > > > >> > > > > > > > > > > > what we want to do
> > > > > >> > > > > > > > > > > here).
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed
> Mushtaq
> > > > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > > > > >> > <javascript:;>>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Paul,
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > I believe with the current
> implementation
> > of
> > > > > >> > Snapshots
> > > > > >> > > on
> > > > > >> > > > > > > managed
> > > > > >> > > > > > > > > > > > > storage
> > > > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never
> > exported
> > > > to
> > > > > >> the
> > > > > >> > > > > secondary
> > > > > >> > > > > > > > > > storage.
> > > > > >> > > > > > > > > > > > > While this solves the problem of having
> > > > > snapshots
> > > > > >> > > taking
> > > > > >> > > > > > > forever
> > > > > >> > > > > > > > > > > > > to get to sec storage, this leaves us
> > with a
> > > > > >> > > > > > > > > > > > huge
> > > > > >> > > > > > > > > > > > > liability if our primary storage goes
> > down.
> > > We
> > > > > see
> > > > > >> > > > > snapshots
> > > > > >> > > > > > as
> > > > > >> > > > > > > > > > > > > our recovery path because we store them
> in
> > > > Swift
> > > > > >> > which
> > > > > >> > > is
> > > > > >> > > > > > > > reliable
> > > > > >> > > > > > > > > > > > > and resilient to failures.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > With Storage snpashots our goal is to
> have
> > > > > Volume
> > > > > >> > > > snapshots
> > > > > >> > > > > > > > always
> > > > > >> > > > > > > > > > > > > backed up to secondary storage and
> Storage
> > > > > >> Snapshots
> > > > > >> > > stay
> > > > > >> > > > > on
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > > primary
> > > > > >> > > > > > > > > > > > storage.
> > > > > >> > > > > > > > > > > > > A provider could potentially mix both
> > these
> > > > and
> > > > > >> solve
> > > > > >> > > the
> > > > > >> > > > > > > problem
> > > > > >> > > > > > > > > > > > > that you mentioned where you want to
> meet
> > > > user's
> > > > > >> > > > > expectation
> > > > > >> > > > > > > of a
> > > > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage)
> while
> > > > having
> > > > > >> an
> > > > > >> > > > ability
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the
> > > device)
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > > > > -Syed
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul
> Angus
> > > > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com
> <javascript:;>
> > > > > >> > > <javascript:;>>
> > > > > >> > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > HI guys,
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Could someone point me to the Jira bug
> > of
> > > FS
> > > > > for
> > > > > >> > the
> > > > > >> > > > > > > > > > > > > > SAN-snapshot
> > > > > >> > > > > > > > > > > > feature
> > > > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > From my discussions with users and
> > > operators
> > > > > >> around
> > > > > >> > > > > > snapshots
> > > > > >> > > > > > > > > > > > > > I'd make
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > following observations:
> > > > > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups
> > (both
> > > > > >> long-term
> > > > > >> > > and
> > > > > >> > > > > > short
> > > > > >> > > > > > > > > > > > > > term)
> > > > > >> > > > > > > > > > > > with
> > > > > >> > > > > > > > > > > > > > the expectation that they can use them
> > for
> > > > > >> recovery
> > > > > >> > > if
> > > > > >> > > > > > > > required.
> > > > > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > > > > something
> > > > > >> > has
> > > > > >> > > > gone
> > > > > >> > > > > > > wrong
> > > > > >> > > > > > > > > > > > > > with primary storage.
> > > > > >> > > > > > > > > > > > > > c. users sometimes want to be able to
> > > export
> > > > > >> > > snapshots
> > > > > >> > > > as
> > > > > >> > > > > > > well
> > > > > >> > > > > > > > > > > > > > as
> > > > > >> > > > > > > > > > > > create
> > > > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > > > >> > > > > > > > > > > > > > d. snapshots are a currently a massive
> > > pain
> > > > > for
> > > > > >> > > > > operators,
> > > > > >> > > > > > I
> > > > > >> > > > > > > > > > > > > > know at
> > > > > >> > > > > > > > > > > > > least
> > > > > >> > > > > > > > > > > > > > one public cloud who have snapshots
> > which
> > > > > take 2
> > > > > >> > days
> > > > > >> > > > to
> > > > > >> > > > > > > > > complete.
> > > > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be
> used
> > > for
> > > > > >> > multiple
> > > > > >> > > > LVM
> > > > > >> > > > > > > > disks.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > I think the process Mike has used in
> the
> > > > > >> SolidFire
> > > > > >> > > > plugin
> > > > > >> > > > > > > (only
> > > > > >> > > > > > > > > > > > > > moving
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > disk image to secondary storage when
> you
> > > > > >> absolutely
> > > > > >> > > > have
> > > > > >> > > > > > to)
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > > > > > > > a very
> > > > > >> > > > > > > > > > > > > good
> > > > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> > > > problems
> > > > > >> an
> > > > > >> > > > > operator
> > > > > >> > > > > > > > might
> > > > > >> > > > > > > > > > > > > experience
> > > > > >> > > > > > > > > > > > > > if they have an issue with a given
> > primary
> > > > > >> storage
> > > > > >> > > pool
> > > > > >> > > > > in
> > > > > >> > > > > > a
> > > > > >> > > > > > > > > > cluster.
> > > > > >> > > > > > > > > > > > (I
> > > > > >> > > > > > > > > > > > > > know that that is REALLY unlikely in
> the
> > > > > >> SolidFire
> > > > > >> > > case
> > > > > >> > > > > > Mike
> > > > > >> > > > > > > :)
> > > > > >> > > > > > > > > > > > > > ) And
> > > > > >> > > > > > > > > > > > if
> > > > > >> > > > > > > > > > > > > > the transfer from primary to secondary
> > is
> > > > > slow,
> > > > > >> the
> > > > > >> > > > time
> > > > > >> > > > > to
> > > > > >> > > > > > > > > > > > > > being able
> > > > > >> > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > create a template or export the volume
> > > will
> > > > be
> > > > > >> > slow.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > So for me the issue is around making
> > sure
> > > > that
> > > > > >> the
> > > > > >> > > end
> > > > > >> > > > > > users
> > > > > >> > > > > > > > > > > > expectations
> > > > > >> > > > > > > > > > > > > > are met (while improving the
> > > > speed/efficiency
> > > > > of
> > > > > >> > the
> > > > > >> > > > back
> > > > > >> > > > > > > end)
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > > > http://www.shapeblue.com>
> > > > > >> Paul
> > > > > >> > > > Angus
> > > > > >> > > > > > VP
> > > > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> > > 0540*
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > >
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > >> > > > > > > |
> > > > > >> > > > > > > > m:
> > > > > >> > > > > > > > > > > > > > *+44 7711 418784*
> <+44%207711%20418784>
> > > > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> > > <javascript:;>
> > > > > >> > > > > <javascript:;> | t:
> > > > > >> > > > > > @cloudyangus*
> > > > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com
> > <javascript:;>
> > > > > >> > > > <javascript:;>
> > > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > > > http://www.shapeblue.com
> > > > > >
> > > > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> > London
> > > > WC2N
> > > > > >> 4HS
> > > > > >> > UK
> > > > > >> > > > > Shape
> > > > > >> > > > > > > > Blue
> > > > > >> > > > > > > > > > > > > > Ltd is a company incorporated in
> > England &
> > > > > >> Wales.
> > > > > >> > > > > ShapeBlue
> > > > > >> > > > > > > > > > > > > > Services India LLP is a company
> > > incorporated
> > > > > in
> > > > > >> > India
> > > > > >> > > > and
> > > > > >> > > > > > is
> > > > > >> > > > > > > > > > > > > > operated
> > > > > >> > > > > > > > > > > > > under
> > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape
> Blue
> > > > Brasil
> > > > > >> > > > > Consultoria
> > > > > >> > > > > > > Ltda
> > > > > >> > > > > > > > > > > > > > is a company incorporated in Brasil
> and
> > is
> > > > > >> operated
> > > > > >> > > > under
> > > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue
> > SA
> > > > Pty
> > > > > >> Ltd
> > > > > >> > is
> > > > > >> > > a
> > > > > >> > > > > > > company
> > > > > >> > > > > > > > > > > > > > registered by The Republic
> > > > > >> > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > > > South Africa and is traded under
> license
> > > > from
> > > > > >> Shape
> > > > > >> > > > Blue
> > > > > >> > > > > > Ltd.
> > > > > >> > > > > > > > > > > > > > ShapeBlue
> > > > > >> > > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > a registered trademark.
> > > > > >> > > > > > > > > > > > > > This email and any attachments to it
> may
> > > be
> > > > > >> > > > confidential
> > > > > >> > > > > > and
> > > > > >> > > > > > > > are
> > > > > >> > > > > > > > > > > > intended
> > > > > >> > > > > > > > > > > > > > solely for the use of the individual
> to
> > > whom
> > > > > it
> > > > > >> is
> > > > > >> > > > > > addressed.
> > > > > >> > > > > > > > > > > > > > Any views
> > > > > >> > > > > > > > > > > > > or
> > > > > >> > > > > > > > > > > > > > opinions expressed are solely those of
> > the
> > > > > >> author
> > > > > >> > and
> > > > > >> > > > do
> > > > > >> > > > > > not
> > > > > >> > > > > > > > > > > > necessarily
> > > > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or
> > > related
> > > > > >> > > companies.
> > > > > >> > > > > If
> > > > > >> > > > > > > you
> > > > > >> > > > > > > > > > > > > > are not
> > > > > >> > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > intended recipient of this email, you
> > must
> > > > > >> neither
> > > > > >> > > take
> > > > > >> > > > > any
> > > > > >> > > > > > > > > > > > > > action
> > > > > >> > > > > > > > > > > > based
> > > > > >> > > > > > > > > > > > > > upon its contents, nor copy or show it
> > to
> > > > > >> anyone.
> > > > > >> > > > Please
> > > > > >> > > > > > > > contact
> > > > > >> > > > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > sender
> > > > > >> > > > > > > > > > > > > > if you believe you have received this
> > > email
> > > > in
> > > > > >> > error.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > -----Original Message-----
> > > > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > > > >> pdion@cloudops.com
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > <javascript:;>]
> > > > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56
> PM
> > > > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature]
> > > Storage
> > > > > >> > Snapshots
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Hi Mike,
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > > > >> StorageSnapshot
> > > > > >> > > for
> > > > > >> > > > > > > managed
> > > > > >> > > > > > > > > > > > > > storage
> > > > > >> > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > > > > expected,
> > > > > >> > > > behavior
> > > > > >> > > > > > is
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> > > > Storage.
> > > > > >> So a
> > > > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > > > >> > > > > > > > > > > > > for
> > > > > >> > > > > > > > > > > > > > snapshot that remain on the managed
> > > storage
> > > > > >> > > appliance.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Quickly looking at the API doc and I
> > don't
> > > > > see a
> > > > > >> > > strong
> > > > > >> > > > > > > > > > > > > > requirement for volume snapshots to be
> > > moved
> > > > > >> into
> > > > > >> > > > > secondary
> > > > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API
> > is
> > > > not
> > > > > >> > useful,
> > > > > >> > > > but
> > > > > >> > > > > > > both
> > > > > >> > > > > > > > > > > > > > use cases are required. A snapshot
> that
> > > > remain
> > > > > >> on
> > > > > >> > the
> > > > > >> > > > > > managed
> > > > > >> > > > > > > > > > > > > > storage, and another type of
> > > > > >> > > > > > > > > > > > snapshot
> > > > > >> > > > > > > > > > > > > > that end up into the secondary
> storage.
> > > > Since
> > > > > >> > you've
> > > > > >> > > > > done a
> > > > > >> > > > > > > lot
> > > > > >> > > > > > > > > > > > > > of
> > > > > >> > > > > > > > > > > > work,
> > > > > >> > > > > > > > > > > > > > might easier to just add a parameter
> to
> > > the
> > > > > >> current
> > > > > >> > > > > > snapshot
> > > > > >> > > > > > > > API
> > > > > >> > > > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > > > trigger an extraction of the storage
> > > > snapshot
> > > > > >> into
> > > > > >> > > the
> > > > > >> > > > > > > > secondary
> > > > > >> > > > > > > > > > > > storage?
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > PL
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > > > > Tutkowski <
> > > > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > I think that all sounds reasonable
> > then
> > > -
> > > > > >> thanks!
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > > > > Mushtaq <
> > > > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > >> <javascript:;>>
> > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >> You are correct Mike in terms of
> the
> > > > > >> > requirements.
> > > > > >> > > > One
> > > > > >> > > > > > of
> > > > > >> > > > > > > > our
> > > > > >> > > > > > > > > > > > earlier
> > > > > >> > > > > > > > > > > > > > >> iterations on this was to have an
> > > > argument
> > > > > to
> > > > > >> > the
> > > > > >> > > > > create
> > > > > >> > > > > > > > > > > > > > >> snapshot
> > > > > >> > > > > > > > > > > > API
> > > > > >> > > > > > > > > > > > > > >> which decides whether to backup the
> > > > volume
> > > > > to
> > > > > >> > sec
> > > > > >> > > > > > storage
> > > > > >> > > > > > > > but
> > > > > >> > > > > > > > > > > > > > >> we realized it would make
> management
> > of
> > > > > >> > snapshots
> > > > > >> > > > > quite
> > > > > >> > > > > > > > messy
> > > > > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > > > >> > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> > > > Tutkowski
> > > > > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > > > > <javascript:;>
> > > > > >> > > > > <javascript:;>>
> > > > > >> > > > > > > > > > > > > > >> wrote:
> > > > > >> > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >>> Hi,
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> Just to make sure I understand all
> > the
> > > > > >> > > requirements
> > > > > >> > > > > > here:
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 1) This relates only to managed
> > > storage
> > > > > (1:1
> > > > > >> > > > mapping
> > > > > >> > > > > > > > between
> > > > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> > > > volume).
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > > > (introduced
> > > > > >> in
> > > > > >> > > 4.6)
> > > > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> > > snapshot
> > > > on
> > > > > >> the
> > > > > >> > > SAN,
> > > > > >> > > > > and
> > > > > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > > > > >> > > > > > > > > > > > > > >>> something) to not only take the
> SAN
> > > > > >> snapshot,
> > > > > >> > but
> > > > > >> > > > to
> > > > > >> > > > > > copy
> > > > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer
> only)
> > to
> > > > > NFS.
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always
> taken.
> > > > It's
> > > > > >> the
> > > > > >> > > > backup
> > > > > >> > > > > to
> > > > > >> > > > > > > NFS
> > > > > >> > > > > > > > > > > > > > >>> that is optional.
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 4) Templates can be created from
> the
> > > > > >> snapshot
> > > > > >> > > > that's
> > > > > >> > > > > on
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be
> created
> > > > from
> > > > > >> the
> > > > > >> > > > > snapshot
> > > > > >> > > > > > > > > > > > > > >>> that's on
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>> SAN (already works as long as the
> > new
> > > > > >> > CloudStack
> > > > > >> > > > > volume
> > > > > >> > > > > > > > ends
> > > > > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > > > > snapshot
> > > > > >> API
> > > > > >> > > > then
> > > > > >> > > > > or
> > > > > >> > > > > > > > > > > > > > >>> would that just be the standard
> > volume
> > > > > >> snapshot
> > > > > >> > > > > without
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>> backup to
> > > > > >> > > > > > > > > > > NFS?
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> Thanks!
> > > > > >> > > > > > > > > > > > > > >>> Mike
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM,
> Syed
> > > > > Mushtaq
> > > > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> > > <javascript:;>
> > > > > >> > > > > <javascript:;>>
> > > > > >> > > > > > > > > > > > > > >>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > > > functionalities
> > > > > >> > > > > (snapshot
> > > > > >> > > > > > on
> > > > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because
> Ideally,
> > we
> > > > > would
> > > > > >> > like
> > > > > >> > > > to
> > > > > >> > > > > > have
> > > > > >> > > > > > > > > both.
> > > > > >> > > > > > > > > > > > > > >>>> For example, some of our
> customers
> > > want
> > > > > to
> > > > > >> > > > implement
> > > > > >> > > > > > > their
> > > > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> > > encryption
> > > > > to
> > > > > >> > their
> > > > > >> > > > > > backups
> > > > > >> > > > > > > > > > > > > > >>>> which is a perfect use case for
> > > Storage
> > > > > >> > Snapshot
> > > > > >> > > > > while
> > > > > >> > > > > > > our
> > > > > >> > > > > > > > > > > > > > >>>> other customers will still keep
> > using
> > > > the
> > > > > >> > > standard
> > > > > >> > > > > > > volume
> > > > > >> > > > > > > > > > > snapshot.
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>> To keep things backward
> compatible,
> > > we
> > > > > can
> > > > > >> > add a
> > > > > >> > > > > > setting
> > > > > >> > > > > > > > > > > > > > >>>> which
> > > > > >> > > > > > > > > > > > says
> > > > > >> > > > > > > > > > > > > > >>>> to not upload on secondary
> storage,
> > > > > >> because,
> > > > > >> > > after
> > > > > >> > > > > > all,
> > > > > >> > > > > > > > you
> > > > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first
> > when
> > > > > doing
> > > > > >> a
> > > > > >> > > > Volume
> > > > > >> > > > > > > > > Snapshot.
> > > > > >> > > > > > > > > > > > > > >>>> You could stop the process there
> > and
> > > > not
> > > > > do
> > > > > >> > the
> > > > > >> > > > > > upload.
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>> What do you think about this
> > > approach?
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > > > >> > > > > > > > > > > > > > >>>> -Syed
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM,
> > Mike
> > > > > >> > Tutkowski <
> > > > > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out
> > > load
> > > > > >> here,
> > > > > >> > but
> > > > > >> > > > if
> > > > > >> > > > > a
> > > > > >> > > > > > > > given
> > > > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't
> actually
> > > need
> > > > > to
> > > > > >> > > provide
> > > > > >> > > > > > both
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > ability
> > > > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and
> export
> > it
> > > > to
> > > > > >> NFS
> > > > > >> > (if
> > > > > >> > > > > just
> > > > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK),
> then
> > > we
> > > > > >> might
> > > > > >> > be
> > > > > >> > > > able
> > > > > >> > > > > > to
> > > > > >> > > > > > > > get
> > > > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > > > >> > > > > > > > > > > > API
> > > > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> > > > custom
> > > > > >> > > snapshot
> > > > > >> > > > > > > strategy
> > > > > >> > > > > > > > > > > > > > >>>>> and
> > > > > >> > > > > > > > > > > > data
> > > > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the
> case
> > > > where
> > > > > >> the
> > > > > >> > > > > > CloudStack
> > > > > >> > > > > > > > > > > > > > >>>>> cloud
> > > > > >> > > > > > > > > > > > does
> > > > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > > > >> exported-to-NFS
> > > > > >> > > > > backup.
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> In other words, the "default"
> > > behavior
> > > > > >> would
> > > > > >> > be
> > > > > >> > > > to
> > > > > >> > > > > > use
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data
> motion
> > > > > strategy
> > > > > >> > that
> > > > > >> > > > we
> > > > > >> > > > > > > > already
> > > > > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a
> > SAN
> > > > > >> > snapshot).
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud,
> however,
> > > > wants
> > > > > >> to
> > > > > >> > > take
> > > > > >> > > > a
> > > > > >> > > > > > SAN
> > > > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data
> > exported
> > > to
> > > > > >> NFS,
> > > > > >> > > then
> > > > > >> > > > we
> > > > > >> > > > > > > could
> > > > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing
> config
> > > > file
> > > > > to
> > > > > >> > make
> > > > > >> > > > use
> > > > > >> > > > > > of
> > > > > >> > > > > > > a
> > > > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data
> > > motion
> > > > > >> > strategy
> > > > > >> > > > that
> > > > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > > > >> > > > > > > > > > > > > > activities.
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is
> > still
> > > > the
> > > > > >> > default
> > > > > >> > > > for
> > > > > >> > > > > > > > users,
> > > > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change
> > > this
> > > > > >> > behavior
> > > > > >> > > > via
> > > > > >> > > > > > > > > > > configuration.
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM,
> > > Mike
> > > > > >> > > Tutkowski <
> > > > > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to
> > > come
> > > > up
> > > > > >> > with a
> > > > > >> > > > > > viable
> > > > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable
> > way
> > > > for
> > > > > >> them
> > > > > >> > > to
> > > > > >> > > > > move
> > > > > >> > > > > > > > from
> > > > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and
> > > some
> > > > > >> obvious
> > > > > >> > > way
> > > > > >> > > > > > that
> > > > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > > > >> > > > > > > > > > > > to
> > > > > >> > > > > > > > > > > > > > do this).
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45
> AM,
> > > Syed
> > > > > >> > Mushtaq
> > > > > >> > > <
> > > > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > > > <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> > > > upgrade
> > > > > >> path
> > > > > >> > > > > however,
> > > > > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6
> and
> > > are
> > > > > on a
> > > > > >> > > > managed
> > > > > >> > > > > > > > storage
> > > > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > > > > >> functionality
> > > > > >> > > > with
> > > > > >> > > > > > > Volume
> > > > > >> > > > > > > > > > > Snapshots.
> > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43
> PM,
> > > Syed
> > > > > >> > Mushtaq
> > > > > >> > > <
> > > > > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > > > > <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly,
> > > > > currently
> > > > > >> > > > taking a
> > > > > >> > > > > > > > Volume
> > > > > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a
> > > managed
> > > > > >> storage
> > > > > >> > > > keeps
> > > > > >> > > > > > it
> > > > > >> > > > > > > on
> > > > > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part
> of
> > > > this
> > > > > >> > > feature,
> > > > > >> > > > we
> > > > > >> > > > > > can
> > > > > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > > > > >> > > > > > > > > > > > that
> > > > > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed
> > > storage
> > > > > are
> > > > > >> > > > uploaded
> > > > > >> > > > > > to
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would
> > > make
> > > > > the
> > > > > >> > > Volume
> > > > > >> > > > > > > Snapshot
> > > > > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same
> > > regardless
> > > > of
> > > > > >> the
> > > > > >> > > > > storage
> > > > > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > > > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for
> utilizing
> > > the
> > > > > >> > > efficient
> > > > > >> > > > > > > backend
> > > > > >> > > > > > > > > > > > > > >>>>>>>> storage
> > > > > >> > > > > > > > > > > > > > capabilities, we can use the new
> storage
> > > > > >> snapshots
> > > > > >> > > API.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36
> PM,
> > > > Mike
> > > > > >> > > > Tutkowski <
> > > > > >> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> > > > > >> > <javascript:;>
> > > > > >> > > > > <javascript:;>>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need
> > to
> > > > > have a
> > > > > >> > plan
> > > > > >> > > > to
> > > > > >> > > > > > deal
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we
> already
> > > > have a
> > > > > >> > > feature
> > > > > >> > > > > (in
> > > > > >> > > > > > > 4.6
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> and
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to
> use
> > > the
> > > > > >> > existing
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to
> > create a
> > > > > >> volume
> > > > > >> > > > > snapshot
> > > > > >> > > > > > > (for
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a
> > > backend
> > > > > SAN
> > > > > >> > > > (using a
> > > > > >> > > > > > > > custom
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a
> custom
> > > > data
> > > > > >> > motion
> > > > > >> > > > > > > strategy).
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in,
> then
> > > how
> > > > > >> should
> > > > > >> > > the
> > > > > >> > > > > > > original
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in
> 4.6
> > > and
> > > > > >> later)
> > > > > >> > > be
> > > > > >> > > > > > > changed?
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > > > > >> > > > > > > > > > > > is
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support
> > > > customers
> > > > > >> who
> > > > > >> > > are
> > > > > >> > > > > > > already
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> using
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API
> > to
> > > > take
> > > > > >> > > > snapshots
> > > > > >> > > > > > on a
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > > > > >> > > > > > > > > > > > > SAN?
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27
> > AM,
> > > > > Will
> > > > > >> > > > Stevens <
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > > > > <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create
> a
> > > > > Template
> > > > > >> > > from a
> > > > > >> > > > > > > > > > > > StorageSnapshot?
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template
> be
> > > > stored
> > > > > >> in
> > > > > >> > the
> > > > > >> > > > > > > secondary
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal
> templates
> > > or
> > > > > will
> > > > > >> > that
> > > > > >> > > > be
> > > > > >> > > > > > > > handled
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > >> > > > > > > > > > > > > > vendor side?
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud
> > Solutions
> > > > > >> Experts
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal
> *|*
> > > > Quebec
> > > > > >> *|*
> > > > > >> > > H3J
> > > > > >> > > > > 1S6
> > > > > >> > > > > > w
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw
> > @CloudOps_
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22
> > PM,
> > > > > Syed
> > > > > >> > > > Mushtaq <
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at
> 1:19
> > > PM,
> > > > > Will
> > > > > >> > > > Stevens
> > > > > >> > > > > <
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>> wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the
> > > Design
> > > > > >> Spec in
> > > > > >> > > the
> > > > > >> > > > > > Jira
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not
> > > clear
> > > > > in
> > > > > >> the
> > > > > >> > > > > > 'mention'
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows
> > as a
> > > > > page
> > > > > >> > 'you
> > > > > >> > > do
> > > > > >> > > > > not
> > > > > >> > > > > > > > have
> > > > > >> > > > > > > > > > > permission to'.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud
> > > Solutions
> > > > > >> Experts
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal
> > *|*
> > > > > Quebec
> > > > > >> > *|*
> > > > > >> > > > H3J
> > > > > >> > > > > > 1S6
> > > > > >> > > > > > > w
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw
> > > @CloudOps_
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at
> 1:02
> > > PM,
> > > > > >> Syed
> > > > > >> > > Ahmed
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > >
> > > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > >
> https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new
> > set
> > > > of
> > > > > >> APIs
> > > > > >> > to
> > > > > >> > > > do
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed
> > storage
> > > > > >> backends
> > > > > >> > > > like
> > > > > >> > > > > > > > > SolidFire.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current
> > managed
> > > > > >> storage
> > > > > >> > > stay
> > > > > >> > > > > on
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary
> > to
> > > > what
> > > > > >> > > > CloudStack
> > > > > >> > > > > > > calls
> > > > > >> > > > > > > > > > > snpshots.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on
> > > > storage
> > > > > >> and
> > > > > >> > > > keeping
> > > > > >> > > > > > it
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own
> > advantages
> > > > > >> > > > > > > > > > > > and
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to
> > > have
> > > > > both
> > > > > >> > ways
> > > > > >> > > > of
> > > > > >> > > > > > > doing
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal
> > > adds
> > > > 4
> > > > > >> new
> > > > > >> > > APIs
> > > > > >> > > > to
> > > > > >> > > > > > > > create
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend
> > > storage.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think
> of
> > > this
> > > > > >> > > feature? I
> > > > > >> > > > > > would
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some
> > feedback.
> > > I
> > > > am
> > > > > >> > > working
> > > > > >> > > > on
> > > > > >> > > > > > > > making
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > >> > > > > > > > > > > > spec
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted
> > to
> > > > > have a
> > > > > >> > high
> > > > > >> > > > > level
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before
> > > starting
> > > > > to
> > > > > >> > work
> > > > > >> > > on
> > > > > >> > > > > it.
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> --
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack
> Developer,
> > > > > >> SolidFire
> > > > > >> > > Inc.*
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> e:
> > mike.tutkowski@solidfire.com
> > > > > >> > > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world
> > uses
> > > > the
> > > > > >> > cloud
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> <
> > > > > >> > > > > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > > > > >*™
> > > > > >> > > > > > > > > > > > > > >>>>>>>>> *
> > > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>> --
> > > > > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer,
> > > > SolidFire
> > > > > >> > Inc.*
> > > > > >> > > > > > > > > > > > > > >>>>>> e:
> mike.tutkowski@solidfire.com
> > > > > >> > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > >>>>>> Advancing the way the world
> uses
> > > the
> > > > > >> cloud
> > > > > >> > > > > > > > > > > > > > >>>>>> <
> > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > > > >*™*
> > > > > >> > > > > > > > > > > > > > >>>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>> --
> > > > > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer,
> > > > SolidFire
> > > > > >> Inc.*
> > > > > >> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > > >> > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > >>>>> Advancing the way the world uses
> > the
> > > > > cloud
> > > > > >> > > > > > > > > > > > > > >>>>> <
> > > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > > > >*™*
> > > > > >> > > > > > > > > > > > > > >>>>>
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>>
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>> --
> > > > > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer,
> > > SolidFire
> > > > > >> Inc.*
> > > > > >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > >>> Advancing the way the world uses
> the
> > > > cloud
> > > > > >> > > > > > > > > > > > > > >>> <
> > > > > >> > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > > >*™*
> > > > > >> > > > > > > > > > > > > > >>>
> > > > > >> > > > > > > > > > > > > > >>
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > > --
> > > > > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > > > > *Senior CloudStack Developer,
> > SolidFire
> > > > > Inc.*
> > > > > >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > >> <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > > > > o: 303.746.7302
> > > > > >> > > > > > > > > > > > > > > Advancing the way the world uses the
> > > cloud
> > > > > >> > > > > > > > > > > > > > > <
> > > > > >> > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > > > > >*™*
> > > > > >> > > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Find out more about ShapeBlue and our
> > > range
> > > > of
> > > > > >> > > > CloudStack
> > > > > >> > > > > > > > > > > > > > related
> > > > > >> > > > > > > > > > > > > services:
> > > > > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > > >> > > > > > > > > > > > > > <
> > > > > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > > > >> > > |
> > > > > >> > > > > > > > CSForge –
> > > > > >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > > > > >> > > > > > > http://shapeblue.com/csforge/
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > > > > > CloudStack Consulting
> > > > > >> > > > > > > > > > > > > > <
> > > > http://shapeblue.com/cloudstack-consultancy/
> > > > > >
> > > > > >> |
> > > > > >> > > > > > > > > > > > > CloudStack
> > > > > >> > > > > > > > > > > > > > Software Engineering
> > > > > >> > > > > > > > > > > > > > <
> > > > > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > > > > >> > > > >
> > > > > >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > > > > >> > > > > > > > > > > > > > <
> > > > > >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > >> > > > > >
> > > > > >> > > > > > |
> > > > > >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > >> > > > > > > > > > > > > > <
> > > http://shapeblue.com/cloudstack-training/>
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > --
> > > > > >> > > > > > > > > > > > *Mike Tutkowski*
> > > > > >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> > Inc.*
> > > > > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > <javascript:;>
> > > > > >> > > > > <javascript:;>
> > > > > >> > > > > > > > > > > > o: 303.746.7302
> > > > > >> > > > > > > > > > > > Advancing the way the world uses the cloud
> > > > > >> > > > > > > > > > > > <
> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > >> > >*™*
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > Find out more about ShapeBlue and our range
> of
> > > > > >> CloudStack
> > > > > >> > > > > related
> > > > > >> > > > > > > > > > services:
> > > > > >> > > > > > > > > > > IaaS Cloud Design & Build
> > > > > >> > > > > > > > > > > <
> > > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > > >
> > > > > >> |
> > > > > >> > > > > CSForge –
> > > > > >> > > > > > > > rapid
> > > > > >> > > > > > > > > > > IaaS deployment framework <
> > > > > >> http://shapeblue.com/csforge/
> > > > > >> > >
> > > > > >> > > > > > > CloudStack
> > > > > >> > > > > > > > > > > Consulting <
> > > > > >> http://shapeblue.com/cloudstack-consultancy/
> > > > > >> > >
> > > > > >> > > |
> > > > > >> > > > > > > > CloudStack
> > > > > >> > > > > > > > > > > Software Engineering
> > > > > >> > > > > > > > > > > <
> > > > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > > > >> > > > > > > > > > > CloudStack Infrastructure Support
> > > > > >> > > > > > > > > > > <
> > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > >> > >
> > > > > >> > > |
> > > > > >> > > > > > > > CloudStack
> > > > > >> > > > > > > > > > > Bootcamp Training Courses <
> > > > > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > Find out more about ShapeBlue and our range of
> > > > > >> CloudStack
> > > > > >> > > > related
> > > > > >> > > > > > > > > services:
> > > > > >> > > > > > > > > > IaaS Cloud Design & Build
> > > > > >> > > > > > > > > > <
> > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > >
> > > > > |
> > > > > >> > > > CSForge –
> > > > > >> > > > > > > rapid
> > > > > >> > > > > > > > > > IaaS deployment framework <
> > > > > >> http://shapeblue.com/csforge/>
> > > > > >> > > > > > > > > > CloudStack Consulting <
> > > > > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > > >> > > > > > > |
> > > > > >> > > > > > > > > CloudStack
> > > > > >> > > > > > > > > > Software Engineering
> > > > > >> > > > > > > > > > <
> > > > > http://shapeblue.com/cloudstack-software-engineering/>
> > > > > >> > > > > > > > > > CloudStack Infrastructure Support
> > > > > >> > > > > > > > > > <
> > > > > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > > > >> > |
> > > > > >> > > > > > > CloudStack
> > > > > >> > > > > > > > > > Bootcamp Training Courses <
> > > > > >> > > > > > http://shapeblue.com/cloudstack-training/
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > --
> > > > > >> > > > > > > *Mike Tutkowski*
> > > > > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > <javascript:;>
> > > > > >> > > > > > > o: 303.746.7302
> > > > > >> > > > > > > Advancing the way the world uses the cloud
> > > > > >> > > > > > > <http://solidfire.com/solution/overview/?video=play
> > >*™*
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > --
> > > > > >> > > > > *Mike Tutkowski*
> > > > > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > >> > > > > o: 303.746.7302
> > > > > >> > > > > Advancing the way the world uses the cloud
> > > > > >> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > --
> > > > > >> > > > Ian Rae
> > > > > >> > > > CEO | PDG
> > > > > >> > > > c: 514.944.4008
> > > > > >> > > >
> > > > > >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> > > > > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J
> > 1S6
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > *Mike Tutkowski*
> > > > > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >> > e: mike.tutkowski@solidfire.com
> > > > > >> > o: 303.746.7302
> > > > > >> > Advancing the way the world uses the cloud
> > > > > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > e: mike.tutkowski@solidfire.com
> > > > > > o: 303.746.7302
> > > > > > Advancing the way the world uses the cloud
> > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Correct, Will.

That Global Settings would only be for managed storage. Non-managed
(traditional) volume snapshots are completely un-impacted by this feature.

If we need to sometimes keep the snapshots on the SAN and sometimes push
them to secondary storage, we'll need a more robust solution than Global
Settings, though.

On Mon, Feb 8, 2016 at 12:11 PM, Will Stevens <ws...@cloudops.com> wrote:

> Sorry.  I missed a bit of context when I responded.  The global setting
> would be only for the managed storage case, currently being called Storage
> Snapshots, and is only to determine if a copy is pushed to secondary
> storage right?  The global setting would not change the behavior of the
> Volume Snapshots right?
>
> I was referring to the need for Volume Snapshots and Storage Snapshots to
> exist together.  Disregard my comment.  I caught up on context after I
> posted.  My bad...
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com
> > wrote:
>
> > Hey Will,
> >
> > Who's picking the behavior? Is it the cloud provider or the end user?
> >
> > Thanks
> >
> > On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> > > I don't think a global setting is a good option because we need both
> > > functionality to be available at the same time and for different use
> > cases
> > > to be able to pick which they choose.
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > *CloudOps* *| *Cloud Solutions Experts
> > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > w cloudops.com *|* tw @CloudOps_
> > >
> > > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com
> > > > wrote:
> > >
> > > > Now that I re-read your e-mail, it dawned on me: The end-user doesn't
> > > care
> > > > where the snapshot is.
> > > >
> > > > If that's true, then we should perhaps control this via Global
> Settings
> > > or
> > > > something.
> > > >
> > > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com> wrote:
> > > >
> > > > > It's not ideal - true, but it does allow us to be backward
> > compatible.
> > > > >
> > > > > If you have other ideas, though, about how to maintain backward
> > > > > compatibility, I'm definitely open to hear them.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> > syed1.mushtaq@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Mike,
> > > > >>
> > > > >> Adding a flag to createSnapshot was the first and the most obvious
> > > thing
> > > > >> that came to our minds. The problem that I had with this was that:
> > > > >>
> > > > >> 1) I feel it is exposing something to the end user that is
> internal
> > to
> > > > the
> > > > >> cloud.
> > > > >>
> > > > >> 2) We have to follow two different ways of restore/deletion in the
> > > same
> > > > >> code path depending on where the Snapshot resides which I find
> kind
> > > of a
> > > > >> bad design.
> > > > >>
> > > > >> But if exposing a archive flag to the end user is acceptable then
> we
> > > can
> > > > >> definitely use this instead of adding the StorageSnapshot API
> > > > >>
> > > > >> Thanks,
> > > > >> -Syed
> > > > >>
> > > > >>
> > > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > > >> mike.tutkowski@solidfire.com
> > > > >> > wrote:
> > > > >>
> > > > >> > Hi Pierre-Luc,
> > > > >> >
> > > > >> > My recommendation would be this:
> > > > >> >
> > > > >> > Add an "archive" flag to the current volume-snapshot API. Its
> > > default
> > > > >> would
> > > > >> > be "false" because that would be backward compatible with how
> 4.6
> > > has
> > > > >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6,
> > 4.7,
> > > > and
> > > > >> > 4.8).
> > > > >> >
> > > > >> > If you set archive=true, then we would perform a background
> > > migration
> > > > of
> > > > >> > the snapshot from the SAN to the secondary storage (then delete
> > the
> > > > SAN
> > > > >> > snapshot).
> > > > >> >
> > > > >> > That archive parameter would only be valid for managed storage.
> > > > >> >
> > > > >> > Sound reasonable?
> > > > >> >
> > > > >> > Also, a VM snapshot that includes disks provided by managed
> > storage
> > > > >> should
> > > > >> > work.
> > > > >> >
> > > > >> > Talk to you later,
> > > > >> > Mike
> > > > >> >
> > > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> > pdion@cloudops.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Mike,
> > > > >> > >
> > > > >> > > In terms of API's, would you prefer introducing a parameter to
> > the
> > > > >> > existing
> > > > >> > > VolumeSnapshot, example:   extract={true|false}  with a
> default
> > > > value
> > > > >> of
> > > > >> > > true  which would extract snapshot into the secondary storage,
> > > which
> > > > >> is
> > > > >> > the
> > > > >> > > current default behavior. Then for SAN snapshot that remain on
> > the
> > > > >> SAN we
> > > > >> > > would just set "extract=false" ?  as oppose to create a new
> > > > >> > >  StorageSnapshot API ?
> > > > >> > >
> > > > >> > >
> > > > >> > > Paul,
> > > > >> > >
> > > > >> > > From what I'm seeing so far, we can't do a VM-snapshot when
> > using
> > > > >> managed
> > > > >> > > storage for VM having more than one Volume. For the reason
> that
> > > > >> snapshot
> > > > >> > > are performed outside of the hypervisor awareness and
> > > > asynchronously.
> > > > >> If
> > > > >> > > someone have a way to address that, it would make thinks much
> > more
> > > > >> > > attractive.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com>
> > > wrote:
> > > > >> > >
> > > > >> > > > I think a service provider backup scenario is more likely to
> > > take
> > > > >> > > advantage
> > > > >> > > > of SAN snapshot. There are a few reasons for this.
> Traditional
> > > > >> backups
> > > > >> > > > involve access to the file system, and there is an
> expectation
> > > > that
> > > > >> > this
> > > > >> > > > can be done with reasonably short time frames without
> > negatively
> > > > >> > > impacting
> > > > >> > > > VM performance, and that the backup orchestrator can apply
> > > various
> > > > >> > logic
> > > > >> > > > and or transformations to the data (compress, encrypt,
> deltas
> > > > >> etc...).
> > > > >> > > > While it is true that one could apply a backup process to a
> > > cloud
> > > > >> > > snapshot,
> > > > >> > > > this would be slow and inefficient requiring the data to be
> > > moved
> > > > >> > several
> > > > >> > > > times and there are some major bottlenecks with cloud
> > snapshots.
> > > > >> With a
> > > > >> > > > cloud snapshot - there seems to be no reasonable expectation
> > of
> > > > >> being
> > > > >> > > able
> > > > >> > > > to do differential snapshots (I think this depends on the
> > > > >> hypervisor)
> > > > >> > and
> > > > >> > > > if you do differential snapshots this will make file backups
> > > from
> > > > >> those
> > > > >> > > > snapshots even more complicated to orchestrate.
> > > > >> > > >
> > > > >> > > > Suspect there needs to be a different thread on how to
> better
> > > > enable
> > > > >> > > > backups, as a service. As per Paul's suggestion, but it is a
> > > > related
> > > > >> > > > workflow so relevant to this discussion.
> > > > >> > > >
> > > > >> > > > Ian
> > > > >> > > >
> > > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > > >> > > mike.tutkowski@solidfire.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > To me it sounds like number two and number three are
> > different
> > > > >> uses
> > > > >> > for
> > > > >> > > > the
> > > > >> > > > > same "thing"(which is totally fine).
> > > > >> > > > >
> > > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > > >> asynchronously, do
> > > > >> > > we
> > > > >> > > > > see the SSVM as performing the export?
> > > > >> > > > >
> > > > >> > > > > To be backwards compatible with what we have in 4.6 and
> > later
> > > > for
> > > > >> > > volume
> > > > >> > > > > snapshots for managed storage, I think it might be easier
> if
> > > we
> > > > >> pass
> > > > >> > > in a
> > > > >> > > > > flag that says whether or not to archive the SAN snapshot
> > > > (which,
> > > > >> I
> > > > >> > > > think,
> > > > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > > > >> > > > >
> > > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > > pdion@cloudops.com
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi Mike,
> > > > >> > > > > >
> > > > >> > > > > > The reason behind the creation of a SAN snapshot which
> is
> > > > >> exported
> > > > >> > > into
> > > > >> > > > > > secondary storage, is because creating a copy of the
> .VHD
> > > > >> directly
> > > > >> > > > would
> > > > >> > > > > > impact uptime of the VM as creating that copy take lots
> of
> > > > time.
> > > > >> > Has
> > > > >> > > > > oppose
> > > > >> > > > > > to a SAN snapshot that is close to instantaneous which
> can
> > > > >> > afterward
> > > > >> > > be
> > > > >> > > > > > clone into Secondary Storage asynchronously.
> > > > >> > > > > >
> > > > >> > > > > > I would suspect an extracted VolumeSnapshot taken from a
> > SAN
> > > > >> > snapshot
> > > > >> > > > > could
> > > > >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> > > > >> consumption
> > > > >> > > on
> > > > >> > > > > the
> > > > >> > > > > > Primary Storage side.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > I see 3 definitions in our current discussion regarding
> > the
> > > > term
> > > > >> > > > snapshot
> > > > >> > > > > > (these are not official terminology but by own
> > > interpretation
> > > > of
> > > > >> > > them):
> > > > >> > > > > >
> > > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition
> > of
> > > a
> > > > >> > > > snapshot):
> > > > >> > > > > > it's a volume snapshot at the storage level, point in
> time
> > > of
> > > > >> your
> > > > >> > > > data.
> > > > >> > > > > it
> > > > >> > > > > > reside on the primary storage. Useful and efficient for
> > > > software
> > > > >> > side
> > > > >> > > > > > incident.
> > > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/
> > cloud
> > > > >> backup
> > > > >> > > > aws-S3
> > > > >> > > > > > style ): Point in time copy of the Virtual Disk that
> > reside
> > > > on a
> > > > >> > > > > different
> > > > >> > > > > > storage array then the original Volume. Facilitate data
> > > > >> migration
> > > > >> > > > between
> > > > >> > > > > > clusters and, in case of primary storage incident,
> Volume
> > > > >> snapshots
> > > > >> > > are
> > > > >> > > > > not
> > > > >> > > > > > impacted and can be reuse.
> > > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that
> > > also
> > > > >> > > validate
> > > > >> > > > > data
> > > > >> > > > > > integrity, provide a storage efficient archiving method
> > and
> > > an
> > > > >> > > > > independent
> > > > >> > > > > > way to restore your data in case of an major
> > infrastructure
> > > > >> > disaster.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Regards,
> > > > >> > > > > >
> > > > >> > > > > > PL
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>
> > > > >> > > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > So, let's see if I currently follow the requirements:
> > > > >> > > > > > >
> > > > >> > > > > > > * Augment volume snapshots for managed storage to
> > > > >> conditionally
> > > > >> > > > export
> > > > >> > > > > > data
> > > > >> > > > > > > to NFS. The current process of taking a snapshot on
> the
> > > SAN
> > > > is
> > > > >> > > fine,
> > > > >> > > > > but
> > > > >> > > > > > > we'd like the option to export the data to NFS, as
> well.
> > > > >> > > > > > >
> > > > >> > > > > > > Questions:
> > > > >> > > > > > >
> > > > >> > > > > > > Once the data has been exported to NFS, do we keep the
> > SAN
> > > > >> > snapshot
> > > > >> > > > or
> > > > >> > > > > > > delete it?
> > > > >> > > > > > >
> > > > >> > > > > > > If we are deleting the SAN snapshot, then why don't we
> > > just
> > > > >> copy
> > > > >> > > the
> > > > >> > > > > VHD
> > > > >> > > > > > > from primary to secondary the way we do today for
> > > > non-managed
> > > > >> > (i.e.
> > > > >> > > > > > > traditional) storage? Why create a SAN snapshot in
> that
> > > > >> scenario?
> > > > >> > > > > Perhaps
> > > > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> > > secondary
> > > > >> > > storage
> > > > >> > > > > > > instead of a XenServer host?
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks for the clarification.
> > > > >> > > > > > >
> > > > >> > > > > > > By the way, to me a backup is when you copy data from
> > one
> > > > >> storage
> > > > >> > > > > system
> > > > >> > > > > > to
> > > > >> > > > > > > another (regardless of features, if any, to restore
> the
> > > data
> > > > >> in
> > > > >> > the
> > > > >> > > > > > > future). A snapshot is a point-in-time view of the
> data
> > > of a
> > > > >> > volume
> > > > >> > > > and
> > > > >> > > > > > > it's stored on the same storage system as the volume.
> > > > >> > > > > > >
> > > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > > >> > > pdion@cloudops.com
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > <javascript:;>>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > That's fun to see that discussion happening. I 100%
> > > agree
> > > > >> with
> > > > >> > > > Paul's
> > > > >> > > > > > > > points of view. VolumeSnapshot are not a backup,
> but I
> > > do
> > > > >> > > consider
> > > > >> > > > > them
> > > > >> > > > > > > as
> > > > >> > > > > > > > a safety vest against Primary Storage failure,
> because
> > > > >> failure
> > > > >> > > > append
> > > > >> > > > > > > :-( .
> > > > >> > > > > > > >
> > > > >> > > > > > > > The current proposal around snapshots that reside on
> > the
> > > > >> > primary
> > > > >> > > > > > storage
> > > > >> > > > > > > or
> > > > >> > > > > > > > snapshots that end in the Secondary Storage  is not
> to
> > > > >> address
> > > > >> > > any
> > > > >> > > > > kind
> > > > >> > > > > > > of
> > > > >> > > > > > > > backups requirement because a snapshot is not a
> > backup,
> > > > >> event
> > > > >> > an
> > > > >> > > > > > > extracted
> > > > >> > > > > > > > VM snapshot.
> > > > >> > > > > > > >
> > > > >> > > > > > > > The main idea, and again this is for managed
> storage;
> > > > >> > > > > > > >
> > > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> > > > >> capability
> > > > >> > > for
> > > > >> > > > > > fast
> > > > >> > > > > > > > response time that support rollback to previous
> > > timestamp,
> > > > >> > create
> > > > >> > > > new
> > > > >> > > > > > > > volume and maybe create template.
> > > > >> > > > > > > >     not required to be a new API if the work is
> > already
> > > > >> done, I
> > > > >> > > > think
> > > > >> > > > > > > this
> > > > >> > > > > > > > is a different behaviors than the user expectation
> of
> > a
> > > > >> > > > > > volume-snapshot.
> > > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> > > behavior
> > > > >> that
> > > > >> > > > create
> > > > >> > > > > > an
> > > > >> > > > > > > > extraction of a volume into SecondaryStorage which
> can
> > > be
> > > > >> reuse
> > > > >> > > to
> > > > >> > > > > > > create a
> > > > >> > > > > > > > new volume into another Primary Storage. This type
> of
> > > > >> snapshot
> > > > >> > > is a
> > > > >> > > > > > slow
> > > > >> > > > > > > > job since yes it would have to copy the full volume
> > size
> > > > on
> > > > >> the
> > > > >> > > > > > Secondary
> > > > >> > > > > > > > storage.
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > PL
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > >> > > > > > <javascript:;>>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > I think I share you view on the 'Ideal world'.
> > Backup
> > > > (via
> > > > >> > > Volume
> > > > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack.
> This
> > is
> > > > >> > > amplified
> > > > >> > > > > > > > especially
> > > > >> > > > > > > > > when you have a object storage as your secondary
> > > storage
> > > > >> > > because
> > > > >> > > > it
> > > > >> > > > > > > > > requires two copies (one to an NFS staging area
> and
> > > from
> > > > >> > there
> > > > >> > > to
> > > > >> > > > > > > object
> > > > >> > > > > > > > > storage). And not to mention that all these copies
> > are
> > > > >> > > consuming
> > > > >> > > > > > > > hypervisor
> > > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> > bottleneck
> > > as
> > > > >> all
> > > > >> > > the
> > > > >> > > > > > > Network
> > > > >> > > > > > > > > and I/O flow through it. So our intention of
> > proposing
> > > > the
> > > > >> > > > "Storage
> > > > >> > > > > > > > > Snapshots" is to give a better way of achiving
> > > snapshots
> > > > >> > while
> > > > >> > > > > still
> > > > >> > > > > > > > > keeping the original definition of volume
> snpashots
> > > (ie
> > > > >> > upload
> > > > >> > > to
> > > > >> > > > > sec
> > > > >> > > > > > > > > storage).
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > > > backups.
> > > > >> > They
> > > > >> > > > > don't
> > > > >> > > > > > > work
> > > > >> > > > > > > > > form multi-disk LVM volume groups and dynamic
> > disks. I
> > > > am
> > > > >> all
> > > > >> > > in
> > > > >> > > > > for
> > > > >> > > > > > a
> > > > >> > > > > > > > > better backup solution which handles these use
> cases
> > > and
> > > > >> > > utilizes
> > > > >> > > > > the
> > > > >> > > > > > > > > storage's advanced features.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > > >> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > In the beginning... there were CloudStack
> > snapshots
> > > > and
> > > > >> > they
> > > > >> > > > were
> > > > >> > > > > > > > > actually
> > > > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > > > snapshots.
> > > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > > point-in-time
> > > > >> > > > > hypervisor
> > > > >> > > > > > > > > > snapshots) and we started referring to the
> > original
> > > > >> > snapshots
> > > > >> > > > as
> > > > >> > > > > > > volume
> > > > >> > > > > > > > > > snapshots.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > CloudStack does not offer 'backups', but many
> > people
> > > > use
> > > > >> > > volume
> > > > >> > > > > > > > snapshots
> > > > >> > > > > > > > > > as backups. However you can't in-place restore
> > > volume
> > > > >> > > snapshots
> > > > >> > > > > and
> > > > >> > > > > > > if
> > > > >> > > > > > > > > you
> > > > >> > > > > > > > > > have a VM with multiple volumes, the volume
> > > snapshots
> > > > >> must
> > > > >> > be
> > > > >> > > > > done
> > > > >> > > > > > in
> > > > >> > > > > > > > > > series, meaning that the state across all of the
> > > > >> volumes is
> > > > >> > > > > > unlikely
> > > > >> > > > > > > to
> > > > >> > > > > > > > > be
> > > > >> > > > > > > > > > consistent.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> > > > options
> > > > >> > > which
> > > > >> > > > > > users
> > > > >> > > > > > > > > might
> > > > >> > > > > > > > > > expect as well options as to where they might be
> > > > >> stored. In
> > > > >> > > my
> > > > >> > > > > > ideal
> > > > >> > > > > > > > > world
> > > > >> > > > > > > > > > they would also be able to leverage back-end
> > > hardware
> > > > >> (such
> > > > >> > > as
> > > > >> > > > > > > > Solidfire,
> > > > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> > > Commvault
> > > > >> etc
> > > > >> > to
> > > > >> > > > > > > accelerate
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > process.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > >> > > > > > > > > > Paul Angus
> > > > >> > > > > > > > > > VP Technology ,  ShapeBlue
> > > > >> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > >> > > > > > > > > >
> > > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> |
> > > > >> > > > > m:
> > > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > >> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> > > > >> > <javascript:;>
> > > > >> > > |
> > > > >> > > > > t: @cloudyangus*
> > > > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > >> > > > > <ja...@cloudyangus>
> > > > >> > > > > > |  w:
> > > > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > >> > > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N
> > 4HS
> > > UK
> > > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in
> > England
> > > &
> > > > >> > Wales.
> > > > >> > > > > > > ShapeBlue
> > > > >> > > > > > > > > > Services India LLP is a company incorporated in
> > > India
> > > > >> and
> > > > >> > is
> > > > >> > > > > > operated
> > > > >> > > > > > > > > under
> > > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > > >> Consultoria
> > > > >> > > Ltda
> > > > >> > > > > is
> > > > >> > > > > > a
> > > > >> > > > > > > > > > company incorporated in Brasil and is operated
> > under
> > > > >> > license
> > > > >> > > > from
> > > > >> > > > > > > Shape
> > > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > registered
> > > > >> by
> > > > >> > The
> > > > >> > > > > > > Republic
> > > > >> > > > > > > > of
> > > > >> > > > > > > > > > South Africa and is traded under license from
> > Shape
> > > > Blue
> > > > >> > Ltd.
> > > > >> > > > > > > ShapeBlue
> > > > >> > > > > > > > > is
> > > > >> > > > > > > > > > a registered trademark.
> > > > >> > > > > > > > > > This email and any attachments to it may be
> > > > confidential
> > > > >> > and
> > > > >> > > > are
> > > > >> > > > > > > > intended
> > > > >> > > > > > > > > > solely for the use of the individual to whom it
> is
> > > > >> > addressed.
> > > > >> > > > Any
> > > > >> > > > > > > views
> > > > >> > > > > > > > > or
> > > > >> > > > > > > > > > opinions expressed are solely those of the
> author
> > > and
> > > > do
> > > > >> > not
> > > > >> > > > > > > > necessarily
> > > > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > > > companies.
> > > > >> If
> > > > >> > > you
> > > > >> > > > > are
> > > > >> > > > > > > not
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > intended recipient of this email, you must
> neither
> > > > take
> > > > >> any
> > > > >> > > > > action
> > > > >> > > > > > > > based
> > > > >> > > > > > > > > > upon its contents, nor copy or show it to
> anyone.
> > > > Please
> > > > >> > > > contact
> > > > >> > > > > > the
> > > > >> > > > > > > > > sender
> > > > >> > > > > > > > > > if you believe you have received this email in
> > > error.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > -----Original Message-----
> > > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> > syed1.mushtaq@gmail.com
> > > > >> > > > > <javascript:;> <javascript:;>]
> > > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > >> > <javascript:;>
> > > > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > Snapshots
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Paul,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > When you say actual backups, how would it be
> > > different
> > > > >> from
> > > > >> > > the
> > > > >> > > > > > > Volume
> > > > >> > > > > > > > > > Snapshots that exist currently. My understanding
> > is
> > > > that
> > > > >> > > > Backups
> > > > >> > > > > > end
> > > > >> > > > > > > up
> > > > >> > > > > > > > > in
> > > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > > point-in-time
> > > > >> > state
> > > > >> > > of
> > > > >> > > > > > your
> > > > >> > > > > > > > > volume
> > > > >> > > > > > > > > > which can be restored back correct?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > -Syed
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > > >> > > > > > > paul.angus@shapeblue.com <javascript:;>
> <javascript:;>>
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > Hi Syed,
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > As I understand it, the SolidFire plugin will
> > > export
> > > > >> the
> > > > >> > > > > snapshot
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > > secondary storage if the user requests a
> > template
> > > > from
> > > > >> > the
> > > > >> > > > > > snapshot
> > > > >> > > > > > > > or
> > > > >> > > > > > > > > > > wants to download the snapshot from the cloud.
> > > This
> > > > >> is a
> > > > >> > > > good,
> > > > >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> > > > storage
> > > > >> is
> > > > >> > > > super
> > > > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays
> take
> > up
> > > > >> next
> > > > >> > to
> > > > >> > > no
> > > > >> > > > > > > space.
> > > > >> > > > > > > > > > > BUT I think that we are talking about a more
> > > general
> > > > >> > > purpose
> > > > >> > > > > API,
> > > > >> > > > > > > and
> > > > >> > > > > > > > > > > other storage systems may not be as awesome as
> > > > Mike's.
> > > > >> > > That's
> > > > >> > > > > my
> > > > >> > > > > > > > > > > concern. Also, the time to transfer for say
> 1TB
> > to
> > > > >> move
> > > > >> > > from
> > > > >> > > > > > > primary
> > > > >> > > > > > > > > > > to sec storage and then create a VM template
> out
> > > of
> > > > it
> > > > >> > may
> > > > >> > > be
> > > > >> > > > > too
> > > > >> > > > > > > > long
> > > > >> > > > > > > > > > for users.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > > > snapshot
> > > > >> for
> > > > >> > > > > backup,
> > > > >> > > > > > > > it's
> > > > >> > > > > > > > > > > just that users want to do backups and a
> volume
> > > > >> snapshot
> > > > >> > is
> > > > >> > > > the
> > > > >> > > > > > > only
> > > > >> > > > > > > > > > > type of snapshot that copies the disk
> elsewhere
> > > and
> > > > >> can
> > > > >> > be
> > > > >> > > > > > > scheduled.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> > > actual
> > > > >> > backups
> > > > >> > > > > > > (through
> > > > >> > > > > > > > > > > recognised backup providers) and the user
> > > > requirements
> > > > >> > > around
> > > > >> > > > > > them
> > > > >> > > > > > > > > > > (particularly restoration use cases) as a
> > separate
> > > > >> thread
> > > > >> > > of
> > > > >> > > > > > work.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > Paul
> > > > >> Angus
> > > > >> > > VP
> > > > >> > > > > > > > Technology
> > > > >> > > > > > > > > > > , ShapeBlue
> > > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > >> > > > > > > > > > >
> > > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > >> > > > |
> > > > >> > > > > m:
> > > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > > >> > > <javascript:;> |
> > > > >> > > > > t: @cloudyangus*
> > > > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > >> <javascript:;>
> > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com
> >
> > > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> > 4HS
> > > > UK
> > > > >> > Shape
> > > > >> > > > > Blue
> > > > >> > > > > > > Ltd
> > > > >> > > > > > > > > > > is a company incorporated in England & Wales.
> > > > >> ShapeBlue
> > > > >> > > > > Services
> > > > >> > > > > > > > India
> > > > >> > > > > > > > > > > LLP is a company incorporated in India and is
> > > > operated
> > > > >> > > under
> > > > >> > > > > > > license
> > > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> > Consultoria
> > > > >> Ltda
> > > > >> > is
> > > > >> > > a
> > > > >> > > > > > > company
> > > > >> > > > > > > > > > > incorporated in Brasil and is operated under
> > > license
> > > > >> from
> > > > >> > > > Shape
> > > > >> > > > > > > Blue
> > > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> > registered
> > > by
> > > > >> The
> > > > >> > > > > Republic
> > > > >> > > > > > > of
> > > > >> > > > > > > > > > > South Africa and is traded under license from
> > > Shape
> > > > >> Blue
> > > > >> > > Ltd.
> > > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > >> > > > > > > > > > > This email and any attachments to it may be
> > > > >> confidential
> > > > >> > > and
> > > > >> > > > > are
> > > > >> > > > > > > > > > > intended solely for the use of the individual
> to
> > > > whom
> > > > >> it
> > > > >> > is
> > > > >> > > > > > > > addressed.
> > > > >> > > > > > > > > > > Any views or opinions expressed are solely
> those
> > > of
> > > > >> the
> > > > >> > > > author
> > > > >> > > > > > and
> > > > >> > > > > > > do
> > > > >> > > > > > > > > > > not necessarily represent those of Shape Blue
> > Ltd
> > > or
> > > > >> > > related
> > > > >> > > > > > > > > > > companies. If you are not the intended
> recipient
> > > of
> > > > >> this
> > > > >> > > > email,
> > > > >> > > > > > you
> > > > >> > > > > > > > > > > must neither take any action based upon its
> > > > contents,
> > > > >> nor
> > > > >> > > > copy
> > > > >> > > > > or
> > > > >> > > > > > > > show
> > > > >> > > > > > > > > > > it to anyone. Please contact the sender if you
> > > > believe
> > > > >> > you
> > > > >> > > > have
> > > > >> > > > > > > > > received
> > > > >> > > > > > > > > > this email in error.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > -----Original Message-----
> > > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > > syed1.mushtaq@gmail.com
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > <javascript:;>]
> > > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > >> > > <javascript:;>
> > > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > > Snapshots
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > I think the terminology confusion comes from
> AWS
> > > > where
> > > > >> > they
> > > > >> > > > do
> > > > >> > > > > > EBS
> > > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort
> of
> > > > >> followed
> > > > >> > > > that.
> > > > >> > > > > > And
> > > > >> > > > > > > > as
> > > > >> > > > > > > > > > > an end user who is oblivious to the internals
> of
> > > my
> > > > >> > > provider,
> > > > >> > > > > my
> > > > >> > > > > > > > > > > expectation would be something similar to what
> > AWS
> > > > as
> > > > >> > that
> > > > >> > > is
> > > > >> > > > > my
> > > > >> > > > > > > > > > > biggest reference point.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > To your point Mike, I agree that a Primary
> > Storage
> > > > >> > failure
> > > > >> > > on
> > > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> > motivations
> > > > >> for us
> > > > >> > > to
> > > > >> > > > > push
> > > > >> > > > > > > > data
> > > > >> > > > > > > > > > > to secondary storage. Primary storage (atleast
> > for
> > > > us)
> > > > >> > > costs
> > > > >> > > > > > > around 3
> > > > >> > > > > > > > > > > times as much as secondary storage and
> snapshots
> > > on
> > > > >> > primary
> > > > >> > > > > > storage
> > > > >> > > > > > > > > > > are rarely used (especially for some of our
> > > > customers
> > > > >> who
> > > > >> > > do
> > > > >> > > > > > daily
> > > > >> > > > > > > > > > backups).
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike
> Tutkowski
> > <
> > > > >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > >> > > <javascript:;>>
> > > > >> > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > Some of the weirdness is around terminology.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > For most systems I've worked on, a snapshot
> > and
> > > a
> > > > >> > backup
> > > > >> > > > are
> > > > >> > > > > > two
> > > > >> > > > > > > > > > > > completely different things (but CloudStack
> > has
> > > > >> > > > traditionally
> > > > >> > > > > > > used
> > > > >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and
> say,
> > > > >> though,
> > > > >> > > that
> > > > >> > > > > if
> > > > >> > > > > > > your
> > > > >> > > > > > > > > > > > primary storage is running on SolidFire that
> > it
> > > is
> > > > >> > > unlikely
> > > > >> > > > > > > you'll
> > > > >> > > > > > > > > > > > encounter an issue where your primary
> storage
> > > goes
> > > > >> > > offline
> > > > >> > > > > (and
> > > > >> > > > > > > > > > > > you'll even maintain your performance
> > guarantees
> > > > >> during
> > > > >> > > > > failure
> > > > >> > > > > > > > > > > > scenarios and upgrades, as well). That being
> > the
> > > > >> case,
> > > > >> > it
> > > > >> > > > is
> > > > >> > > > > > less
> > > > >> > > > > > > > > > > > useful to require a backup to Swift (but
> it's
> > > > >> perfectly
> > > > >> > > OK
> > > > >> > > > if
> > > > >> > > > > > > > that's
> > > > >> > > > > > > > > > > > what we want to do
> > > > >> > > > > > > > > > > here).
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > > > >> > <javascript:;>>
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Paul,
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > I believe with the current implementation
> of
> > > > >> > Snapshots
> > > > >> > > on
> > > > >> > > > > > > managed
> > > > >> > > > > > > > > > > > > storage
> > > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never
> exported
> > > to
> > > > >> the
> > > > >> > > > > secondary
> > > > >> > > > > > > > > > storage.
> > > > >> > > > > > > > > > > > > While this solves the problem of having
> > > > snapshots
> > > > >> > > taking
> > > > >> > > > > > > forever
> > > > >> > > > > > > > > > > > > to get to sec storage, this leaves us
> with a
> > > > >> > > > > > > > > > > > huge
> > > > >> > > > > > > > > > > > > liability if our primary storage goes
> down.
> > We
> > > > see
> > > > >> > > > > snapshots
> > > > >> > > > > > as
> > > > >> > > > > > > > > > > > > our recovery path because we store them in
> > > Swift
> > > > >> > which
> > > > >> > > is
> > > > >> > > > > > > > reliable
> > > > >> > > > > > > > > > > > > and resilient to failures.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> > > > Volume
> > > > >> > > > snapshots
> > > > >> > > > > > > > always
> > > > >> > > > > > > > > > > > > backed up to secondary storage and Storage
> > > > >> Snapshots
> > > > >> > > stay
> > > > >> > > > > on
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > > primary
> > > > >> > > > > > > > > > > > storage.
> > > > >> > > > > > > > > > > > > A provider could potentially mix both
> these
> > > and
> > > > >> solve
> > > > >> > > the
> > > > >> > > > > > > problem
> > > > >> > > > > > > > > > > > > that you mentioned where you want to meet
> > > user's
> > > > >> > > > > expectation
> > > > >> > > > > > > of a
> > > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while
> > > having
> > > > >> an
> > > > >> > > > ability
> > > > >> > > > > > to
> > > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the
> > device)
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > > > > -Syed
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > >> > > <javascript:;>>
> > > > >> > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > HI guys,
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Could someone point me to the Jira bug
> of
> > FS
> > > > for
> > > > >> > the
> > > > >> > > > > > > > > > > > > > SAN-snapshot
> > > > >> > > > > > > > > > > > feature
> > > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > From my discussions with users and
> > operators
> > > > >> around
> > > > >> > > > > > snapshots
> > > > >> > > > > > > > > > > > > > I'd make
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > following observations:
> > > > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups
> (both
> > > > >> long-term
> > > > >> > > and
> > > > >> > > > > > short
> > > > >> > > > > > > > > > > > > > term)
> > > > >> > > > > > > > > > > > with
> > > > >> > > > > > > > > > > > > > the expectation that they can use them
> for
> > > > >> recovery
> > > > >> > > if
> > > > >> > > > > > > > required.
> > > > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > > > something
> > > > >> > has
> > > > >> > > > gone
> > > > >> > > > > > > wrong
> > > > >> > > > > > > > > > > > > > with primary storage.
> > > > >> > > > > > > > > > > > > > c. users sometimes want to be able to
> > export
> > > > >> > > snapshots
> > > > >> > > > as
> > > > >> > > > > > > well
> > > > >> > > > > > > > > > > > > > as
> > > > >> > > > > > > > > > > > create
> > > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > > >> > > > > > > > > > > > > > d. snapshots are a currently a massive
> > pain
> > > > for
> > > > >> > > > > operators,
> > > > >> > > > > > I
> > > > >> > > > > > > > > > > > > > know at
> > > > >> > > > > > > > > > > > > least
> > > > >> > > > > > > > > > > > > > one public cloud who have snapshots
> which
> > > > take 2
> > > > >> > days
> > > > >> > > > to
> > > > >> > > > > > > > > complete.
> > > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used
> > for
> > > > >> > multiple
> > > > >> > > > LVM
> > > > >> > > > > > > > disks.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > I think the process Mike has used in the
> > > > >> SolidFire
> > > > >> > > > plugin
> > > > >> > > > > > > (only
> > > > >> > > > > > > > > > > > > > moving
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > disk image to secondary storage when you
> > > > >> absolutely
> > > > >> > > > have
> > > > >> > > > > > to)
> > > > >> > > > > > > is
> > > > >> > > > > > > > > > > > > > a very
> > > > >> > > > > > > > > > > > > good
> > > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> > > problems
> > > > >> an
> > > > >> > > > > operator
> > > > >> > > > > > > > might
> > > > >> > > > > > > > > > > > > experience
> > > > >> > > > > > > > > > > > > > if they have an issue with a given
> primary
> > > > >> storage
> > > > >> > > pool
> > > > >> > > > > in
> > > > >> > > > > > a
> > > > >> > > > > > > > > > cluster.
> > > > >> > > > > > > > > > > > (I
> > > > >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> > > > >> SolidFire
> > > > >> > > case
> > > > >> > > > > > Mike
> > > > >> > > > > > > :)
> > > > >> > > > > > > > > > > > > > ) And
> > > > >> > > > > > > > > > > > if
> > > > >> > > > > > > > > > > > > > the transfer from primary to secondary
> is
> > > > slow,
> > > > >> the
> > > > >> > > > time
> > > > >> > > > > to
> > > > >> > > > > > > > > > > > > > being able
> > > > >> > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > create a template or export the volume
> > will
> > > be
> > > > >> > slow.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > So for me the issue is around making
> sure
> > > that
> > > > >> the
> > > > >> > > end
> > > > >> > > > > > users
> > > > >> > > > > > > > > > > > expectations
> > > > >> > > > > > > > > > > > > > are met (while improving the
> > > speed/efficiency
> > > > of
> > > > >> > the
> > > > >> > > > back
> > > > >> > > > > > > end)
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > > http://www.shapeblue.com>
> > > > >> Paul
> > > > >> > > > Angus
> > > > >> > > > > > VP
> > > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> > 0540*
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > >
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > >> > > > > > > |
> > > > >> > > > > > > > m:
> > > > >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> > <javascript:;>
> > > > >> > > > > <javascript:;> | t:
> > > > >> > > > > > @cloudyangus*
> > > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com
> <javascript:;>
> > > > >> > > > <javascript:;>
> > > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > > http://www.shapeblue.com
> > > > >
> > > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden
> London
> > > WC2N
> > > > >> 4HS
> > > > >> > UK
> > > > >> > > > > Shape
> > > > >> > > > > > > > Blue
> > > > >> > > > > > > > > > > > > > Ltd is a company incorporated in
> England &
> > > > >> Wales.
> > > > >> > > > > ShapeBlue
> > > > >> > > > > > > > > > > > > > Services India LLP is a company
> > incorporated
> > > > in
> > > > >> > India
> > > > >> > > > and
> > > > >> > > > > > is
> > > > >> > > > > > > > > > > > > > operated
> > > > >> > > > > > > > > > > > > under
> > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> > > Brasil
> > > > >> > > > > Consultoria
> > > > >> > > > > > > Ltda
> > > > >> > > > > > > > > > > > > > is a company incorporated in Brasil and
> is
> > > > >> operated
> > > > >> > > > under
> > > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue
> SA
> > > Pty
> > > > >> Ltd
> > > > >> > is
> > > > >> > > a
> > > > >> > > > > > > company
> > > > >> > > > > > > > > > > > > > registered by The Republic
> > > > >> > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > > > South Africa and is traded under license
> > > from
> > > > >> Shape
> > > > >> > > > Blue
> > > > >> > > > > > Ltd.
> > > > >> > > > > > > > > > > > > > ShapeBlue
> > > > >> > > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > a registered trademark.
> > > > >> > > > > > > > > > > > > > This email and any attachments to it may
> > be
> > > > >> > > > confidential
> > > > >> > > > > > and
> > > > >> > > > > > > > are
> > > > >> > > > > > > > > > > > intended
> > > > >> > > > > > > > > > > > > > solely for the use of the individual to
> > whom
> > > > it
> > > > >> is
> > > > >> > > > > > addressed.
> > > > >> > > > > > > > > > > > > > Any views
> > > > >> > > > > > > > > > > > > or
> > > > >> > > > > > > > > > > > > > opinions expressed are solely those of
> the
> > > > >> author
> > > > >> > and
> > > > >> > > > do
> > > > >> > > > > > not
> > > > >> > > > > > > > > > > > necessarily
> > > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or
> > related
> > > > >> > > companies.
> > > > >> > > > > If
> > > > >> > > > > > > you
> > > > >> > > > > > > > > > > > > > are not
> > > > >> > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > intended recipient of this email, you
> must
> > > > >> neither
> > > > >> > > take
> > > > >> > > > > any
> > > > >> > > > > > > > > > > > > > action
> > > > >> > > > > > > > > > > > based
> > > > >> > > > > > > > > > > > > > upon its contents, nor copy or show it
> to
> > > > >> anyone.
> > > > >> > > > Please
> > > > >> > > > > > > > contact
> > > > >> > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > sender
> > > > >> > > > > > > > > > > > > > if you believe you have received this
> > email
> > > in
> > > > >> > error.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > -----Original Message-----
> > > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > > >> pdion@cloudops.com
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > <javascript:;>]
> > > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature]
> > Storage
> > > > >> > Snapshots
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Hi Mike,
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > > >> StorageSnapshot
> > > > >> > > for
> > > > >> > > > > > > managed
> > > > >> > > > > > > > > > > > > > storage
> > > > >> > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > > > expected,
> > > > >> > > > behavior
> > > > >> > > > > > is
> > > > >> > > > > > > to
> > > > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> > > Storage.
> > > > >> So a
> > > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > > >> > > > > > > > > > > > > for
> > > > >> > > > > > > > > > > > > > snapshot that remain on the managed
> > storage
> > > > >> > > appliance.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Quickly looking at the API doc and I
> don't
> > > > see a
> > > > >> > > strong
> > > > >> > > > > > > > > > > > > > requirement for volume snapshots to be
> > moved
> > > > >> into
> > > > >> > > > > secondary
> > > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API
> is
> > > not
> > > > >> > useful,
> > > > >> > > > but
> > > > >> > > > > > > both
> > > > >> > > > > > > > > > > > > > use cases are required. A snapshot that
> > > remain
> > > > >> on
> > > > >> > the
> > > > >> > > > > > managed
> > > > >> > > > > > > > > > > > > > storage, and another type of
> > > > >> > > > > > > > > > > > snapshot
> > > > >> > > > > > > > > > > > > > that end up into the secondary storage.
> > > Since
> > > > >> > you've
> > > > >> > > > > done a
> > > > >> > > > > > > lot
> > > > >> > > > > > > > > > > > > > of
> > > > >> > > > > > > > > > > > work,
> > > > >> > > > > > > > > > > > > > might easier to just add a parameter to
> > the
> > > > >> current
> > > > >> > > > > > snapshot
> > > > >> > > > > > > > API
> > > > >> > > > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > > trigger an extraction of the storage
> > > snapshot
> > > > >> into
> > > > >> > > the
> > > > >> > > > > > > > secondary
> > > > >> > > > > > > > > > > > storage?
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > PL
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > > > Tutkowski <
> > > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > I think that all sounds reasonable
> then
> > -
> > > > >> thanks!
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > > > Mushtaq <
> > > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > >> <javascript:;>>
> > > > >> > > > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> > > > >> > requirements.
> > > > >> > > > One
> > > > >> > > > > > of
> > > > >> > > > > > > > our
> > > > >> > > > > > > > > > > > earlier
> > > > >> > > > > > > > > > > > > > >> iterations on this was to have an
> > > argument
> > > > to
> > > > >> > the
> > > > >> > > > > create
> > > > >> > > > > > > > > > > > > > >> snapshot
> > > > >> > > > > > > > > > > > API
> > > > >> > > > > > > > > > > > > > >> which decides whether to backup the
> > > volume
> > > > to
> > > > >> > sec
> > > > >> > > > > > storage
> > > > >> > > > > > > > but
> > > > >> > > > > > > > > > > > > > >> we realized it would make management
> of
> > > > >> > snapshots
> > > > >> > > > > quite
> > > > >> > > > > > > > messy
> > > > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > > >> > > > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> > > Tutkowski
> > > > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > > > <javascript:;>
> > > > >> > > > > <javascript:;>>
> > > > >> > > > > > > > > > > > > > >> wrote:
> > > > >> > > > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > > > >>> Hi,
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> Just to make sure I understand all
> the
> > > > >> > > requirements
> > > > >> > > > > > here:
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 1) This relates only to managed
> > storage
> > > > (1:1
> > > > >> > > > mapping
> > > > >> > > > > > > > between
> > > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> > > volume).
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > > (introduced
> > > > >> in
> > > > >> > > 4.6)
> > > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> > snapshot
> > > on
> > > > >> the
> > > > >> > > SAN,
> > > > >> > > > > and
> > > > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > > > >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> > > > >> snapshot,
> > > > >> > but
> > > > >> > > > to
> > > > >> > > > > > copy
> > > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only)
> to
> > > > NFS.
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken.
> > > It's
> > > > >> the
> > > > >> > > > backup
> > > > >> > > > > to
> > > > >> > > > > > > NFS
> > > > >> > > > > > > > > > > > > > >>> that is optional.
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> > > > >> snapshot
> > > > >> > > > that's
> > > > >> > > > > on
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created
> > > from
> > > > >> the
> > > > >> > > > > snapshot
> > > > >> > > > > > > > > > > > > > >>> that's on
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > >>> SAN (already works as long as the
> new
> > > > >> > CloudStack
> > > > >> > > > > volume
> > > > >> > > > > > > > ends
> > > > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > > > snapshot
> > > > >> API
> > > > >> > > > then
> > > > >> > > > > or
> > > > >> > > > > > > > > > > > > > >>> would that just be the standard
> volume
> > > > >> snapshot
> > > > >> > > > > without
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > > > >>> backup to
> > > > >> > > > > > > > > > > NFS?
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> Thanks!
> > > > >> > > > > > > > > > > > > > >>> Mike
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> > > > Mushtaq
> > > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> > <javascript:;>
> > > > >> > > > > <javascript:;>>
> > > > >> > > > > > > > > > > > > > >>> wrote:
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > > functionalities
> > > > >> > > > > (snapshot
> > > > >> > > > > > on
> > > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally,
> we
> > > > would
> > > > >> > like
> > > > >> > > > to
> > > > >> > > > > > have
> > > > >> > > > > > > > > both.
> > > > >> > > > > > > > > > > > > > >>>> For example, some of our customers
> > want
> > > > to
> > > > >> > > > implement
> > > > >> > > > > > > their
> > > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> > encryption
> > > > to
> > > > >> > their
> > > > >> > > > > > backups
> > > > >> > > > > > > > > > > > > > >>>> which is a perfect use case for
> > Storage
> > > > >> > Snapshot
> > > > >> > > > > while
> > > > >> > > > > > > our
> > > > >> > > > > > > > > > > > > > >>>> other customers will still keep
> using
> > > the
> > > > >> > > standard
> > > > >> > > > > > > volume
> > > > >> > > > > > > > > > > snapshot.
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>> To keep things backward compatible,
> > we
> > > > can
> > > > >> > add a
> > > > >> > > > > > setting
> > > > >> > > > > > > > > > > > > > >>>> which
> > > > >> > > > > > > > > > > > says
> > > > >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> > > > >> because,
> > > > >> > > after
> > > > >> > > > > > all,
> > > > >> > > > > > > > you
> > > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first
> when
> > > > doing
> > > > >> a
> > > > >> > > > Volume
> > > > >> > > > > > > > > Snapshot.
> > > > >> > > > > > > > > > > > > > >>>> You could stop the process there
> and
> > > not
> > > > do
> > > > >> > the
> > > > >> > > > > > upload.
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>> What do you think about this
> > approach?
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > > >> > > > > > > > > > > > > > >>>> -Syed
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM,
> Mike
> > > > >> > Tutkowski <
> > > > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out
> > load
> > > > >> here,
> > > > >> > but
> > > > >> > > > if
> > > > >> > > > > a
> > > > >> > > > > > > > given
> > > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually
> > need
> > > > to
> > > > >> > > provide
> > > > >> > > > > > both
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > ability
> > > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export
> it
> > > to
> > > > >> NFS
> > > > >> > (if
> > > > >> > > > > just
> > > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then
> > we
> > > > >> might
> > > > >> > be
> > > > >> > > > able
> > > > >> > > > > > to
> > > > >> > > > > > > > get
> > > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > > >> > > > > > > > > > > > API
> > > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> > > custom
> > > > >> > > snapshot
> > > > >> > > > > > > strategy
> > > > >> > > > > > > > > > > > > > >>>>> and
> > > > >> > > > > > > > > > > > data
> > > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case
> > > where
> > > > >> the
> > > > >> > > > > > CloudStack
> > > > >> > > > > > > > > > > > > > >>>>> cloud
> > > > >> > > > > > > > > > > > does
> > > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > > >> exported-to-NFS
> > > > >> > > > > backup.
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> In other words, the "default"
> > behavior
> > > > >> would
> > > > >> > be
> > > > >> > > > to
> > > > >> > > > > > use
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> > > > strategy
> > > > >> > that
> > > > >> > > > we
> > > > >> > > > > > > > already
> > > > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a
> SAN
> > > > >> > snapshot).
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however,
> > > wants
> > > > >> to
> > > > >> > > take
> > > > >> > > > a
> > > > >> > > > > > SAN
> > > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data
> exported
> > to
> > > > >> NFS,
> > > > >> > > then
> > > > >> > > > we
> > > > >> > > > > > > could
> > > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config
> > > file
> > > > to
> > > > >> > make
> > > > >> > > > use
> > > > >> > > > > > of
> > > > >> > > > > > > a
> > > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data
> > motion
> > > > >> > strategy
> > > > >> > > > that
> > > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > > >> > > > > > > > > > > > > > activities.
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is
> still
> > > the
> > > > >> > default
> > > > >> > > > for
> > > > >> > > > > > > > users,
> > > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change
> > this
> > > > >> > behavior
> > > > >> > > > via
> > > > >> > > > > > > > > > > configuration.
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM,
> > Mike
> > > > >> > > Tutkowski <
> > > > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to
> > come
> > > up
> > > > >> > with a
> > > > >> > > > > > viable
> > > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable
> way
> > > for
> > > > >> them
> > > > >> > > to
> > > > >> > > > > move
> > > > >> > > > > > > > from
> > > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and
> > some
> > > > >> obvious
> > > > >> > > way
> > > > >> > > > > > that
> > > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > > >> > > > > > > > > > > > to
> > > > >> > > > > > > > > > > > > > do this).
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM,
> > Syed
> > > > >> > Mushtaq
> > > > >> > > <
> > > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > > <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> > > upgrade
> > > > >> path
> > > > >> > > > > however,
> > > > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and
> > are
> > > > on a
> > > > >> > > > managed
> > > > >> > > > > > > > storage
> > > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > > > >> functionality
> > > > >> > > > with
> > > > >> > > > > > > Volume
> > > > >> > > > > > > > > > > Snapshots.
> > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM,
> > Syed
> > > > >> > Mushtaq
> > > > >> > > <
> > > > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > > > <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly,
> > > > currently
> > > > >> > > > taking a
> > > > >> > > > > > > > Volume
> > > > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a
> > managed
> > > > >> storage
> > > > >> > > > keeps
> > > > >> > > > > > it
> > > > >> > > > > > > on
> > > > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part of
> > > this
> > > > >> > > feature,
> > > > >> > > > we
> > > > >> > > > > > can
> > > > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > > > >> > > > > > > > > > > > that
> > > > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed
> > storage
> > > > are
> > > > >> > > > uploaded
> > > > >> > > > > > to
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would
> > make
> > > > the
> > > > >> > > Volume
> > > > >> > > > > > > Snapshot
> > > > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same
> > regardless
> > > of
> > > > >> the
> > > > >> > > > > storage
> > > > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing
> > the
> > > > >> > > efficient
> > > > >> > > > > > > backend
> > > > >> > > > > > > > > > > > > > >>>>>>>> storage
> > > > >> > > > > > > > > > > > > > capabilities, we can use the new storage
> > > > >> snapshots
> > > > >> > > API.
> > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM,
> > > Mike
> > > > >> > > > Tutkowski <
> > > > >> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> > > > >> > <javascript:;>
> > > > >> > > > > <javascript:;>>
> > > > >> > > > > > wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need
> to
> > > > have a
> > > > >> > plan
> > > > >> > > > to
> > > > >> > > > > > deal
> > > > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we already
> > > have a
> > > > >> > > feature
> > > > >> > > > > (in
> > > > >> > > > > > > 4.6
> > > > >> > > > > > > > > > > > > > >>>>>>>>> and
> > > > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to use
> > the
> > > > >> > existing
> > > > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to
> create a
> > > > >> volume
> > > > >> > > > > snapshot
> > > > >> > > > > > > (for
> > > > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > > > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a
> > backend
> > > > SAN
> > > > >> > > > (using a
> > > > >> > > > > > > > custom
> > > > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom
> > > data
> > > > >> > motion
> > > > >> > > > > > > strategy).
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then
> > how
> > > > >> should
> > > > >> > > the
> > > > >> > > > > > > original
> > > > >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6
> > and
> > > > >> later)
> > > > >> > > be
> > > > >> > > > > > > changed?
> > > > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > > > >> > > > > > > > > > > > is
> > > > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support
> > > customers
> > > > >> who
> > > > >> > > are
> > > > >> > > > > > > already
> > > > >> > > > > > > > > > > > > > >>>>>>>>> using
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API
> to
> > > take
> > > > >> > > > snapshots
> > > > >> > > > > > on a
> > > > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > > > >> > > > > > > > > > > > > SAN?
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27
> AM,
> > > > Will
> > > > >> > > > Stevens <
> > > > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > > > <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create a
> > > > Template
> > > > >> > > from a
> > > > >> > > > > > > > > > > > StorageSnapshot?
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template be
> > > stored
> > > > >> in
> > > > >> > the
> > > > >> > > > > > > secondary
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal templates
> > or
> > > > will
> > > > >> > that
> > > > >> > > > be
> > > > >> > > > > > > > handled
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > >> > > > > > > > > > > > > > vendor side?
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud
> Solutions
> > > > >> Experts
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|*
> > > Quebec
> > > > >> *|*
> > > > >> > > H3J
> > > > >> > > > > 1S6
> > > > >> > > > > > w
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw
> @CloudOps_
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22
> PM,
> > > > Syed
> > > > >> > > > Mushtaq <
> > > > >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19
> > PM,
> > > > Will
> > > > >> > > > Stevens
> > > > >> > > > > <
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>> wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the
> > Design
> > > > >> Spec in
> > > > >> > > the
> > > > >> > > > > > Jira
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not
> > clear
> > > > in
> > > > >> the
> > > > >> > > > > > 'mention'
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows
> as a
> > > > page
> > > > >> > 'you
> > > > >> > > do
> > > > >> > > > > not
> > > > >> > > > > > > > have
> > > > >> > > > > > > > > > > permission to'.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud
> > Solutions
> > > > >> Experts
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal
> *|*
> > > > Quebec
> > > > >> > *|*
> > > > >> > > > H3J
> > > > >> > > > > > 1S6
> > > > >> > > > > > > w
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw
> > @CloudOps_
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02
> > PM,
> > > > >> Syed
> > > > >> > > Ahmed
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > >
> > > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new
> set
> > > of
> > > > >> APIs
> > > > >> > to
> > > > >> > > > do
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed
> storage
> > > > >> backends
> > > > >> > > > like
> > > > >> > > > > > > > > SolidFire.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current
> managed
> > > > >> storage
> > > > >> > > stay
> > > > >> > > > > on
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary
> to
> > > what
> > > > >> > > > CloudStack
> > > > >> > > > > > > calls
> > > > >> > > > > > > > > > > snpshots.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on
> > > storage
> > > > >> and
> > > > >> > > > keeping
> > > > >> > > > > > it
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own
> advantages
> > > > >> > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to
> > have
> > > > both
> > > > >> > ways
> > > > >> > > > of
> > > > >> > > > > > > doing
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal
> > adds
> > > 4
> > > > >> new
> > > > >> > > APIs
> > > > >> > > > to
> > > > >> > > > > > > > create
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend
> > storage.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of
> > this
> > > > >> > > feature? I
> > > > >> > > > > > would
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some
> feedback.
> > I
> > > am
> > > > >> > > working
> > > > >> > > > on
> > > > >> > > > > > > > making
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > >> > > > > > > > > > > > spec
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted
> to
> > > > have a
> > > > >> > high
> > > > >> > > > > level
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before
> > starting
> > > > to
> > > > >> > work
> > > > >> > > on
> > > > >> > > > > it.
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> --
> > > > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer,
> > > > >> SolidFire
> > > > >> > > Inc.*
> > > > >> > > > > > > > > > > > > > >>>>>>>>> e:
> mike.tutkowski@solidfire.com
> > > > >> > > > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world
> uses
> > > the
> > > > >> > cloud
> > > > >> > > > > > > > > > > > > > >>>>>>>>> <
> > > > >> > > > > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > > > > >*™
> > > > >> > > > > > > > > > > > > > >>>>>>>>> *
> > > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>> --
> > > > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer,
> > > SolidFire
> > > > >> > Inc.*
> > > > >> > > > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > > >> > > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > >> > > > > > > > > > > > > > >>>>>> Advancing the way the world uses
> > the
> > > > >> cloud
> > > > >> > > > > > > > > > > > > > >>>>>> <
> > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > > > >*™*
> > > > >> > > > > > > > > > > > > > >>>>>>
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>> --
> > > > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer,
> > > SolidFire
> > > > >> Inc.*
> > > > >> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > >> > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > >> > > > > > > > > > > > > > >>>>> Advancing the way the world uses
> the
> > > > cloud
> > > > >> > > > > > > > > > > > > > >>>>> <
> > > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > > > >*™*
> > > > >> > > > > > > > > > > > > > >>>>>
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>>
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>> --
> > > > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer,
> > SolidFire
> > > > >> Inc.*
> > > > >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > > >> > > > > > > > > > > > > > >>> Advancing the way the world uses the
> > > cloud
> > > > >> > > > > > > > > > > > > > >>> <
> > > > >> > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > > >*™*
> > > > >> > > > > > > > > > > > > > >>>
> > > > >> > > > > > > > > > > > > > >>
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > --
> > > > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > > >> > > > > > > > > > > > > > > *Senior CloudStack Developer,
> SolidFire
> > > > Inc.*
> > > > >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > >> <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > > > > o: 303.746.7302
> > > > >> > > > > > > > > > > > > > > Advancing the way the world uses the
> > cloud
> > > > >> > > > > > > > > > > > > > > <
> > > > >> > > http://solidfire.com/solution/overview/?video=play
> > > > >> > > > > >*™*
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Find out more about ShapeBlue and our
> > range
> > > of
> > > > >> > > > CloudStack
> > > > >> > > > > > > > > > > > > > related
> > > > >> > > > > > > > > > > > > services:
> > > > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > >> > > > > > > > > > > > > > <
> > > > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > > >> > > |
> > > > >> > > > > > > > CSForge –
> > > > >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > > > >> > > > > > > http://shapeblue.com/csforge/
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > > > > > CloudStack Consulting
> > > > >> > > > > > > > > > > > > > <
> > > http://shapeblue.com/cloudstack-consultancy/
> > > > >
> > > > >> |
> > > > >> > > > > > > > > > > > > CloudStack
> > > > >> > > > > > > > > > > > > > Software Engineering
> > > > >> > > > > > > > > > > > > > <
> > > > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > > > >> > > > >
> > > > >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > > > >> > > > > > > > > > > > > > <
> > > > >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> > > > >> > > > > >
> > > > >> > > > > > |
> > > > >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > >> > > > > > > > > > > > > > <
> > http://shapeblue.com/cloudstack-training/>
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > --
> > > > >> > > > > > > > > > > > *Mike Tutkowski*
> > > > >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> Inc.*
> > > > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > <javascript:;>
> > > > >> > > > > <javascript:;>
> > > > >> > > > > > > > > > > > o: 303.746.7302
> > > > >> > > > > > > > > > > > Advancing the way the world uses the cloud
> > > > >> > > > > > > > > > > > <
> > > > http://solidfire.com/solution/overview/?video=play
> > > > >> > >*™*
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > Find out more about ShapeBlue and our range of
> > > > >> CloudStack
> > > > >> > > > > related
> > > > >> > > > > > > > > > services:
> > > > >> > > > > > > > > > > IaaS Cloud Design & Build
> > > > >> > > > > > > > > > > <
> > > http://shapeblue.com/iaas-cloud-design-and-build//
> > > > >
> > > > >> |
> > > > >> > > > > CSForge –
> > > > >> > > > > > > > rapid
> > > > >> > > > > > > > > > > IaaS deployment framework <
> > > > >> http://shapeblue.com/csforge/
> > > > >> > >
> > > > >> > > > > > > CloudStack
> > > > >> > > > > > > > > > > Consulting <
> > > > >> http://shapeblue.com/cloudstack-consultancy/
> > > > >> > >
> > > > >> > > |
> > > > >> > > > > > > > CloudStack
> > > > >> > > > > > > > > > > Software Engineering
> > > > >> > > > > > > > > > > <
> > > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > > >> > > > > > > > > > > CloudStack Infrastructure Support
> > > > >> > > > > > > > > > > <
> > > > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > > > >> > >
> > > > >> > > |
> > > > >> > > > > > > > CloudStack
> > > > >> > > > > > > > > > > Bootcamp Training Courses <
> > > > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > Find out more about ShapeBlue and our range of
> > > > >> CloudStack
> > > > >> > > > related
> > > > >> > > > > > > > > services:
> > > > >> > > > > > > > > > IaaS Cloud Design & Build
> > > > >> > > > > > > > > > <
> > http://shapeblue.com/iaas-cloud-design-and-build//
> > > >
> > > > |
> > > > >> > > > CSForge –
> > > > >> > > > > > > rapid
> > > > >> > > > > > > > > > IaaS deployment framework <
> > > > >> http://shapeblue.com/csforge/>
> > > > >> > > > > > > > > > CloudStack Consulting <
> > > > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > >> > > > > > > |
> > > > >> > > > > > > > > CloudStack
> > > > >> > > > > > > > > > Software Engineering
> > > > >> > > > > > > > > > <
> > > > http://shapeblue.com/cloudstack-software-engineering/>
> > > > >> > > > > > > > > > CloudStack Infrastructure Support
> > > > >> > > > > > > > > > <
> > > > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > > >> > |
> > > > >> > > > > > > CloudStack
> > > > >> > > > > > > > > > Bootcamp Training Courses <
> > > > >> > > > > > http://shapeblue.com/cloudstack-training/
> > > > >> > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > --
> > > > >> > > > > > > *Mike Tutkowski*
> > > > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>
> > > > >> > > > > > > o: 303.746.7302
> > > > >> > > > > > > Advancing the way the world uses the cloud
> > > > >> > > > > > > <http://solidfire.com/solution/overview/?video=play
> >*™*
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > --
> > > > >> > > > > *Mike Tutkowski*
> > > > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > >> > > > > o: 303.746.7302
> > > > >> > > > > Advancing the way the world uses the cloud
> > > > >> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > > Ian Rae
> > > > >> > > > CEO | PDG
> > > > >> > > > c: 514.944.4008
> > > > >> > > >
> > > > >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> > > > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J
> 1S6
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > *Mike Tutkowski*
> > > > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > > > >> > e: mike.tutkowski@solidfire.com
> > > > >> > o: 303.746.7302
> > > > >> > Advancing the way the world uses the cloud
> > > > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Will Stevens <ws...@cloudops.com>]

Sorry.  I missed a bit of context when I responded.  The global setting
would be only for the managed storage case, currently being called Storage
Snapshots, and is only to determine if a copy is pushed to secondary
storage right?  The global setting would not change the behavior of the
Volume Snapshots right?

I was referring to the need for Volume Snapshots and Storage Snapshots to
exist together.  Disregard my comment.  I caught up on context after I
posted.  My bad...

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Mon, Feb 8, 2016 at 2:05 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> Hey Will,
>
> Who's picking the behavior? Is it the cloud provider or the end user?
>
> Thanks
>
> On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > I don't think a global setting is a good option because we need both
> > functionality to be available at the same time and for different use
> cases
> > to be able to pick which they choose.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com
> > > wrote:
> >
> > > Now that I re-read your e-mail, it dawned on me: The end-user doesn't
> > care
> > > where the snapshot is.
> > >
> > > If that's true, then we should perhaps control this via Global Settings
> > or
> > > something.
> > >
> > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com> wrote:
> > >
> > > > It's not ideal - true, but it does allow us to be backward
> compatible.
> > > >
> > > > If you have other ideas, though, about how to maintain backward
> > > > compatibility, I'm definitely open to hear them.
> > > >
> > > > Thanks!
> > > >
> > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> syed1.mushtaq@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi Mike,
> > > >>
> > > >> Adding a flag to createSnapshot was the first and the most obvious
> > thing
> > > >> that came to our minds. The problem that I had with this was that:
> > > >>
> > > >> 1) I feel it is exposing something to the end user that is internal
> to
> > > the
> > > >> cloud.
> > > >>
> > > >> 2) We have to follow two different ways of restore/deletion in the
> > same
> > > >> code path depending on where the Snapshot resides which I find kind
> > of a
> > > >> bad design.
> > > >>
> > > >> But if exposing a archive flag to the end user is acceptable then we
> > can
> > > >> definitely use this instead of adding the StorageSnapshot API
> > > >>
> > > >> Thanks,
> > > >> -Syed
> > > >>
> > > >>
> > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > >> mike.tutkowski@solidfire.com
> > > >> > wrote:
> > > >>
> > > >> > Hi Pierre-Luc,
> > > >> >
> > > >> > My recommendation would be this:
> > > >> >
> > > >> > Add an "archive" flag to the current volume-snapshot API. Its
> > default
> > > >> would
> > > >> > be "false" because that would be backward compatible with how 4.6
> > has
> > > >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6,
> 4.7,
> > > and
> > > >> > 4.8).
> > > >> >
> > > >> > If you set archive=true, then we would perform a background
> > migration
> > > of
> > > >> > the snapshot from the SAN to the secondary storage (then delete
> the
> > > SAN
> > > >> > snapshot).
> > > >> >
> > > >> > That archive parameter would only be valid for managed storage.
> > > >> >
> > > >> > Sound reasonable?
> > > >> >
> > > >> > Also, a VM snapshot that includes disks provided by managed
> storage
> > > >> should
> > > >> > work.
> > > >> >
> > > >> > Talk to you later,
> > > >> > Mike
> > > >> >
> > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> pdion@cloudops.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Mike,
> > > >> > >
> > > >> > > In terms of API's, would you prefer introducing a parameter to
> the
> > > >> > existing
> > > >> > > VolumeSnapshot, example:   extract={true|false}  with a default
> > > value
> > > >> of
> > > >> > > true  which would extract snapshot into the secondary storage,
> > which
> > > >> is
> > > >> > the
> > > >> > > current default behavior. Then for SAN snapshot that remain on
> the
> > > >> SAN we
> > > >> > > would just set "extract=false" ?  as oppose to create a new
> > > >> > >  StorageSnapshot API ?
> > > >> > >
> > > >> > >
> > > >> > > Paul,
> > > >> > >
> > > >> > > From what I'm seeing so far, we can't do a VM-snapshot when
> using
> > > >> managed
> > > >> > > storage for VM having more than one Volume. For the reason that
> > > >> snapshot
> > > >> > > are performed outside of the hypervisor awareness and
> > > asynchronously.
> > > >> If
> > > >> > > someone have a way to address that, it would make thinks much
> more
> > > >> > > attractive.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com>
> > wrote:
> > > >> > >
> > > >> > > > I think a service provider backup scenario is more likely to
> > take
> > > >> > > advantage
> > > >> > > > of SAN snapshot. There are a few reasons for this. Traditional
> > > >> backups
> > > >> > > > involve access to the file system, and there is an expectation
> > > that
> > > >> > this
> > > >> > > > can be done with reasonably short time frames without
> negatively
> > > >> > > impacting
> > > >> > > > VM performance, and that the backup orchestrator can apply
> > various
> > > >> > logic
> > > >> > > > and or transformations to the data (compress, encrypt, deltas
> > > >> etc...).
> > > >> > > > While it is true that one could apply a backup process to a
> > cloud
> > > >> > > snapshot,
> > > >> > > > this would be slow and inefficient requiring the data to be
> > moved
> > > >> > several
> > > >> > > > times and there are some major bottlenecks with cloud
> snapshots.
> > > >> With a
> > > >> > > > cloud snapshot - there seems to be no reasonable expectation
> of
> > > >> being
> > > >> > > able
> > > >> > > > to do differential snapshots (I think this depends on the
> > > >> hypervisor)
> > > >> > and
> > > >> > > > if you do differential snapshots this will make file backups
> > from
> > > >> those
> > > >> > > > snapshots even more complicated to orchestrate.
> > > >> > > >
> > > >> > > > Suspect there needs to be a different thread on how to better
> > > enable
> > > >> > > > backups, as a service. As per Paul's suggestion, but it is a
> > > related
> > > >> > > > workflow so relevant to this discussion.
> > > >> > > >
> > > >> > > > Ian
> > > >> > > >
> > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > >> > > mike.tutkowski@solidfire.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > To me it sounds like number two and number three are
> different
> > > >> uses
> > > >> > for
> > > >> > > > the
> > > >> > > > > same "thing"(which is totally fine).
> > > >> > > > >
> > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > >> asynchronously, do
> > > >> > > we
> > > >> > > > > see the SSVM as performing the export?
> > > >> > > > >
> > > >> > > > > To be backwards compatible with what we have in 4.6 and
> later
> > > for
> > > >> > > volume
> > > >> > > > > snapshots for managed storage, I think it might be easier if
> > we
> > > >> pass
> > > >> > > in a
> > > >> > > > > flag that says whether or not to archive the SAN snapshot
> > > (which,
> > > >> I
> > > >> > > > think,
> > > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > > >> > > > >
> > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > pdion@cloudops.com
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > >
> > > >> > > > > > Hi Mike,
> > > >> > > > > >
> > > >> > > > > > The reason behind the creation of a SAN snapshot which is
> > > >> exported
> > > >> > > into
> > > >> > > > > > secondary storage, is because creating a copy of the .VHD
> > > >> directly
> > > >> > > > would
> > > >> > > > > > impact uptime of the VM as creating that copy take lots of
> > > time.
> > > >> > Has
> > > >> > > > > oppose
> > > >> > > > > > to a SAN snapshot that is close to instantaneous which can
> > > >> > afterward
> > > >> > > be
> > > >> > > > > > clone into Secondary Storage asynchronously.
> > > >> > > > > >
> > > >> > > > > > I would suspect an extracted VolumeSnapshot taken from a
> SAN
> > > >> > snapshot
> > > >> > > > > could
> > > >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> > > >> consumption
> > > >> > > on
> > > >> > > > > the
> > > >> > > > > > Primary Storage side.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > I see 3 definitions in our current discussion regarding
> the
> > > term
> > > >> > > > snapshot
> > > >> > > > > > (these are not official terminology but by own
> > interpretation
> > > of
> > > >> > > them):
> > > >> > > > > >
> > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition
> of
> > a
> > > >> > > > snapshot):
> > > >> > > > > > it's a volume snapshot at the storage level, point in time
> > of
> > > >> your
> > > >> > > > data.
> > > >> > > > > it
> > > >> > > > > > reside on the primary storage. Useful and efficient for
> > > software
> > > >> > side
> > > >> > > > > > incident.
> > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/
> cloud
> > > >> backup
> > > >> > > > aws-S3
> > > >> > > > > > style ): Point in time copy of the Virtual Disk that
> reside
> > > on a
> > > >> > > > > different
> > > >> > > > > > storage array then the original Volume. Facilitate data
> > > >> migration
> > > >> > > > between
> > > >> > > > > > clusters and, in case of primary storage incident, Volume
> > > >> snapshots
> > > >> > > are
> > > >> > > > > not
> > > >> > > > > > impacted and can be reuse.
> > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that
> > also
> > > >> > > validate
> > > >> > > > > data
> > > >> > > > > > integrity, provide a storage efficient archiving method
> and
> > an
> > > >> > > > > independent
> > > >> > > > > > way to restore your data in case of an major
> infrastructure
> > > >> > disaster.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Regards,
> > > >> > > > > >
> > > >> > > > > > PL
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> > > >> > > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > So, let's see if I currently follow the requirements:
> > > >> > > > > > >
> > > >> > > > > > > * Augment volume snapshots for managed storage to
> > > >> conditionally
> > > >> > > > export
> > > >> > > > > > data
> > > >> > > > > > > to NFS. The current process of taking a snapshot on the
> > SAN
> > > is
> > > >> > > fine,
> > > >> > > > > but
> > > >> > > > > > > we'd like the option to export the data to NFS, as well.
> > > >> > > > > > >
> > > >> > > > > > > Questions:
> > > >> > > > > > >
> > > >> > > > > > > Once the data has been exported to NFS, do we keep the
> SAN
> > > >> > snapshot
> > > >> > > > or
> > > >> > > > > > > delete it?
> > > >> > > > > > >
> > > >> > > > > > > If we are deleting the SAN snapshot, then why don't we
> > just
> > > >> copy
> > > >> > > the
> > > >> > > > > VHD
> > > >> > > > > > > from primary to secondary the way we do today for
> > > non-managed
> > > >> > (i.e.
> > > >> > > > > > > traditional) storage? Why create a SAN snapshot in that
> > > >> scenario?
> > > >> > > > > Perhaps
> > > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> > secondary
> > > >> > > storage
> > > >> > > > > > > instead of a XenServer host?
> > > >> > > > > > >
> > > >> > > > > > > Thanks for the clarification.
> > > >> > > > > > >
> > > >> > > > > > > By the way, to me a backup is when you copy data from
> one
> > > >> storage
> > > >> > > > > system
> > > >> > > > > > to
> > > >> > > > > > > another (regardless of features, if any, to restore the
> > data
> > > >> in
> > > >> > the
> > > >> > > > > > > future). A snapshot is a point-in-time view of the data
> > of a
> > > >> > volume
> > > >> > > > and
> > > >> > > > > > > it's stored on the same storage system as the volume.
> > > >> > > > > > >
> > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > >> > > pdion@cloudops.com
> > > >> > > > > <javascript:;>
> > > >> > > > > > <javascript:;>>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > That's fun to see that discussion happening. I 100%
> > agree
> > > >> with
> > > >> > > > Paul's
> > > >> > > > > > > > points of view. VolumeSnapshot are not a backup, but I
> > do
> > > >> > > consider
> > > >> > > > > them
> > > >> > > > > > > as
> > > >> > > > > > > > a safety vest against Primary Storage failure, because
> > > >> failure
> > > >> > > > append
> > > >> > > > > > > :-( .
> > > >> > > > > > > >
> > > >> > > > > > > > The current proposal around snapshots that reside on
> the
> > > >> > primary
> > > >> > > > > > storage
> > > >> > > > > > > or
> > > >> > > > > > > > snapshots that end in the Secondary Storage  is not to
> > > >> address
> > > >> > > any
> > > >> > > > > kind
> > > >> > > > > > > of
> > > >> > > > > > > > backups requirement because a snapshot is not a
> backup,
> > > >> event
> > > >> > an
> > > >> > > > > > > extracted
> > > >> > > > > > > > VM snapshot.
> > > >> > > > > > > >
> > > >> > > > > > > > The main idea, and again this is for managed storage;
> > > >> > > > > > > >
> > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> > > >> capability
> > > >> > > for
> > > >> > > > > > fast
> > > >> > > > > > > > response time that support rollback to previous
> > timestamp,
> > > >> > create
> > > >> > > > new
> > > >> > > > > > > > volume and maybe create template.
> > > >> > > > > > > >     not required to be a new API if the work is
> already
> > > >> done, I
> > > >> > > > think
> > > >> > > > > > > this
> > > >> > > > > > > > is a different behaviors than the user expectation of
> a
> > > >> > > > > > volume-snapshot.
> > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> > behavior
> > > >> that
> > > >> > > > create
> > > >> > > > > > an
> > > >> > > > > > > > extraction of a volume into SecondaryStorage which can
> > be
> > > >> reuse
> > > >> > > to
> > > >> > > > > > > create a
> > > >> > > > > > > > new volume into another Primary Storage. This type of
> > > >> snapshot
> > > >> > > is a
> > > >> > > > > > slow
> > > >> > > > > > > > job since yes it would have to copy the full volume
> size
> > > on
> > > >> the
> > > >> > > > > > Secondary
> > > >> > > > > > > > storage.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > PL
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > >> > > > > > <javascript:;>>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > I think I share you view on the 'Ideal world'.
> Backup
> > > (via
> > > >> > > Volume
> > > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This
> is
> > > >> > > amplified
> > > >> > > > > > > > especially
> > > >> > > > > > > > > when you have a object storage as your secondary
> > storage
> > > >> > > because
> > > >> > > > it
> > > >> > > > > > > > > requires two copies (one to an NFS staging area and
> > from
> > > >> > there
> > > >> > > to
> > > >> > > > > > > object
> > > >> > > > > > > > > storage). And not to mention that all these copies
> are
> > > >> > > consuming
> > > >> > > > > > > > hypervisor
> > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> bottleneck
> > as
> > > >> all
> > > >> > > the
> > > >> > > > > > > Network
> > > >> > > > > > > > > and I/O flow through it. So our intention of
> proposing
> > > the
> > > >> > > > "Storage
> > > >> > > > > > > > > Snapshots" is to give a better way of achiving
> > snapshots
> > > >> > while
> > > >> > > > > still
> > > >> > > > > > > > > keeping the original definition of volume snpashots
> > (ie
> > > >> > upload
> > > >> > > to
> > > >> > > > > sec
> > > >> > > > > > > > > storage).
> > > >> > > > > > > > >
> > > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > > backups.
> > > >> > They
> > > >> > > > > don't
> > > >> > > > > > > work
> > > >> > > > > > > > > form multi-disk LVM volume groups and dynamic
> disks. I
> > > am
> > > >> all
> > > >> > > in
> > > >> > > > > for
> > > >> > > > > > a
> > > >> > > > > > > > > better backup solution which handles these use cases
> > and
> > > >> > > utilizes
> > > >> > > > > the
> > > >> > > > > > > > > storage's advanced features.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > >> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > In the beginning... there were CloudStack
> snapshots
> > > and
> > > >> > they
> > > >> > > > were
> > > >> > > > > > > > > actually
> > > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > > snapshots.
> > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > point-in-time
> > > >> > > > > hypervisor
> > > >> > > > > > > > > > snapshots) and we started referring to the
> original
> > > >> > snapshots
> > > >> > > > as
> > > >> > > > > > > volume
> > > >> > > > > > > > > > snapshots.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > CloudStack does not offer 'backups', but many
> people
> > > use
> > > >> > > volume
> > > >> > > > > > > > snapshots
> > > >> > > > > > > > > > as backups. However you can't in-place restore
> > volume
> > > >> > > snapshots
> > > >> > > > > and
> > > >> > > > > > > if
> > > >> > > > > > > > > you
> > > >> > > > > > > > > > have a VM with multiple volumes, the volume
> > snapshots
> > > >> must
> > > >> > be
> > > >> > > > > done
> > > >> > > > > > in
> > > >> > > > > > > > > > series, meaning that the state across all of the
> > > >> volumes is
> > > >> > > > > > unlikely
> > > >> > > > > > > to
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > consistent.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> > > options
> > > >> > > which
> > > >> > > > > > users
> > > >> > > > > > > > > might
> > > >> > > > > > > > > > expect as well options as to where they might be
> > > >> stored. In
> > > >> > > my
> > > >> > > > > > ideal
> > > >> > > > > > > > > world
> > > >> > > > > > > > > > they would also be able to leverage back-end
> > hardware
> > > >> (such
> > > >> > > as
> > > >> > > > > > > > Solidfire,
> > > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> > Commvault
> > > >> etc
> > > >> > to
> > > >> > > > > > > accelerate
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > process.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > >> > > > > > > > > > Paul Angus
> > > >> > > > > > > > > > VP Technology ,  ShapeBlue
> > > >> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > >> > > > > > > > > >
> > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> > > >> > > > > m:
> > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > >> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> > > >> > <javascript:;>
> > > >> > > |
> > > >> > > > > t: @cloudyangus*
> > > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > >> > > > > <ja...@cloudyangus>
> > > >> > > > > > |  w:
> > > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > >> > > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N
> 4HS
> > UK
> > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in
> England
> > &
> > > >> > Wales.
> > > >> > > > > > > ShapeBlue
> > > >> > > > > > > > > > Services India LLP is a company incorporated in
> > India
> > > >> and
> > > >> > is
> > > >> > > > > > operated
> > > >> > > > > > > > > under
> > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > >> Consultoria
> > > >> > > Ltda
> > > >> > > > > is
> > > >> > > > > > a
> > > >> > > > > > > > > > company incorporated in Brasil and is operated
> under
> > > >> > license
> > > >> > > > from
> > > >> > > > > > > Shape
> > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > registered
> > > >> by
> > > >> > The
> > > >> > > > > > > Republic
> > > >> > > > > > > > of
> > > >> > > > > > > > > > South Africa and is traded under license from
> Shape
> > > Blue
> > > >> > Ltd.
> > > >> > > > > > > ShapeBlue
> > > >> > > > > > > > > is
> > > >> > > > > > > > > > a registered trademark.
> > > >> > > > > > > > > > This email and any attachments to it may be
> > > confidential
> > > >> > and
> > > >> > > > are
> > > >> > > > > > > > intended
> > > >> > > > > > > > > > solely for the use of the individual to whom it is
> > > >> > addressed.
> > > >> > > > Any
> > > >> > > > > > > views
> > > >> > > > > > > > > or
> > > >> > > > > > > > > > opinions expressed are solely those of the author
> > and
> > > do
> > > >> > not
> > > >> > > > > > > > necessarily
> > > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > > companies.
> > > >> If
> > > >> > > you
> > > >> > > > > are
> > > >> > > > > > > not
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > intended recipient of this email, you must neither
> > > take
> > > >> any
> > > >> > > > > action
> > > >> > > > > > > > based
> > > >> > > > > > > > > > upon its contents, nor copy or show it to anyone.
> > > Please
> > > >> > > > contact
> > > >> > > > > > the
> > > >> > > > > > > > > sender
> > > >> > > > > > > > > > if you believe you have received this email in
> > error.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > -----Original Message-----
> > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> syed1.mushtaq@gmail.com
> > > >> > > > > <javascript:;> <javascript:;>]
> > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > >> > <javascript:;>
> > > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > Snapshots
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Paul,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > When you say actual backups, how would it be
> > different
> > > >> from
> > > >> > > the
> > > >> > > > > > > Volume
> > > >> > > > > > > > > > Snapshots that exist currently. My understanding
> is
> > > that
> > > >> > > > Backups
> > > >> > > > > > end
> > > >> > > > > > > up
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > point-in-time
> > > >> > state
> > > >> > > of
> > > >> > > > > > your
> > > >> > > > > > > > > volume
> > > >> > > > > > > > > > which can be restored back correct?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > -Syed
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > >> > > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi Syed,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > As I understand it, the SolidFire plugin will
> > export
> > > >> the
> > > >> > > > > snapshot
> > > >> > > > > > > to
> > > >> > > > > > > > > > > secondary storage if the user requests a
> template
> > > from
> > > >> > the
> > > >> > > > > > snapshot
> > > >> > > > > > > > or
> > > >> > > > > > > > > > > wants to download the snapshot from the cloud.
> > This
> > > >> is a
> > > >> > > > good,
> > > >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> > > storage
> > > >> is
> > > >> > > > super
> > > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays take
> up
> > > >> next
> > > >> > to
> > > >> > > no
> > > >> > > > > > > space.
> > > >> > > > > > > > > > > BUT I think that we are talking about a more
> > general
> > > >> > > purpose
> > > >> > > > > API,
> > > >> > > > > > > and
> > > >> > > > > > > > > > > other storage systems may not be as awesome as
> > > Mike's.
> > > >> > > That's
> > > >> > > > > my
> > > >> > > > > > > > > > > concern. Also, the time to transfer for say 1TB
> to
> > > >> move
> > > >> > > from
> > > >> > > > > > > primary
> > > >> > > > > > > > > > > to sec storage and then create a VM template out
> > of
> > > it
> > > >> > may
> > > >> > > be
> > > >> > > > > too
> > > >> > > > > > > > long
> > > >> > > > > > > > > > for users.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > > snapshot
> > > >> for
> > > >> > > > > backup,
> > > >> > > > > > > > it's
> > > >> > > > > > > > > > > just that users want to do backups and a volume
> > > >> snapshot
> > > >> > is
> > > >> > > > the
> > > >> > > > > > > only
> > > >> > > > > > > > > > > type of snapshot that copies the disk elsewhere
> > and
> > > >> can
> > > >> > be
> > > >> > > > > > > scheduled.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> > actual
> > > >> > backups
> > > >> > > > > > > (through
> > > >> > > > > > > > > > > recognised backup providers) and the user
> > > requirements
> > > >> > > around
> > > >> > > > > > them
> > > >> > > > > > > > > > > (particularly restoration use cases) as a
> separate
> > > >> thread
> > > >> > > of
> > > >> > > > > > work.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > Paul
> > > >> Angus
> > > >> > > VP
> > > >> > > > > > > > Technology
> > > >> > > > > > > > > > > , ShapeBlue
> > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > >> > > > > > > > > > >
> > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > >> > > > |
> > > >> > > > > m:
> > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > >> > > <javascript:;> |
> > > >> > > > > t: @cloudyangus*
> > > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > >> <javascript:;>
> > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> 4HS
> > > UK
> > > >> > Shape
> > > >> > > > > Blue
> > > >> > > > > > > Ltd
> > > >> > > > > > > > > > > is a company incorporated in England & Wales.
> > > >> ShapeBlue
> > > >> > > > > Services
> > > >> > > > > > > > India
> > > >> > > > > > > > > > > LLP is a company incorporated in India and is
> > > operated
> > > >> > > under
> > > >> > > > > > > license
> > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> Consultoria
> > > >> Ltda
> > > >> > is
> > > >> > > a
> > > >> > > > > > > company
> > > >> > > > > > > > > > > incorporated in Brasil and is operated under
> > license
> > > >> from
> > > >> > > > Shape
> > > >> > > > > > > Blue
> > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> registered
> > by
> > > >> The
> > > >> > > > > Republic
> > > >> > > > > > > of
> > > >> > > > > > > > > > > South Africa and is traded under license from
> > Shape
> > > >> Blue
> > > >> > > Ltd.
> > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > >> > > > > > > > > > > This email and any attachments to it may be
> > > >> confidential
> > > >> > > and
> > > >> > > > > are
> > > >> > > > > > > > > > > intended solely for the use of the individual to
> > > whom
> > > >> it
> > > >> > is
> > > >> > > > > > > > addressed.
> > > >> > > > > > > > > > > Any views or opinions expressed are solely those
> > of
> > > >> the
> > > >> > > > author
> > > >> > > > > > and
> > > >> > > > > > > do
> > > >> > > > > > > > > > > not necessarily represent those of Shape Blue
> Ltd
> > or
> > > >> > > related
> > > >> > > > > > > > > > > companies. If you are not the intended recipient
> > of
> > > >> this
> > > >> > > > email,
> > > >> > > > > > you
> > > >> > > > > > > > > > > must neither take any action based upon its
> > > contents,
> > > >> nor
> > > >> > > > copy
> > > >> > > > > or
> > > >> > > > > > > > show
> > > >> > > > > > > > > > > it to anyone. Please contact the sender if you
> > > believe
> > > >> > you
> > > >> > > > have
> > > >> > > > > > > > > received
> > > >> > > > > > > > > > this email in error.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > -----Original Message-----
> > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > syed1.mushtaq@gmail.com
> > > >> > > > > <javascript:;>
> > > >> > > > > > <javascript:;>]
> > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > >> > > <javascript:;>
> > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > Snapshots
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I think the terminology confusion comes from AWS
> > > where
> > > >> > they
> > > >> > > > do
> > > >> > > > > > EBS
> > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
> > > >> followed
> > > >> > > > that.
> > > >> > > > > > And
> > > >> > > > > > > > as
> > > >> > > > > > > > > > > an end user who is oblivious to the internals of
> > my
> > > >> > > provider,
> > > >> > > > > my
> > > >> > > > > > > > > > > expectation would be something similar to what
> AWS
> > > as
> > > >> > that
> > > >> > > is
> > > >> > > > > my
> > > >> > > > > > > > > > > biggest reference point.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > To your point Mike, I agree that a Primary
> Storage
> > > >> > failure
> > > >> > > on
> > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> motivations
> > > >> for us
> > > >> > > to
> > > >> > > > > push
> > > >> > > > > > > > data
> > > >> > > > > > > > > > > to secondary storage. Primary storage (atleast
> for
> > > us)
> > > >> > > costs
> > > >> > > > > > > around 3
> > > >> > > > > > > > > > > times as much as secondary storage and snapshots
> > on
> > > >> > primary
> > > >> > > > > > storage
> > > >> > > > > > > > > > > are rarely used (especially for some of our
> > > customers
> > > >> who
> > > >> > > do
> > > >> > > > > > daily
> > > >> > > > > > > > > > backups).
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski
> <
> > > >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > >> > > <javascript:;>>
> > > >> > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Some of the weirdness is around terminology.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > For most systems I've worked on, a snapshot
> and
> > a
> > > >> > backup
> > > >> > > > are
> > > >> > > > > > two
> > > >> > > > > > > > > > > > completely different things (but CloudStack
> has
> > > >> > > > traditionally
> > > >> > > > > > > used
> > > >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
> > > >> though,
> > > >> > > that
> > > >> > > > > if
> > > >> > > > > > > your
> > > >> > > > > > > > > > > > primary storage is running on SolidFire that
> it
> > is
> > > >> > > unlikely
> > > >> > > > > > > you'll
> > > >> > > > > > > > > > > > encounter an issue where your primary storage
> > goes
> > > >> > > offline
> > > >> > > > > (and
> > > >> > > > > > > > > > > > you'll even maintain your performance
> guarantees
> > > >> during
> > > >> > > > > failure
> > > >> > > > > > > > > > > > scenarios and upgrades, as well). That being
> the
> > > >> case,
> > > >> > it
> > > >> > > > is
> > > >> > > > > > less
> > > >> > > > > > > > > > > > useful to require a backup to Swift (but it's
> > > >> perfectly
> > > >> > > OK
> > > >> > > > if
> > > >> > > > > > > > that's
> > > >> > > > > > > > > > > > what we want to do
> > > >> > > > > > > > > > > here).
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > > >> > <javascript:;>>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Paul,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > I believe with the current implementation of
> > > >> > Snapshots
> > > >> > > on
> > > >> > > > > > > managed
> > > >> > > > > > > > > > > > > storage
> > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never exported
> > to
> > > >> the
> > > >> > > > > secondary
> > > >> > > > > > > > > > storage.
> > > >> > > > > > > > > > > > > While this solves the problem of having
> > > snapshots
> > > >> > > taking
> > > >> > > > > > > forever
> > > >> > > > > > > > > > > > > to get to sec storage, this leaves us with a
> > > >> > > > > > > > > > > > huge
> > > >> > > > > > > > > > > > > liability if our primary storage goes down.
> We
> > > see
> > > >> > > > > snapshots
> > > >> > > > > > as
> > > >> > > > > > > > > > > > > our recovery path because we store them in
> > Swift
> > > >> > which
> > > >> > > is
> > > >> > > > > > > > reliable
> > > >> > > > > > > > > > > > > and resilient to failures.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> > > Volume
> > > >> > > > snapshots
> > > >> > > > > > > > always
> > > >> > > > > > > > > > > > > backed up to secondary storage and Storage
> > > >> Snapshots
> > > >> > > stay
> > > >> > > > > on
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > primary
> > > >> > > > > > > > > > > > storage.
> > > >> > > > > > > > > > > > > A provider could potentially mix both these
> > and
> > > >> solve
> > > >> > > the
> > > >> > > > > > > problem
> > > >> > > > > > > > > > > > > that you mentioned where you want to meet
> > user's
> > > >> > > > > expectation
> > > >> > > > > > > of a
> > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while
> > having
> > > >> an
> > > >> > > > ability
> > > >> > > > > > to
> > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the
> device)
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > -Syed
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > >> > > <javascript:;>>
> > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > HI guys,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Could someone point me to the Jira bug of
> FS
> > > for
> > > >> > the
> > > >> > > > > > > > > > > > > > SAN-snapshot
> > > >> > > > > > > > > > > > feature
> > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > From my discussions with users and
> operators
> > > >> around
> > > >> > > > > > snapshots
> > > >> > > > > > > > > > > > > > I'd make
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > following observations:
> > > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
> > > >> long-term
> > > >> > > and
> > > >> > > > > > short
> > > >> > > > > > > > > > > > > > term)
> > > >> > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > the expectation that they can use them for
> > > >> recovery
> > > >> > > if
> > > >> > > > > > > > required.
> > > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > > something
> > > >> > has
> > > >> > > > gone
> > > >> > > > > > > wrong
> > > >> > > > > > > > > > > > > > with primary storage.
> > > >> > > > > > > > > > > > > > c. users sometimes want to be able to
> export
> > > >> > > snapshots
> > > >> > > > as
> > > >> > > > > > > well
> > > >> > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > create
> > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > >> > > > > > > > > > > > > > d. snapshots are a currently a massive
> pain
> > > for
> > > >> > > > > operators,
> > > >> > > > > > I
> > > >> > > > > > > > > > > > > > know at
> > > >> > > > > > > > > > > > > least
> > > >> > > > > > > > > > > > > > one public cloud who have snapshots which
> > > take 2
> > > >> > days
> > > >> > > > to
> > > >> > > > > > > > > complete.
> > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used
> for
> > > >> > multiple
> > > >> > > > LVM
> > > >> > > > > > > > disks.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > I think the process Mike has used in the
> > > >> SolidFire
> > > >> > > > plugin
> > > >> > > > > > > (only
> > > >> > > > > > > > > > > > > > moving
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > disk image to secondary storage when you
> > > >> absolutely
> > > >> > > > have
> > > >> > > > > > to)
> > > >> > > > > > > is
> > > >> > > > > > > > > > > > > > a very
> > > >> > > > > > > > > > > > > good
> > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> > problems
> > > >> an
> > > >> > > > > operator
> > > >> > > > > > > > might
> > > >> > > > > > > > > > > > > experience
> > > >> > > > > > > > > > > > > > if they have an issue with a given primary
> > > >> storage
> > > >> > > pool
> > > >> > > > > in
> > > >> > > > > > a
> > > >> > > > > > > > > > cluster.
> > > >> > > > > > > > > > > > (I
> > > >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> > > >> SolidFire
> > > >> > > case
> > > >> > > > > > Mike
> > > >> > > > > > > :)
> > > >> > > > > > > > > > > > > > ) And
> > > >> > > > > > > > > > > > if
> > > >> > > > > > > > > > > > > > the transfer from primary to secondary is
> > > slow,
> > > >> the
> > > >> > > > time
> > > >> > > > > to
> > > >> > > > > > > > > > > > > > being able
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > create a template or export the volume
> will
> > be
> > > >> > slow.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > So for me the issue is around making sure
> > that
> > > >> the
> > > >> > > end
> > > >> > > > > > users
> > > >> > > > > > > > > > > > expectations
> > > >> > > > > > > > > > > > > > are met (while improving the
> > speed/efficiency
> > > of
> > > >> > the
> > > >> > > > back
> > > >> > > > > > > end)
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > http://www.shapeblue.com>
> > > >> Paul
> > > >> > > > Angus
> > > >> > > > > > VP
> > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> 0540*
> > > >> > > > > > > > > > > > > >
> > > >> > > > > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > >> > > > > > > |
> > > >> > > > > > > > m:
> > > >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> <javascript:;>
> > > >> > > > > <javascript:;> | t:
> > > >> > > > > > @cloudyangus*
> > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > >> > > > <javascript:;>
> > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > http://www.shapeblue.com
> > > >
> > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> > WC2N
> > > >> 4HS
> > > >> > UK
> > > >> > > > > Shape
> > > >> > > > > > > > Blue
> > > >> > > > > > > > > > > > > > Ltd is a company incorporated in England &
> > > >> Wales.
> > > >> > > > > ShapeBlue
> > > >> > > > > > > > > > > > > > Services India LLP is a company
> incorporated
> > > in
> > > >> > India
> > > >> > > > and
> > > >> > > > > > is
> > > >> > > > > > > > > > > > > > operated
> > > >> > > > > > > > > > > > > under
> > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> > Brasil
> > > >> > > > > Consultoria
> > > >> > > > > > > Ltda
> > > >> > > > > > > > > > > > > > is a company incorporated in Brasil and is
> > > >> operated
> > > >> > > > under
> > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA
> > Pty
> > > >> Ltd
> > > >> > is
> > > >> > > a
> > > >> > > > > > > company
> > > >> > > > > > > > > > > > > > registered by The Republic
> > > >> > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > South Africa and is traded under license
> > from
> > > >> Shape
> > > >> > > > Blue
> > > >> > > > > > Ltd.
> > > >> > > > > > > > > > > > > > ShapeBlue
> > > >> > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > a registered trademark.
> > > >> > > > > > > > > > > > > > This email and any attachments to it may
> be
> > > >> > > > confidential
> > > >> > > > > > and
> > > >> > > > > > > > are
> > > >> > > > > > > > > > > > intended
> > > >> > > > > > > > > > > > > > solely for the use of the individual to
> whom
> > > it
> > > >> is
> > > >> > > > > > addressed.
> > > >> > > > > > > > > > > > > > Any views
> > > >> > > > > > > > > > > > > or
> > > >> > > > > > > > > > > > > > opinions expressed are solely those of the
> > > >> author
> > > >> > and
> > > >> > > > do
> > > >> > > > > > not
> > > >> > > > > > > > > > > > necessarily
> > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or
> related
> > > >> > > companies.
> > > >> > > > > If
> > > >> > > > > > > you
> > > >> > > > > > > > > > > > > > are not
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > intended recipient of this email, you must
> > > >> neither
> > > >> > > take
> > > >> > > > > any
> > > >> > > > > > > > > > > > > > action
> > > >> > > > > > > > > > > > based
> > > >> > > > > > > > > > > > > > upon its contents, nor copy or show it to
> > > >> anyone.
> > > >> > > > Please
> > > >> > > > > > > > contact
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > sender
> > > >> > > > > > > > > > > > > > if you believe you have received this
> email
> > in
> > > >> > error.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > -----Original Message-----
> > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > >> pdion@cloudops.com
> > > >> > > > > <javascript:;>
> > > >> > > > > > <javascript:;>]
> > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature]
> Storage
> > > >> > Snapshots
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Hi Mike,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > >> StorageSnapshot
> > > >> > > for
> > > >> > > > > > > managed
> > > >> > > > > > > > > > > > > > storage
> > > >> > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > > expected,
> > > >> > > > behavior
> > > >> > > > > > is
> > > >> > > > > > > to
> > > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> > Storage.
> > > >> So a
> > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > >> > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > snapshot that remain on the managed
> storage
> > > >> > > appliance.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Quickly looking at the API doc and I don't
> > > see a
> > > >> > > strong
> > > >> > > > > > > > > > > > > > requirement for volume snapshots to be
> moved
> > > >> into
> > > >> > > > > secondary
> > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is
> > not
> > > >> > useful,
> > > >> > > > but
> > > >> > > > > > > both
> > > >> > > > > > > > > > > > > > use cases are required. A snapshot that
> > remain
> > > >> on
> > > >> > the
> > > >> > > > > > managed
> > > >> > > > > > > > > > > > > > storage, and another type of
> > > >> > > > > > > > > > > > snapshot
> > > >> > > > > > > > > > > > > > that end up into the secondary storage.
> > Since
> > > >> > you've
> > > >> > > > > done a
> > > >> > > > > > > lot
> > > >> > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > work,
> > > >> > > > > > > > > > > > > > might easier to just add a parameter to
> the
> > > >> current
> > > >> > > > > > snapshot
> > > >> > > > > > > > API
> > > >> > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > trigger an extraction of the storage
> > snapshot
> > > >> into
> > > >> > > the
> > > >> > > > > > > > secondary
> > > >> > > > > > > > > > > > storage?
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > PL
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > > Tutkowski <
> > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > I think that all sounds reasonable then
> -
> > > >> thanks!
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > > Mushtaq <
> > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > >> <javascript:;>>
> > > >> > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> > > >> > requirements.
> > > >> > > > One
> > > >> > > > > > of
> > > >> > > > > > > > our
> > > >> > > > > > > > > > > > earlier
> > > >> > > > > > > > > > > > > > >> iterations on this was to have an
> > argument
> > > to
> > > >> > the
> > > >> > > > > create
> > > >> > > > > > > > > > > > > > >> snapshot
> > > >> > > > > > > > > > > > API
> > > >> > > > > > > > > > > > > > >> which decides whether to backup the
> > volume
> > > to
> > > >> > sec
> > > >> > > > > > storage
> > > >> > > > > > > > but
> > > >> > > > > > > > > > > > > > >> we realized it would make management of
> > > >> > snapshots
> > > >> > > > > quite
> > > >> > > > > > > > messy
> > > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > >> > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> > Tutkowski
> > > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > > <javascript:;>
> > > >> > > > > <javascript:;>>
> > > >> > > > > > > > > > > > > > >> wrote:
> > > >> > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > >>> Hi,
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> Just to make sure I understand all the
> > > >> > > requirements
> > > >> > > > > > here:
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 1) This relates only to managed
> storage
> > > (1:1
> > > >> > > > mapping
> > > >> > > > > > > > between
> > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> > volume).
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > (introduced
> > > >> in
> > > >> > > 4.6)
> > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> snapshot
> > on
> > > >> the
> > > >> > > SAN,
> > > >> > > > > and
> > > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > > >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> > > >> snapshot,
> > > >> > but
> > > >> > > > to
> > > >> > > > > > copy
> > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to
> > > NFS.
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken.
> > It's
> > > >> the
> > > >> > > > backup
> > > >> > > > > to
> > > >> > > > > > > NFS
> > > >> > > > > > > > > > > > > > >>> that is optional.
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> > > >> snapshot
> > > >> > > > that's
> > > >> > > > > on
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created
> > from
> > > >> the
> > > >> > > > > snapshot
> > > >> > > > > > > > > > > > > > >>> that's on
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > >>> SAN (already works as long as the new
> > > >> > CloudStack
> > > >> > > > > volume
> > > >> > > > > > > > ends
> > > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > > snapshot
> > > >> API
> > > >> > > > then
> > > >> > > > > or
> > > >> > > > > > > > > > > > > > >>> would that just be the standard volume
> > > >> snapshot
> > > >> > > > > without
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > > >>> backup to
> > > >> > > > > > > > > > > NFS?
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> Thanks!
> > > >> > > > > > > > > > > > > > >>> Mike
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> > > Mushtaq
> > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> <javascript:;>
> > > >> > > > > <javascript:;>>
> > > >> > > > > > > > > > > > > > >>> wrote:
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > functionalities
> > > >> > > > > (snapshot
> > > >> > > > > > on
> > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we
> > > would
> > > >> > like
> > > >> > > > to
> > > >> > > > > > have
> > > >> > > > > > > > > both.
> > > >> > > > > > > > > > > > > > >>>> For example, some of our customers
> want
> > > to
> > > >> > > > implement
> > > >> > > > > > > their
> > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> encryption
> > > to
> > > >> > their
> > > >> > > > > > backups
> > > >> > > > > > > > > > > > > > >>>> which is a perfect use case for
> Storage
> > > >> > Snapshot
> > > >> > > > > while
> > > >> > > > > > > our
> > > >> > > > > > > > > > > > > > >>>> other customers will still keep using
> > the
> > > >> > > standard
> > > >> > > > > > > volume
> > > >> > > > > > > > > > > snapshot.
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> To keep things backward compatible,
> we
> > > can
> > > >> > add a
> > > >> > > > > > setting
> > > >> > > > > > > > > > > > > > >>>> which
> > > >> > > > > > > > > > > > says
> > > >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> > > >> because,
> > > >> > > after
> > > >> > > > > > all,
> > > >> > > > > > > > you
> > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when
> > > doing
> > > >> a
> > > >> > > > Volume
> > > >> > > > > > > > > Snapshot.
> > > >> > > > > > > > > > > > > > >>>> You could stop the process there and
> > not
> > > do
> > > >> > the
> > > >> > > > > > upload.
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> What do you think about this
> approach?
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > >> > > > > > > > > > > > > > >>>> -Syed
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> > > >> > Tutkowski <
> > > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out
> load
> > > >> here,
> > > >> > but
> > > >> > > > if
> > > >> > > > > a
> > > >> > > > > > > > given
> > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually
> need
> > > to
> > > >> > > provide
> > > >> > > > > > both
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > ability
> > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it
> > to
> > > >> NFS
> > > >> > (if
> > > >> > > > > just
> > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then
> we
> > > >> might
> > > >> > be
> > > >> > > > able
> > > >> > > > > > to
> > > >> > > > > > > > get
> > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > >> > > > > > > > > > > > API
> > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> > custom
> > > >> > > snapshot
> > > >> > > > > > > strategy
> > > >> > > > > > > > > > > > > > >>>>> and
> > > >> > > > > > > > > > > > data
> > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case
> > where
> > > >> the
> > > >> > > > > > CloudStack
> > > >> > > > > > > > > > > > > > >>>>> cloud
> > > >> > > > > > > > > > > > does
> > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > >> exported-to-NFS
> > > >> > > > > backup.
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> In other words, the "default"
> behavior
> > > >> would
> > > >> > be
> > > >> > > > to
> > > >> > > > > > use
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> > > strategy
> > > >> > that
> > > >> > > > we
> > > >> > > > > > > > already
> > > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> > > >> > snapshot).
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however,
> > wants
> > > >> to
> > > >> > > take
> > > >> > > > a
> > > >> > > > > > SAN
> > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data exported
> to
> > > >> NFS,
> > > >> > > then
> > > >> > > > we
> > > >> > > > > > > could
> > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config
> > file
> > > to
> > > >> > make
> > > >> > > > use
> > > >> > > > > > of
> > > >> > > > > > > a
> > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data
> motion
> > > >> > strategy
> > > >> > > > that
> > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > >> > > > > > > > > > > > > > activities.
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is still
> > the
> > > >> > default
> > > >> > > > for
> > > >> > > > > > > > users,
> > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change
> this
> > > >> > behavior
> > > >> > > > via
> > > >> > > > > > > > > > > configuration.
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM,
> Mike
> > > >> > > Tutkowski <
> > > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to
> come
> > up
> > > >> > with a
> > > >> > > > > > viable
> > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way
> > for
> > > >> them
> > > >> > > to
> > > >> > > > > move
> > > >> > > > > > > > from
> > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and
> some
> > > >> obvious
> > > >> > > way
> > > >> > > > > > that
> > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > do this).
> > > >> > > > > > > > > > > > > > >>>>>>
> > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM,
> Syed
> > > >> > Mushtaq
> > > >> > > <
> > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> > upgrade
> > > >> path
> > > >> > > > > however,
> > > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and
> are
> > > on a
> > > >> > > > managed
> > > >> > > > > > > > storage
> > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > > >> functionality
> > > >> > > > with
> > > >> > > > > > > Volume
> > > >> > > > > > > > > > > Snapshots.
> > > >> > > > > > > > > > > > > > >>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM,
> Syed
> > > >> > Mushtaq
> > > >> > > <
> > > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > > <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly,
> > > currently
> > > >> > > > taking a
> > > >> > > > > > > > Volume
> > > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a
> managed
> > > >> storage
> > > >> > > > keeps
> > > >> > > > > > it
> > > >> > > > > > > on
> > > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part of
> > this
> > > >> > > feature,
> > > >> > > > we
> > > >> > > > > > can
> > > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > > >> > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed
> storage
> > > are
> > > >> > > > uploaded
> > > >> > > > > > to
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would
> make
> > > the
> > > >> > > Volume
> > > >> > > > > > > Snapshot
> > > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same
> regardless
> > of
> > > >> the
> > > >> > > > > storage
> > > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing
> the
> > > >> > > efficient
> > > >> > > > > > > backend
> > > >> > > > > > > > > > > > > > >>>>>>>> storage
> > > >> > > > > > > > > > > > > > capabilities, we can use the new storage
> > > >> snapshots
> > > >> > > API.
> > > >> > > > > > > > > > > > > > >>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM,
> > Mike
> > > >> > > > Tutkowski <
> > > >> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> > > >> > <javascript:;>
> > > >> > > > > <javascript:;>>
> > > >> > > > > > wrote:
> > > >> > > > > > > > > > > > > > >>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need to
> > > have a
> > > >> > plan
> > > >> > > > to
> > > >> > > > > > deal
> > > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we already
> > have a
> > > >> > > feature
> > > >> > > > > (in
> > > >> > > > > > > 4.6
> > > >> > > > > > > > > > > > > > >>>>>>>>> and
> > > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to use
> the
> > > >> > existing
> > > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a
> > > >> volume
> > > >> > > > > snapshot
> > > >> > > > > > > (for
> > > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a
> backend
> > > SAN
> > > >> > > > (using a
> > > >> > > > > > > > custom
> > > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom
> > data
> > > >> > motion
> > > >> > > > > > > strategy).
> > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then
> how
> > > >> should
> > > >> > > the
> > > >> > > > > > > original
> > > >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6
> and
> > > >> later)
> > > >> > > be
> > > >> > > > > > > changed?
> > > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > > >> > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support
> > customers
> > > >> who
> > > >> > > are
> > > >> > > > > > > already
> > > >> > > > > > > > > > > > > > >>>>>>>>> using
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API to
> > take
> > > >> > > > snapshots
> > > >> > > > > > on a
> > > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > > >> > > > > > > > > > > > > SAN?
> > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM,
> > > Will
> > > >> > > > Stevens <
> > > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > > <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create a
> > > Template
> > > >> > > from a
> > > >> > > > > > > > > > > > StorageSnapshot?
> > > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template be
> > stored
> > > >> in
> > > >> > the
> > > >> > > > > > > secondary
> > > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal templates
> or
> > > will
> > > >> > that
> > > >> > > > be
> > > >> > > > > > > > handled
> > > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > >> > > > > > > > > > > > > > vendor side?
> > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions
> > > >> Experts
> > > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|*
> > Quebec
> > > >> *|*
> > > >> > > H3J
> > > >> > > > > 1S6
> > > >> > > > > > w
> > > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM,
> > > Syed
> > > >> > > > Mushtaq <
> > > >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19
> PM,
> > > Will
> > > >> > > > Stevens
> > > >> > > > > <
> > > >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the
> Design
> > > >> Spec in
> > > >> > > the
> > > >> > > > > > Jira
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not
> clear
> > > in
> > > >> the
> > > >> > > > > > 'mention'
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows as a
> > > page
> > > >> > 'you
> > > >> > > do
> > > >> > > > > not
> > > >> > > > > > > > have
> > > >> > > > > > > > > > > permission to'.
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud
> Solutions
> > > >> Experts
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|*
> > > Quebec
> > > >> > *|*
> > > >> > > > H3J
> > > >> > > > > > 1S6
> > > >> > > > > > > w
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw
> @CloudOps_
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02
> PM,
> > > >> Syed
> > > >> > > Ahmed
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > >
> > > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set
> > of
> > > >> APIs
> > > >> > to
> > > >> > > > do
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage
> > > >> backends
> > > >> > > > like
> > > >> > > > > > > > > SolidFire.
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed
> > > >> storage
> > > >> > > stay
> > > >> > > > > on
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary to
> > what
> > > >> > > > CloudStack
> > > >> > > > > > > calls
> > > >> > > > > > > > > > > snpshots.
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on
> > storage
> > > >> and
> > > >> > > > keeping
> > > >> > > > > > it
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > >> > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to
> have
> > > both
> > > >> > ways
> > > >> > > > of
> > > >> > > > > > > doing
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal
> adds
> > 4
> > > >> new
> > > >> > > APIs
> > > >> > > > to
> > > >> > > > > > > > create
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend
> storage.
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of
> this
> > > >> > > feature? I
> > > >> > > > > > would
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some feedback.
> I
> > am
> > > >> > > working
> > > >> > > > on
> > > >> > > > > > > > making
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > >> > > > > > > > > > > > spec
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to
> > > have a
> > > >> > high
> > > >> > > > > level
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before
> starting
> > > to
> > > >> > work
> > > >> > > on
> > > >> > > > > it.
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>> --
> > > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer,
> > > >> SolidFire
> > > >> > > Inc.*
> > > >> > > > > > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > >> > > > <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world uses
> > the
> > > >> > cloud
> > > >> > > > > > > > > > > > > > >>>>>>>>> <
> > > >> > > > > > > http://solidfire.com/solution/overview/?video=play
> > > >> > > > > > > > >*™
> > > >> > > > > > > > > > > > > > >>>>>>>>> *
> > > >> > > > > > > > > > > > > > >>>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>
> > > >> > > > > > > > > > > > > > >>>>>> --
> > > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer,
> > SolidFire
> > > >> > Inc.*
> > > >> > > > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > >> > > <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > >> > > > > > > > > > > > > > >>>>>> Advancing the way the world uses
> the
> > > >> cloud
> > > >> > > > > > > > > > > > > > >>>>>> <
> > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > >> > > > > > > >*™*
> > > >> > > > > > > > > > > > > > >>>>>>
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> --
> > > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer,
> > SolidFire
> > > >> Inc.*
> > > >> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > >> > <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > >> > > > > > > > > > > > > > >>>>> Advancing the way the world uses the
> > > cloud
> > > >> > > > > > > > > > > > > > >>>>> <
> > > >> > > > > http://solidfire.com/solution/overview/?video=play
> > > >> > > > > > > >*™*
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> --
> > > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer,
> SolidFire
> > > >> Inc.*
> > > >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > >> > > > > > > > > > > > > > >>> Advancing the way the world uses the
> > cloud
> > > >> > > > > > > > > > > > > > >>> <
> > > >> > > > http://solidfire.com/solution/overview/?video=play
> > > >> > > > > > >*™*
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > --
> > > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > >> > > > > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> > > Inc.*
> > > >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > > o: 303.746.7302
> > > >> > > > > > > > > > > > > > > Advancing the way the world uses the
> cloud
> > > >> > > > > > > > > > > > > > > <
> > > >> > > http://solidfire.com/solution/overview/?video=play
> > > >> > > > > >*™*
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Find out more about ShapeBlue and our
> range
> > of
> > > >> > > > CloudStack
> > > >> > > > > > > > > > > > > > related
> > > >> > > > > > > > > > > > > services:
> > > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > >> > > > > > > > > > > > > > <
> > > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > >> > > |
> > > >> > > > > > > > CSForge –
> > > >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > > >> > > > > > > http://shapeblue.com/csforge/
> > > >> > > > > > > > >
> > > >> > > > > > > > > > > > > > CloudStack Consulting
> > > >> > > > > > > > > > > > > > <
> > http://shapeblue.com/cloudstack-consultancy/
> > > >
> > > >> |
> > > >> > > > > > > > > > > > > CloudStack
> > > >> > > > > > > > > > > > > > Software Engineering
> > > >> > > > > > > > > > > > > > <
> > > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > > >> > > > >
> > > >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > > >> > > > > > > > > > > > > > <
> > > >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> > > >> > > > > >
> > > >> > > > > > |
> > > >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > >> > > > > > > > > > > > > > <
> http://shapeblue.com/cloudstack-training/>
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > --
> > > >> > > > > > > > > > > > *Mike Tutkowski*
> > > >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > o: 303.746.7302
> > > >> > > > > > > > > > > > Advancing the way the world uses the cloud
> > > >> > > > > > > > > > > > <
> > > http://solidfire.com/solution/overview/?video=play
> > > >> > >*™*
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > Find out more about ShapeBlue and our range of
> > > >> CloudStack
> > > >> > > > > related
> > > >> > > > > > > > > > services:
> > > >> > > > > > > > > > > IaaS Cloud Design & Build
> > > >> > > > > > > > > > > <
> > http://shapeblue.com/iaas-cloud-design-and-build//
> > > >
> > > >> |
> > > >> > > > > CSForge –
> > > >> > > > > > > > rapid
> > > >> > > > > > > > > > > IaaS deployment framework <
> > > >> http://shapeblue.com/csforge/
> > > >> > >
> > > >> > > > > > > CloudStack
> > > >> > > > > > > > > > > Consulting <
> > > >> http://shapeblue.com/cloudstack-consultancy/
> > > >> > >
> > > >> > > |
> > > >> > > > > > > > CloudStack
> > > >> > > > > > > > > > > Software Engineering
> > > >> > > > > > > > > > > <
> > > >> http://shapeblue.com/cloudstack-software-engineering/>
> > > >> > > > > > > > > > > CloudStack Infrastructure Support
> > > >> > > > > > > > > > > <
> > > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > > >> > >
> > > >> > > |
> > > >> > > > > > > > CloudStack
> > > >> > > > > > > > > > > Bootcamp Training Courses <
> > > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > > >> > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > Find out more about ShapeBlue and our range of
> > > >> CloudStack
> > > >> > > > related
> > > >> > > > > > > > > services:
> > > >> > > > > > > > > > IaaS Cloud Design & Build
> > > >> > > > > > > > > > <
> http://shapeblue.com/iaas-cloud-design-and-build//
> > >
> > > |
> > > >> > > > CSForge –
> > > >> > > > > > > rapid
> > > >> > > > > > > > > > IaaS deployment framework <
> > > >> http://shapeblue.com/csforge/>
> > > >> > > > > > > > > > CloudStack Consulting <
> > > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > >> > > > > > > |
> > > >> > > > > > > > > CloudStack
> > > >> > > > > > > > > > Software Engineering
> > > >> > > > > > > > > > <
> > > http://shapeblue.com/cloudstack-software-engineering/>
> > > >> > > > > > > > > > CloudStack Infrastructure Support
> > > >> > > > > > > > > > <
> > > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > >> > |
> > > >> > > > > > > CloudStack
> > > >> > > > > > > > > > Bootcamp Training Courses <
> > > >> > > > > > http://shapeblue.com/cloudstack-training/
> > > >> > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > --
> > > >> > > > > > > *Mike Tutkowski*
> > > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>
> > > >> > > > > > > o: 303.746.7302
> > > >> > > > > > > Advancing the way the world uses the cloud
> > > >> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > > *Mike Tutkowski*
> > > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > >> > > > > o: 303.746.7302
> > > >> > > > > Advancing the way the world uses the cloud
> > > >> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >> > > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > Ian Rae
> > > >> > > > CEO | PDG
> > > >> > > > c: 514.944.4008
> > > >> > > >
> > > >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> > > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > *Mike Tutkowski*
> > > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > > >> > e: mike.tutkowski@solidfire.com
> > > >> > o: 303.746.7302
> > > >> > Advancing the way the world uses the cloud
> > > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Hey Will,

Who's picking the behavior? Is it the cloud provider or the end user?

Thanks

On Mon, Feb 8, 2016 at 11:52 AM, Will Stevens <ws...@cloudops.com> wrote:

> I don't think a global setting is a good option because we need both
> functionality to be available at the same time and for different use cases
> to be able to pick which they choose.
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com
> > wrote:
>
> > Now that I re-read your e-mail, it dawned on me: The end-user doesn't
> care
> > where the snapshot is.
> >
> > If that's true, then we should perhaps control this via Global Settings
> or
> > something.
> >
> > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com> wrote:
> >
> > > It's not ideal - true, but it does allow us to be backward compatible.
> > >
> > > If you have other ideas, though, about how to maintain backward
> > > compatibility, I'm definitely open to hear them.
> > >
> > > Thanks!
> > >
> > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <syed1.mushtaq@gmail.com
> >
> > > wrote:
> > >
> > >> Hi Mike,
> > >>
> > >> Adding a flag to createSnapshot was the first and the most obvious
> thing
> > >> that came to our minds. The problem that I had with this was that:
> > >>
> > >> 1) I feel it is exposing something to the end user that is internal to
> > the
> > >> cloud.
> > >>
> > >> 2) We have to follow two different ways of restore/deletion in the
> same
> > >> code path depending on where the Snapshot resides which I find kind
> of a
> > >> bad design.
> > >>
> > >> But if exposing a archive flag to the end user is acceptable then we
> can
> > >> definitely use this instead of adding the StorageSnapshot API
> > >>
> > >> Thanks,
> > >> -Syed
> > >>
> > >>
> > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > >> mike.tutkowski@solidfire.com
> > >> > wrote:
> > >>
> > >> > Hi Pierre-Luc,
> > >> >
> > >> > My recommendation would be this:
> > >> >
> > >> > Add an "archive" flag to the current volume-snapshot API. Its
> default
> > >> would
> > >> > be "false" because that would be backward compatible with how 4.6
> has
> > >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6, 4.7,
> > and
> > >> > 4.8).
> > >> >
> > >> > If you set archive=true, then we would perform a background
> migration
> > of
> > >> > the snapshot from the SAN to the secondary storage (then delete the
> > SAN
> > >> > snapshot).
> > >> >
> > >> > That archive parameter would only be valid for managed storage.
> > >> >
> > >> > Sound reasonable?
> > >> >
> > >> > Also, a VM snapshot that includes disks provided by managed storage
> > >> should
> > >> > work.
> > >> >
> > >> > Talk to you later,
> > >> > Mike
> > >> >
> > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <pdion@cloudops.com
> >
> > >> > wrote:
> > >> >
> > >> > > Mike,
> > >> > >
> > >> > > In terms of API's, would you prefer introducing a parameter to the
> > >> > existing
> > >> > > VolumeSnapshot, example:   extract={true|false}  with a default
> > value
> > >> of
> > >> > > true  which would extract snapshot into the secondary storage,
> which
> > >> is
> > >> > the
> > >> > > current default behavior. Then for SAN snapshot that remain on the
> > >> SAN we
> > >> > > would just set "extract=false" ?  as oppose to create a new
> > >> > >  StorageSnapshot API ?
> > >> > >
> > >> > >
> > >> > > Paul,
> > >> > >
> > >> > > From what I'm seeing so far, we can't do a VM-snapshot when using
> > >> managed
> > >> > > storage for VM having more than one Volume. For the reason that
> > >> snapshot
> > >> > > are performed outside of the hypervisor awareness and
> > asynchronously.
> > >> If
> > >> > > someone have a way to address that, it would make thinks much more
> > >> > > attractive.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com>
> wrote:
> > >> > >
> > >> > > > I think a service provider backup scenario is more likely to
> take
> > >> > > advantage
> > >> > > > of SAN snapshot. There are a few reasons for this. Traditional
> > >> backups
> > >> > > > involve access to the file system, and there is an expectation
> > that
> > >> > this
> > >> > > > can be done with reasonably short time frames without negatively
> > >> > > impacting
> > >> > > > VM performance, and that the backup orchestrator can apply
> various
> > >> > logic
> > >> > > > and or transformations to the data (compress, encrypt, deltas
> > >> etc...).
> > >> > > > While it is true that one could apply a backup process to a
> cloud
> > >> > > snapshot,
> > >> > > > this would be slow and inefficient requiring the data to be
> moved
> > >> > several
> > >> > > > times and there are some major bottlenecks with cloud snapshots.
> > >> With a
> > >> > > > cloud snapshot - there seems to be no reasonable expectation of
> > >> being
> > >> > > able
> > >> > > > to do differential snapshots (I think this depends on the
> > >> hypervisor)
> > >> > and
> > >> > > > if you do differential snapshots this will make file backups
> from
> > >> those
> > >> > > > snapshots even more complicated to orchestrate.
> > >> > > >
> > >> > > > Suspect there needs to be a different thread on how to better
> > enable
> > >> > > > backups, as a service. As per Paul's suggestion, but it is a
> > related
> > >> > > > workflow so relevant to this discussion.
> > >> > > >
> > >> > > > Ian
> > >> > > >
> > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > >> > > mike.tutkowski@solidfire.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > To me it sounds like number two and number three are different
> > >> uses
> > >> > for
> > >> > > > the
> > >> > > > > same "thing"(which is totally fine).
> > >> > > > >
> > >> > > > > As for taking a fast SAN snapshot and exporting it
> > >> asynchronously, do
> > >> > > we
> > >> > > > > see the SSVM as performing the export?
> > >> > > > >
> > >> > > > > To be backwards compatible with what we have in 4.6 and later
> > for
> > >> > > volume
> > >> > > > > snapshots for managed storage, I think it might be easier if
> we
> > >> pass
> > >> > > in a
> > >> > > > > flag that says whether or not to archive the SAN snapshot
> > (which,
> > >> I
> > >> > > > think,
> > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > >> > > > >
> > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > pdion@cloudops.com
> > >> > > > > <javascript:;>> wrote:
> > >> > > > >
> > >> > > > > > Hi Mike,
> > >> > > > > >
> > >> > > > > > The reason behind the creation of a SAN snapshot which is
> > >> exported
> > >> > > into
> > >> > > > > > secondary storage, is because creating a copy of the .VHD
> > >> directly
> > >> > > > would
> > >> > > > > > impact uptime of the VM as creating that copy take lots of
> > time.
> > >> > Has
> > >> > > > > oppose
> > >> > > > > > to a SAN snapshot that is close to instantaneous which can
> > >> > afterward
> > >> > > be
> > >> > > > > > clone into Secondary Storage asynchronously.
> > >> > > > > >
> > >> > > > > > I would suspect an extracted VolumeSnapshot taken from a SAN
> > >> > snapshot
> > >> > > > > could
> > >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> > >> consumption
> > >> > > on
> > >> > > > > the
> > >> > > > > > Primary Storage side.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > I see 3 definitions in our current discussion regarding the
> > term
> > >> > > > snapshot
> > >> > > > > > (these are not official terminology but by own
> interpretation
> > of
> > >> > > them):
> > >> > > > > >
> > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of
> a
> > >> > > > snapshot):
> > >> > > > > > it's a volume snapshot at the storage level, point in time
> of
> > >> your
> > >> > > > data.
> > >> > > > > it
> > >> > > > > > reside on the primary storage. Useful and efficient for
> > software
> > >> > side
> > >> > > > > > incident.
> > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud
> > >> backup
> > >> > > > aws-S3
> > >> > > > > > style ): Point in time copy of the Virtual Disk that reside
> > on a
> > >> > > > > different
> > >> > > > > > storage array then the original Volume. Facilitate data
> > >> migration
> > >> > > > between
> > >> > > > > > clusters and, in case of primary storage incident, Volume
> > >> snapshots
> > >> > > are
> > >> > > > > not
> > >> > > > > > impacted and can be reuse.
> > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that
> also
> > >> > > validate
> > >> > > > > data
> > >> > > > > > integrity, provide a storage efficient archiving method and
> an
> > >> > > > > independent
> > >> > > > > > way to restore your data in case of an major infrastructure
> > >> > disaster.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Regards,
> > >> > > > > >
> > >> > > > > > PL
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > >> > > > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > >> > > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > So, let's see if I currently follow the requirements:
> > >> > > > > > >
> > >> > > > > > > * Augment volume snapshots for managed storage to
> > >> conditionally
> > >> > > > export
> > >> > > > > > data
> > >> > > > > > > to NFS. The current process of taking a snapshot on the
> SAN
> > is
> > >> > > fine,
> > >> > > > > but
> > >> > > > > > > we'd like the option to export the data to NFS, as well.
> > >> > > > > > >
> > >> > > > > > > Questions:
> > >> > > > > > >
> > >> > > > > > > Once the data has been exported to NFS, do we keep the SAN
> > >> > snapshot
> > >> > > > or
> > >> > > > > > > delete it?
> > >> > > > > > >
> > >> > > > > > > If we are deleting the SAN snapshot, then why don't we
> just
> > >> copy
> > >> > > the
> > >> > > > > VHD
> > >> > > > > > > from primary to secondary the way we do today for
> > non-managed
> > >> > (i.e.
> > >> > > > > > > traditional) storage? Why create a SAN snapshot in that
> > >> scenario?
> > >> > > > > Perhaps
> > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> secondary
> > >> > > storage
> > >> > > > > > > instead of a XenServer host?
> > >> > > > > > >
> > >> > > > > > > Thanks for the clarification.
> > >> > > > > > >
> > >> > > > > > > By the way, to me a backup is when you copy data from one
> > >> storage
> > >> > > > > system
> > >> > > > > > to
> > >> > > > > > > another (regardless of features, if any, to restore the
> data
> > >> in
> > >> > the
> > >> > > > > > > future). A snapshot is a point-in-time view of the data
> of a
> > >> > volume
> > >> > > > and
> > >> > > > > > > it's stored on the same storage system as the volume.
> > >> > > > > > >
> > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > >> > > pdion@cloudops.com
> > >> > > > > <javascript:;>
> > >> > > > > > <javascript:;>>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > That's fun to see that discussion happening. I 100%
> agree
> > >> with
> > >> > > > Paul's
> > >> > > > > > > > points of view. VolumeSnapshot are not a backup, but I
> do
> > >> > > consider
> > >> > > > > them
> > >> > > > > > > as
> > >> > > > > > > > a safety vest against Primary Storage failure, because
> > >> failure
> > >> > > > append
> > >> > > > > > > :-( .
> > >> > > > > > > >
> > >> > > > > > > > The current proposal around snapshots that reside on the
> > >> > primary
> > >> > > > > > storage
> > >> > > > > > > or
> > >> > > > > > > > snapshots that end in the Secondary Storage  is not to
> > >> address
> > >> > > any
> > >> > > > > kind
> > >> > > > > > > of
> > >> > > > > > > > backups requirement because a snapshot is not a backup,
> > >> event
> > >> > an
> > >> > > > > > > extracted
> > >> > > > > > > > VM snapshot.
> > >> > > > > > > >
> > >> > > > > > > > The main idea, and again this is for managed storage;
> > >> > > > > > > >
> > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> > >> capability
> > >> > > for
> > >> > > > > > fast
> > >> > > > > > > > response time that support rollback to previous
> timestamp,
> > >> > create
> > >> > > > new
> > >> > > > > > > > volume and maybe create template.
> > >> > > > > > > >     not required to be a new API if the work is already
> > >> done, I
> > >> > > > think
> > >> > > > > > > this
> > >> > > > > > > > is a different behaviors than the user expectation of a
> > >> > > > > > volume-snapshot.
> > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> behavior
> > >> that
> > >> > > > create
> > >> > > > > > an
> > >> > > > > > > > extraction of a volume into SecondaryStorage which can
> be
> > >> reuse
> > >> > > to
> > >> > > > > > > create a
> > >> > > > > > > > new volume into another Primary Storage. This type of
> > >> snapshot
> > >> > > is a
> > >> > > > > > slow
> > >> > > > > > > > job since yes it would have to copy the full volume size
> > on
> > >> the
> > >> > > > > > Secondary
> > >> > > > > > > > storage.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > PL
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > >> > > > > > <javascript:;>>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > I think I share you view on the 'Ideal world'. Backup
> > (via
> > >> > > Volume
> > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is
> > >> > > amplified
> > >> > > > > > > > especially
> > >> > > > > > > > > when you have a object storage as your secondary
> storage
> > >> > > because
> > >> > > > it
> > >> > > > > > > > > requires two copies (one to an NFS staging area and
> from
> > >> > there
> > >> > > to
> > >> > > > > > > object
> > >> > > > > > > > > storage). And not to mention that all these copies are
> > >> > > consuming
> > >> > > > > > > > hypervisor
> > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge bottleneck
> as
> > >> all
> > >> > > the
> > >> > > > > > > Network
> > >> > > > > > > > > and I/O flow through it. So our intention of proposing
> > the
> > >> > > > "Storage
> > >> > > > > > > > > Snapshots" is to give a better way of achiving
> snapshots
> > >> > while
> > >> > > > > still
> > >> > > > > > > > > keeping the original definition of volume snpashots
> (ie
> > >> > upload
> > >> > > to
> > >> > > > > sec
> > >> > > > > > > > > storage).
> > >> > > > > > > > >
> > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > backups.
> > >> > They
> > >> > > > > don't
> > >> > > > > > > work
> > >> > > > > > > > > form multi-disk LVM volume groups and dynamic disks. I
> > am
> > >> all
> > >> > > in
> > >> > > > > for
> > >> > > > > > a
> > >> > > > > > > > > better backup solution which handles these use cases
> and
> > >> > > utilizes
> > >> > > > > the
> > >> > > > > > > > > storage's advanced features.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > >> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > In the beginning... there were CloudStack snapshots
> > and
> > >> > they
> > >> > > > were
> > >> > > > > > > > > actually
> > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > snapshots.
> > >> > > > > > > > > > Then VM snapshots were created (which are
> > point-in-time
> > >> > > > > hypervisor
> > >> > > > > > > > > > snapshots) and we started referring to the original
> > >> > snapshots
> > >> > > > as
> > >> > > > > > > volume
> > >> > > > > > > > > > snapshots.
> > >> > > > > > > > > >
> > >> > > > > > > > > > CloudStack does not offer 'backups', but many people
> > use
> > >> > > volume
> > >> > > > > > > > snapshots
> > >> > > > > > > > > > as backups. However you can't in-place restore
> volume
> > >> > > snapshots
> > >> > > > > and
> > >> > > > > > > if
> > >> > > > > > > > > you
> > >> > > > > > > > > > have a VM with multiple volumes, the volume
> snapshots
> > >> must
> > >> > be
> > >> > > > > done
> > >> > > > > > in
> > >> > > > > > > > > > series, meaning that the state across all of the
> > >> volumes is
> > >> > > > > > unlikely
> > >> > > > > > > to
> > >> > > > > > > > > be
> > >> > > > > > > > > > consistent.
> > >> > > > > > > > > >
> > >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> > options
> > >> > > which
> > >> > > > > > users
> > >> > > > > > > > > might
> > >> > > > > > > > > > expect as well options as to where they might be
> > >> stored. In
> > >> > > my
> > >> > > > > > ideal
> > >> > > > > > > > > world
> > >> > > > > > > > > > they would also be able to leverage back-end
> hardware
> > >> (such
> > >> > > as
> > >> > > > > > > > Solidfire,
> > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> Commvault
> > >> etc
> > >> > to
> > >> > > > > > > accelerate
> > >> > > > > > > > > the
> > >> > > > > > > > > > process.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > >> > > > > > > > > > Paul Angus
> > >> > > > > > > > > > VP Technology ,  ShapeBlue
> > >> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > >> > > > > > > > > >
> > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> > >> > > > > m:
> > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > >> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> > >> > <javascript:;>
> > >> > > |
> > >> > > > > t: @cloudyangus*
> > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > >> > > > > <ja...@cloudyangus>
> > >> > > > > > |  w:
> > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > >> > > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS
> UK
> > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in England
> &
> > >> > Wales.
> > >> > > > > > > ShapeBlue
> > >> > > > > > > > > > Services India LLP is a company incorporated in
> India
> > >> and
> > >> > is
> > >> > > > > > operated
> > >> > > > > > > > > under
> > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > >> Consultoria
> > >> > > Ltda
> > >> > > > > is
> > >> > > > > > a
> > >> > > > > > > > > > company incorporated in Brasil and is operated under
> > >> > license
> > >> > > > from
> > >> > > > > > > Shape
> > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> registered
> > >> by
> > >> > The
> > >> > > > > > > Republic
> > >> > > > > > > > of
> > >> > > > > > > > > > South Africa and is traded under license from Shape
> > Blue
> > >> > Ltd.
> > >> > > > > > > ShapeBlue
> > >> > > > > > > > > is
> > >> > > > > > > > > > a registered trademark.
> > >> > > > > > > > > > This email and any attachments to it may be
> > confidential
> > >> > and
> > >> > > > are
> > >> > > > > > > > intended
> > >> > > > > > > > > > solely for the use of the individual to whom it is
> > >> > addressed.
> > >> > > > Any
> > >> > > > > > > views
> > >> > > > > > > > > or
> > >> > > > > > > > > > opinions expressed are solely those of the author
> and
> > do
> > >> > not
> > >> > > > > > > > necessarily
> > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > companies.
> > >> If
> > >> > > you
> > >> > > > > are
> > >> > > > > > > not
> > >> > > > > > > > > the
> > >> > > > > > > > > > intended recipient of this email, you must neither
> > take
> > >> any
> > >> > > > > action
> > >> > > > > > > > based
> > >> > > > > > > > > > upon its contents, nor copy or show it to anyone.
> > Please
> > >> > > > contact
> > >> > > > > > the
> > >> > > > > > > > > sender
> > >> > > > > > > > > > if you believe you have received this email in
> error.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > -----Original Message-----
> > >> > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > >> > > > > <javascript:;> <javascript:;>]
> > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > >> > <javascript:;>
> > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> Snapshots
> > >> > > > > > > > > >
> > >> > > > > > > > > > Paul,
> > >> > > > > > > > > >
> > >> > > > > > > > > > When you say actual backups, how would it be
> different
> > >> from
> > >> > > the
> > >> > > > > > > Volume
> > >> > > > > > > > > > Snapshots that exist currently. My understanding is
> > that
> > >> > > > Backups
> > >> > > > > > end
> > >> > > > > > > up
> > >> > > > > > > > > in
> > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> point-in-time
> > >> > state
> > >> > > of
> > >> > > > > > your
> > >> > > > > > > > > volume
> > >> > > > > > > > > > which can be restored back correct?
> > >> > > > > > > > > >
> > >> > > > > > > > > > -Syed
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > >> > > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Syed,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > As I understand it, the SolidFire plugin will
> export
> > >> the
> > >> > > > > snapshot
> > >> > > > > > > to
> > >> > > > > > > > > > > secondary storage if the user requests a template
> > from
> > >> > the
> > >> > > > > > snapshot
> > >> > > > > > > > or
> > >> > > > > > > > > > > wants to download the snapshot from the cloud.
> This
> > >> is a
> > >> > > > good,
> > >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> > storage
> > >> is
> > >> > > > super
> > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays take up
> > >> next
> > >> > to
> > >> > > no
> > >> > > > > > > space.
> > >> > > > > > > > > > > BUT I think that we are talking about a more
> general
> > >> > > purpose
> > >> > > > > API,
> > >> > > > > > > and
> > >> > > > > > > > > > > other storage systems may not be as awesome as
> > Mike's.
> > >> > > That's
> > >> > > > > my
> > >> > > > > > > > > > > concern. Also, the time to transfer for say 1TB to
> > >> move
> > >> > > from
> > >> > > > > > > primary
> > >> > > > > > > > > > > to sec storage and then create a VM template out
> of
> > it
> > >> > may
> > >> > > be
> > >> > > > > too
> > >> > > > > > > > long
> > >> > > > > > > > > > for users.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > snapshot
> > >> for
> > >> > > > > backup,
> > >> > > > > > > > it's
> > >> > > > > > > > > > > just that users want to do backups and a volume
> > >> snapshot
> > >> > is
> > >> > > > the
> > >> > > > > > > only
> > >> > > > > > > > > > > type of snapshot that copies the disk elsewhere
> and
> > >> can
> > >> > be
> > >> > > > > > > scheduled.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> actual
> > >> > backups
> > >> > > > > > > (through
> > >> > > > > > > > > > > recognised backup providers) and the user
> > requirements
> > >> > > around
> > >> > > > > > them
> > >> > > > > > > > > > > (particularly restoration use cases) as a separate
> > >> thread
> > >> > > of
> > >> > > > > > work.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> Paul
> > >> Angus
> > >> > > VP
> > >> > > > > > > > Technology
> > >> > > > > > > > > > > , ShapeBlue
> > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > >> > > > > > > > > > >
> > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > >> > > > |
> > >> > > > > m:
> > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > >> > > <javascript:;> |
> > >> > > > > t: @cloudyangus*
> > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > >> <javascript:;>
> > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS
> > UK
> > >> > Shape
> > >> > > > > Blue
> > >> > > > > > > Ltd
> > >> > > > > > > > > > > is a company incorporated in England & Wales.
> > >> ShapeBlue
> > >> > > > > Services
> > >> > > > > > > > India
> > >> > > > > > > > > > > LLP is a company incorporated in India and is
> > operated
> > >> > > under
> > >> > > > > > > license
> > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria
> > >> Ltda
> > >> > is
> > >> > > a
> > >> > > > > > > company
> > >> > > > > > > > > > > incorporated in Brasil and is operated under
> license
> > >> from
> > >> > > > Shape
> > >> > > > > > > Blue
> > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered
> by
> > >> The
> > >> > > > > Republic
> > >> > > > > > > of
> > >> > > > > > > > > > > South Africa and is traded under license from
> Shape
> > >> Blue
> > >> > > Ltd.
> > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > >> > > > > > > > > > > This email and any attachments to it may be
> > >> confidential
> > >> > > and
> > >> > > > > are
> > >> > > > > > > > > > > intended solely for the use of the individual to
> > whom
> > >> it
> > >> > is
> > >> > > > > > > > addressed.
> > >> > > > > > > > > > > Any views or opinions expressed are solely those
> of
> > >> the
> > >> > > > author
> > >> > > > > > and
> > >> > > > > > > do
> > >> > > > > > > > > > > not necessarily represent those of Shape Blue Ltd
> or
> > >> > > related
> > >> > > > > > > > > > > companies. If you are not the intended recipient
> of
> > >> this
> > >> > > > email,
> > >> > > > > > you
> > >> > > > > > > > > > > must neither take any action based upon its
> > contents,
> > >> nor
> > >> > > > copy
> > >> > > > > or
> > >> > > > > > > > show
> > >> > > > > > > > > > > it to anyone. Please contact the sender if you
> > believe
> > >> > you
> > >> > > > have
> > >> > > > > > > > > received
> > >> > > > > > > > > > this email in error.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > -----Original Message-----
> > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> syed1.mushtaq@gmail.com
> > >> > > > > <javascript:;>
> > >> > > > > > <javascript:;>]
> > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > >> > > <javascript:;>
> > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > Snapshots
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I think the terminology confusion comes from AWS
> > where
> > >> > they
> > >> > > > do
> > >> > > > > > EBS
> > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
> > >> followed
> > >> > > > that.
> > >> > > > > > And
> > >> > > > > > > > as
> > >> > > > > > > > > > > an end user who is oblivious to the internals of
> my
> > >> > > provider,
> > >> > > > > my
> > >> > > > > > > > > > > expectation would be something similar to what AWS
> > as
> > >> > that
> > >> > > is
> > >> > > > > my
> > >> > > > > > > > > > > biggest reference point.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > To your point Mike, I agree that a Primary Storage
> > >> > failure
> > >> > > on
> > >> > > > > > > > > > > SolidFire is unlikely, there are other motivations
> > >> for us
> > >> > > to
> > >> > > > > push
> > >> > > > > > > > data
> > >> > > > > > > > > > > to secondary storage. Primary storage (atleast for
> > us)
> > >> > > costs
> > >> > > > > > > around 3
> > >> > > > > > > > > > > times as much as secondary storage and snapshots
> on
> > >> > primary
> > >> > > > > > storage
> > >> > > > > > > > > > > are rarely used (especially for some of our
> > customers
> > >> who
> > >> > > do
> > >> > > > > > daily
> > >> > > > > > > > > > backups).
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > >> > > <javascript:;>>
> > >> > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Some of the weirdness is around terminology.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > For most systems I've worked on, a snapshot and
> a
> > >> > backup
> > >> > > > are
> > >> > > > > > two
> > >> > > > > > > > > > > > completely different things (but CloudStack has
> > >> > > > traditionally
> > >> > > > > > > used
> > >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
> > >> though,
> > >> > > that
> > >> > > > > if
> > >> > > > > > > your
> > >> > > > > > > > > > > > primary storage is running on SolidFire that it
> is
> > >> > > unlikely
> > >> > > > > > > you'll
> > >> > > > > > > > > > > > encounter an issue where your primary storage
> goes
> > >> > > offline
> > >> > > > > (and
> > >> > > > > > > > > > > > you'll even maintain your performance guarantees
> > >> during
> > >> > > > > failure
> > >> > > > > > > > > > > > scenarios and upgrades, as well). That being the
> > >> case,
> > >> > it
> > >> > > > is
> > >> > > > > > less
> > >> > > > > > > > > > > > useful to require a backup to Swift (but it's
> > >> perfectly
> > >> > > OK
> > >> > > > if
> > >> > > > > > > > that's
> > >> > > > > > > > > > > > what we want to do
> > >> > > > > > > > > > > here).
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > >> > <javascript:;>>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Paul,
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > I believe with the current implementation of
> > >> > Snapshots
> > >> > > on
> > >> > > > > > > managed
> > >> > > > > > > > > > > > > storage
> > >> > > > > > > > > > > > > (SolidFire) the snapshots are never exported
> to
> > >> the
> > >> > > > > secondary
> > >> > > > > > > > > > storage.
> > >> > > > > > > > > > > > > While this solves the problem of having
> > snapshots
> > >> > > taking
> > >> > > > > > > forever
> > >> > > > > > > > > > > > > to get to sec storage, this leaves us with a
> > >> > > > > > > > > > > > huge
> > >> > > > > > > > > > > > > liability if our primary storage goes down. We
> > see
> > >> > > > > snapshots
> > >> > > > > > as
> > >> > > > > > > > > > > > > our recovery path because we store them in
> Swift
> > >> > which
> > >> > > is
> > >> > > > > > > > reliable
> > >> > > > > > > > > > > > > and resilient to failures.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> > Volume
> > >> > > > snapshots
> > >> > > > > > > > always
> > >> > > > > > > > > > > > > backed up to secondary storage and Storage
> > >> Snapshots
> > >> > > stay
> > >> > > > > on
> > >> > > > > > > the
> > >> > > > > > > > > > > > > primary
> > >> > > > > > > > > > > > storage.
> > >> > > > > > > > > > > > > A provider could potentially mix both these
> and
> > >> solve
> > >> > > the
> > >> > > > > > > problem
> > >> > > > > > > > > > > > > that you mentioned where you want to meet
> user's
> > >> > > > > expectation
> > >> > > > > > > of a
> > >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while
> having
> > >> an
> > >> > > > ability
> > >> > > > > > to
> > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the device)
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hope this clarifies things.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > -Syed
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > >> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > >> > > <javascript:;>>
> > >> > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > HI guys,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Could someone point me to the Jira bug of FS
> > for
> > >> > the
> > >> > > > > > > > > > > > > > SAN-snapshot
> > >> > > > > > > > > > > > feature
> > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > From my discussions with users and operators
> > >> around
> > >> > > > > > snapshots
> > >> > > > > > > > > > > > > > I'd make
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > following observations:
> > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
> > >> long-term
> > >> > > and
> > >> > > > > > short
> > >> > > > > > > > > > > > > > term)
> > >> > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > the expectation that they can use them for
> > >> recovery
> > >> > > if
> > >> > > > > > > > required.
> > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > something
> > >> > has
> > >> > > > gone
> > >> > > > > > > wrong
> > >> > > > > > > > > > > > > > with primary storage.
> > >> > > > > > > > > > > > > > c. users sometimes want to be able to export
> > >> > > snapshots
> > >> > > > as
> > >> > > > > > > well
> > >> > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > create
> > >> > > > > > > > > > > > > > new VMs from their snapshots
> > >> > > > > > > > > > > > > > d. snapshots are a currently a massive pain
> > for
> > >> > > > > operators,
> > >> > > > > > I
> > >> > > > > > > > > > > > > > know at
> > >> > > > > > > > > > > > > least
> > >> > > > > > > > > > > > > > one public cloud who have snapshots which
> > take 2
> > >> > days
> > >> > > > to
> > >> > > > > > > > > complete.
> > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used for
> > >> > multiple
> > >> > > > LVM
> > >> > > > > > > > disks.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I think the process Mike has used in the
> > >> SolidFire
> > >> > > > plugin
> > >> > > > > > > (only
> > >> > > > > > > > > > > > > > moving
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > disk image to secondary storage when you
> > >> absolutely
> > >> > > > have
> > >> > > > > > to)
> > >> > > > > > > is
> > >> > > > > > > > > > > > > > a very
> > >> > > > > > > > > > > > > good
> > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> problems
> > >> an
> > >> > > > > operator
> > >> > > > > > > > might
> > >> > > > > > > > > > > > > experience
> > >> > > > > > > > > > > > > > if they have an issue with a given primary
> > >> storage
> > >> > > pool
> > >> > > > > in
> > >> > > > > > a
> > >> > > > > > > > > > cluster.
> > >> > > > > > > > > > > > (I
> > >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> > >> SolidFire
> > >> > > case
> > >> > > > > > Mike
> > >> > > > > > > :)
> > >> > > > > > > > > > > > > > ) And
> > >> > > > > > > > > > > > if
> > >> > > > > > > > > > > > > > the transfer from primary to secondary is
> > slow,
> > >> the
> > >> > > > time
> > >> > > > > to
> > >> > > > > > > > > > > > > > being able
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > create a template or export the volume will
> be
> > >> > slow.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > So for me the issue is around making sure
> that
> > >> the
> > >> > > end
> > >> > > > > > users
> > >> > > > > > > > > > > > expectations
> > >> > > > > > > > > > > > > > are met (while improving the
> speed/efficiency
> > of
> > >> > the
> > >> > > > back
> > >> > > > > > > end)
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> http://www.shapeblue.com>
> > >> Paul
> > >> > > > Angus
> > >> > > > > > VP
> > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > >> > > > > > > > > > > > > >
> > >> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > >> > > > > > > |
> > >> > > > > > > > m:
> > >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > >> > > > > <javascript:;> | t:
> > >> > > > > > @cloudyangus*
> > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > >> > > > <javascript:;>
> > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> http://www.shapeblue.com
> > >
> > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> WC2N
> > >> 4HS
> > >> > UK
> > >> > > > > Shape
> > >> > > > > > > > Blue
> > >> > > > > > > > > > > > > > Ltd is a company incorporated in England &
> > >> Wales.
> > >> > > > > ShapeBlue
> > >> > > > > > > > > > > > > > Services India LLP is a company incorporated
> > in
> > >> > India
> > >> > > > and
> > >> > > > > > is
> > >> > > > > > > > > > > > > > operated
> > >> > > > > > > > > > > > > under
> > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> Brasil
> > >> > > > > Consultoria
> > >> > > > > > > Ltda
> > >> > > > > > > > > > > > > > is a company incorporated in Brasil and is
> > >> operated
> > >> > > > under
> > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA
> Pty
> > >> Ltd
> > >> > is
> > >> > > a
> > >> > > > > > > company
> > >> > > > > > > > > > > > > > registered by The Republic
> > >> > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > South Africa and is traded under license
> from
> > >> Shape
> > >> > > > Blue
> > >> > > > > > Ltd.
> > >> > > > > > > > > > > > > > ShapeBlue
> > >> > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > a registered trademark.
> > >> > > > > > > > > > > > > > This email and any attachments to it may be
> > >> > > > confidential
> > >> > > > > > and
> > >> > > > > > > > are
> > >> > > > > > > > > > > > intended
> > >> > > > > > > > > > > > > > solely for the use of the individual to whom
> > it
> > >> is
> > >> > > > > > addressed.
> > >> > > > > > > > > > > > > > Any views
> > >> > > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > opinions expressed are solely those of the
> > >> author
> > >> > and
> > >> > > > do
> > >> > > > > > not
> > >> > > > > > > > > > > > necessarily
> > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or related
> > >> > > companies.
> > >> > > > > If
> > >> > > > > > > you
> > >> > > > > > > > > > > > > > are not
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > intended recipient of this email, you must
> > >> neither
> > >> > > take
> > >> > > > > any
> > >> > > > > > > > > > > > > > action
> > >> > > > > > > > > > > > based
> > >> > > > > > > > > > > > > > upon its contents, nor copy or show it to
> > >> anyone.
> > >> > > > Please
> > >> > > > > > > > contact
> > >> > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > sender
> > >> > > > > > > > > > > > > > if you believe you have received this email
> in
> > >> > error.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > -----Original Message-----
> > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > >> pdion@cloudops.com
> > >> > > > > <javascript:;>
> > >> > > > > > <javascript:;>]
> > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > >> > Snapshots
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Hi Mike,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > >> StorageSnapshot
> > >> > > for
> > >> > > > > > > managed
> > >> > > > > > > > > > > > > > storage
> > >> > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > expected,
> > >> > > > behavior
> > >> > > > > > is
> > >> > > > > > > to
> > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> Storage.
> > >> So a
> > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > >> > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > snapshot that remain on the managed storage
> > >> > > appliance.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Quickly looking at the API doc and I don't
> > see a
> > >> > > strong
> > >> > > > > > > > > > > > > > requirement for volume snapshots to be moved
> > >> into
> > >> > > > > secondary
> > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is
> not
> > >> > useful,
> > >> > > > but
> > >> > > > > > > both
> > >> > > > > > > > > > > > > > use cases are required. A snapshot that
> remain
> > >> on
> > >> > the
> > >> > > > > > managed
> > >> > > > > > > > > > > > > > storage, and another type of
> > >> > > > > > > > > > > > snapshot
> > >> > > > > > > > > > > > > > that end up into the secondary storage.
> Since
> > >> > you've
> > >> > > > > done a
> > >> > > > > > > lot
> > >> > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > work,
> > >> > > > > > > > > > > > > > might easier to just add a parameter to the
> > >> current
> > >> > > > > > snapshot
> > >> > > > > > > > API
> > >> > > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > would
> > >> > > > > > > > > > > > > > trigger an extraction of the storage
> snapshot
> > >> into
> > >> > > the
> > >> > > > > > > > secondary
> > >> > > > > > > > > > > > storage?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > PL
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > Tutkowski <
> > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > I think that all sounds reasonable then -
> > >> thanks!
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > Mushtaq <
> > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > >> <javascript:;>>
> > >> > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> > >> > requirements.
> > >> > > > One
> > >> > > > > > of
> > >> > > > > > > > our
> > >> > > > > > > > > > > > earlier
> > >> > > > > > > > > > > > > > >> iterations on this was to have an
> argument
> > to
> > >> > the
> > >> > > > > create
> > >> > > > > > > > > > > > > > >> snapshot
> > >> > > > > > > > > > > > API
> > >> > > > > > > > > > > > > > >> which decides whether to backup the
> volume
> > to
> > >> > sec
> > >> > > > > > storage
> > >> > > > > > > > but
> > >> > > > > > > > > > > > > > >> we realized it would make management of
> > >> > snapshots
> > >> > > > > quite
> > >> > > > > > > > messy
> > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > >> > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> Tutkowski
> > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > <javascript:;>
> > >> > > > > <javascript:;>>
> > >> > > > > > > > > > > > > > >> wrote:
> > >> > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > >>> Hi,
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> Just to make sure I understand all the
> > >> > > requirements
> > >> > > > > > here:
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 1) This relates only to managed storage
> > (1:1
> > >> > > > mapping
> > >> > > > > > > > between
> > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> volume).
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> (introduced
> > >> in
> > >> > > 4.6)
> > >> > > > > > > > > > > > > > >>> functionality, which creates a snapshot
> on
> > >> the
> > >> > > SAN,
> > >> > > > > and
> > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> > >> snapshot,
> > >> > but
> > >> > > > to
> > >> > > > > > copy
> > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to
> > NFS.
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken.
> It's
> > >> the
> > >> > > > backup
> > >> > > > > to
> > >> > > > > > > NFS
> > >> > > > > > > > > > > > > > >>> that is optional.
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> > >> snapshot
> > >> > > > that's
> > >> > > > > on
> > >> > > > > > > the
> > >> > > > > > > > > > > > > > >>> SAN (already works).
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created
> from
> > >> the
> > >> > > > > snapshot
> > >> > > > > > > > > > > > > > >>> that's on
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > >>> SAN (already works as long as the new
> > >> > CloudStack
> > >> > > > > volume
> > >> > > > > > > > ends
> > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > snapshot
> > >> API
> > >> > > > then
> > >> > > > > or
> > >> > > > > > > > > > > > > > >>> would that just be the standard volume
> > >> snapshot
> > >> > > > > without
> > >> > > > > > > the
> > >> > > > > > > > > > > > > > >>> backup to
> > >> > > > > > > > > > > NFS?
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> Thanks!
> > >> > > > > > > > > > > > > > >>> Mike
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> > Mushtaq
> > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
> > >> > > > > <javascript:;>>
> > >> > > > > > > > > > > > > > >>> wrote:
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > functionalities
> > >> > > > > (snapshot
> > >> > > > > > on
> > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we
> > would
> > >> > like
> > >> > > > to
> > >> > > > > > have
> > >> > > > > > > > > both.
> > >> > > > > > > > > > > > > > >>>> For example, some of our customers want
> > to
> > >> > > > implement
> > >> > > > > > > their
> > >> > > > > > > > > > > > > > >>>> own backup strategies and do encryption
> > to
> > >> > their
> > >> > > > > > backups
> > >> > > > > > > > > > > > > > >>>> which is a perfect use case for Storage
> > >> > Snapshot
> > >> > > > > while
> > >> > > > > > > our
> > >> > > > > > > > > > > > > > >>>> other customers will still keep using
> the
> > >> > > standard
> > >> > > > > > > volume
> > >> > > > > > > > > > > snapshot.
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> To keep things backward compatible, we
> > can
> > >> > add a
> > >> > > > > > setting
> > >> > > > > > > > > > > > > > >>>> which
> > >> > > > > > > > > > > > says
> > >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> > >> because,
> > >> > > after
> > >> > > > > > all,
> > >> > > > > > > > you
> > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when
> > doing
> > >> a
> > >> > > > Volume
> > >> > > > > > > > > Snapshot.
> > >> > > > > > > > > > > > > > >>>> You could stop the process there and
> not
> > do
> > >> > the
> > >> > > > > > upload.
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> What do you think about this approach?
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> Thanks,
> > >> > > > > > > > > > > > > > >>>> -Syed
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> > >> > Tutkowski <
> > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out load
> > >> here,
> > >> > but
> > >> > > > if
> > >> > > > > a
> > >> > > > > > > > given
> > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need
> > to
> > >> > > provide
> > >> > > > > > both
> > >> > > > > > > > the
> > >> > > > > > > > > > > > ability
> > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it
> to
> > >> NFS
> > >> > (if
> > >> > > > > just
> > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we
> > >> might
> > >> > be
> > >> > > > able
> > >> > > > > > to
> > >> > > > > > > > get
> > >> > > > > > > > > > > > > > >>>>> away with no new
> > >> > > > > > > > > > > > API
> > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> custom
> > >> > > snapshot
> > >> > > > > > > strategy
> > >> > > > > > > > > > > > > > >>>>> and
> > >> > > > > > > > > > > > data
> > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case
> where
> > >> the
> > >> > > > > > CloudStack
> > >> > > > > > > > > > > > > > >>>>> cloud
> > >> > > > > > > > > > > > does
> > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > >> exported-to-NFS
> > >> > > > > backup.
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> In other words, the "default" behavior
> > >> would
> > >> > be
> > >> > > > to
> > >> > > > > > use
> > >> > > > > > > > the
> > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> > strategy
> > >> > that
> > >> > > > we
> > >> > > > > > > > already
> > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> > >> > snapshot).
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however,
> wants
> > >> to
> > >> > > take
> > >> > > > a
> > >> > > > > > SAN
> > >> > > > > > > > > > > > > > >>>>> snapshot and have the data exported to
> > >> NFS,
> > >> > > then
> > >> > > > we
> > >> > > > > > > could
> > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config
> file
> > to
> > >> > make
> > >> > > > use
> > >> > > > > > of
> > >> > > > > > > a
> > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data motion
> > >> > strategy
> > >> > > > that
> > >> > > > > > > > > > > > > > >>>>> performs both of these
> > >> > > > > > > > > > > > > > activities.
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is still
> the
> > >> > default
> > >> > > > for
> > >> > > > > > > > users,
> > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change this
> > >> > behavior
> > >> > > > via
> > >> > > > > > > > > > > configuration.
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike
> > >> > > Tutkowski <
> > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to come
> up
> > >> > with a
> > >> > > > > > viable
> > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way
> for
> > >> them
> > >> > > to
> > >> > > > > move
> > >> > > > > > > > from
> > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and some
> > >> obvious
> > >> > > way
> > >> > > > > > that
> > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > do this).
> > >> > > > > > > > > > > > > > >>>>>>
> > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed
> > >> > Mushtaq
> > >> > > <
> > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>>
> > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> upgrade
> > >> path
> > >> > > > > however,
> > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and are
> > on a
> > >> > > > managed
> > >> > > > > > > > storage
> > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > >> functionality
> > >> > > > with
> > >> > > > > > > Volume
> > >> > > > > > > > > > > Snapshots.
> > >> > > > > > > > > > > > > > >>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed
> > >> > Mushtaq
> > >> > > <
> > >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> > <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly,
> > currently
> > >> > > > taking a
> > >> > > > > > > > Volume
> > >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a managed
> > >> storage
> > >> > > > keeps
> > >> > > > > > it
> > >> > > > > > > on
> > >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part of
> this
> > >> > > feature,
> > >> > > > we
> > >> > > > > > can
> > >> > > > > > > > > > > > > > >>>>>>>> make sure
> > >> > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed storage
> > are
> > >> > > > uploaded
> > >> > > > > > to
> > >> > > > > > > > the
> > >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would make
> > the
> > >> > > Volume
> > >> > > > > > > Snapshot
> > >> > > > > > > > > > > > > > >>>>>>>> feature behave the same regardless
> of
> > >> the
> > >> > > > > storage
> > >> > > > > > > > > > > > > > >>>>>>>> (managed or
> > >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing the
> > >> > > efficient
> > >> > > > > > > backend
> > >> > > > > > > > > > > > > > >>>>>>>> storage
> > >> > > > > > > > > > > > > > capabilities, we can use the new storage
> > >> snapshots
> > >> > > API.
> > >> > > > > > > > > > > > > > >>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM,
> Mike
> > >> > > > Tutkowski <
> > >> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> > >> > <javascript:;>
> > >> > > > > <javascript:;>>
> > >> > > > > > wrote:
> > >> > > > > > > > > > > > > > >>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > >> > > > > > > > > > > > > > >>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need to
> > have a
> > >> > plan
> > >> > > > to
> > >> > > > > > deal
> > >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we already
> have a
> > >> > > feature
> > >> > > > > (in
> > >> > > > > > > 4.6
> > >> > > > > > > > > > > > > > >>>>>>>>> and
> > >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to use the
> > >> > existing
> > >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a
> > >> volume
> > >> > > > > snapshot
> > >> > > > > > > (for
> > >> > > > > > > > > > > > > > >>>>>>>>> managed
> > >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a backend
> > SAN
> > >> > > > (using a
> > >> > > > > > > > custom
> > >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom
> data
> > >> > motion
> > >> > > > > > > strategy).
> > >> > > > > > > > > > > > > > >>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then how
> > >> should
> > >> > > the
> > >> > > > > > > original
> > >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6 and
> > >> later)
> > >> > > be
> > >> > > > > > > changed?
> > >> > > > > > > > > > > > > > >>>>>>>>> If it
> > >> > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support
> customers
> > >> who
> > >> > > are
> > >> > > > > > > already
> > >> > > > > > > > > > > > > > >>>>>>>>> using
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API to
> take
> > >> > > > snapshots
> > >> > > > > > on a
> > >> > > > > > > > > > > > > > >>>>>>>>> backend
> > >> > > > > > > > > > > > > SAN?
> > >> > > > > > > > > > > > > > >>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > >> > > > > > > > > > > > > > >>>>>>>>> Mike
> > >> > > > > > > > > > > > > > >>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM,
> > Will
> > >> > > > Stevens <
> > >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> > <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create a
> > Template
> > >> > > from a
> > >> > > > > > > > > > > > StorageSnapshot?
> > >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template be
> stored
> > >> in
> > >> > the
> > >> > > > > > > secondary
> > >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal templates or
> > will
> > >> > that
> > >> > > > be
> > >> > > > > > > > handled
> > >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > >> > > > > > > > > > > > > > vendor side?
> > >> > > > > > > > > > > > > > >>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > >> > > > > > > > > > > > > > >>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions
> > >> Experts
> > >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|*
> Quebec
> > >> *|*
> > >> > > H3J
> > >> > > > > 1S6
> > >> > > > > > w
> > >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > >> > > > > > > > > > > > > > >>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM,
> > Syed
> > >> > > > Mushtaq <
> > >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM,
> > Will
> > >> > > > Stevens
> > >> > > > > <
> > >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design
> > >> Spec in
> > >> > > the
> > >> > > > > > Jira
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not clear
> > in
> > >> the
> > >> > > > > > 'mention'
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows as a
> > page
> > >> > 'you
> > >> > > do
> > >> > > > > not
> > >> > > > > > > > have
> > >> > > > > > > > > > > permission to'.
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions
> > >> Experts
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|*
> > Quebec
> > >> > *|*
> > >> > > > H3J
> > >> > > > > > 1S6
> > >> > > > > > > w
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM,
> > >> Syed
> > >> > > Ahmed
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > >
> > >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set
> of
> > >> APIs
> > >> > to
> > >> > > > do
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage
> > >> backends
> > >> > > > like
> > >> > > > > > > > > SolidFire.
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed
> > >> storage
> > >> > > stay
> > >> > > > > on
> > >> > > > > > > the
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary to
> what
> > >> > > > CloudStack
> > >> > > > > > > calls
> > >> > > > > > > > > > > snpshots.
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on
> storage
> > >> and
> > >> > > > keeping
> > >> > > > > > it
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > >> > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to have
> > both
> > >> > ways
> > >> > > > of
> > >> > > > > > > doing
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds
> 4
> > >> new
> > >> > > APIs
> > >> > > > to
> > >> > > > > > > > create
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of this
> > >> > > feature? I
> > >> > > > > > would
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I
> am
> > >> > > working
> > >> > > > on
> > >> > > > > > > > making
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > >> > > > > > > > > > > > spec
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to
> > have a
> > >> > high
> > >> > > > > level
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before starting
> > to
> > >> > work
> > >> > > on
> > >> > > > > it.
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>> --
> > >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer,
> > >> SolidFire
> > >> > > Inc.*
> > >> > > > > > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > >> > > > <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world uses
> the
> > >> > cloud
> > >> > > > > > > > > > > > > > >>>>>>>>> <
> > >> > > > > > > http://solidfire.com/solution/overview/?video=play
> > >> > > > > > > > >*™
> > >> > > > > > > > > > > > > > >>>>>>>>> *
> > >> > > > > > > > > > > > > > >>>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>>
> > >> > > > > > > > > > > > > > >>>>>>
> > >> > > > > > > > > > > > > > >>>>>>
> > >> > > > > > > > > > > > > > >>>>>> --
> > >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer,
> SolidFire
> > >> > Inc.*
> > >> > > > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > >> > > <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > >> > > > > > > > > > > > > > >>>>>> Advancing the way the world uses the
> > >> cloud
> > >> > > > > > > > > > > > > > >>>>>> <
> > >> > > > > http://solidfire.com/solution/overview/?video=play
> > >> > > > > > > >*™*
> > >> > > > > > > > > > > > > > >>>>>>
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> --
> > >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer,
> SolidFire
> > >> Inc.*
> > >> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > >> > <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > >> > > > > > > > > > > > > > >>>>> Advancing the way the world uses the
> > cloud
> > >> > > > > > > > > > > > > > >>>>> <
> > >> > > > > http://solidfire.com/solution/overview/?video=play
> > >> > > > > > > >*™*
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> --
> > >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer, SolidFire
> > >> Inc.*
> > >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > >>> o: 303.746.7302
> > >> > > > > > > > > > > > > > >>> Advancing the way the world uses the
> cloud
> > >> > > > > > > > > > > > > > >>> <
> > >> > > > http://solidfire.com/solution/overview/?video=play
> > >> > > > > > >*™*
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > --
> > >> > > > > > > > > > > > > > > *Mike Tutkowski*
> > >> > > > > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> > Inc.*
> > >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > > o: 303.746.7302
> > >> > > > > > > > > > > > > > > Advancing the way the world uses the cloud
> > >> > > > > > > > > > > > > > > <
> > >> > > http://solidfire.com/solution/overview/?video=play
> > >> > > > > >*™*
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Find out more about ShapeBlue and our range
> of
> > >> > > > CloudStack
> > >> > > > > > > > > > > > > > related
> > >> > > > > > > > > > > > > services:
> > >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > >> > > > > > > > > > > > > > <
> > >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > >> > > |
> > >> > > > > > > > CSForge –
> > >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > >> > > > > > > http://shapeblue.com/csforge/
> > >> > > > > > > > >
> > >> > > > > > > > > > > > > > CloudStack Consulting
> > >> > > > > > > > > > > > > > <
> http://shapeblue.com/cloudstack-consultancy/
> > >
> > >> |
> > >> > > > > > > > > > > > > CloudStack
> > >> > > > > > > > > > > > > > Software Engineering
> > >> > > > > > > > > > > > > > <
> > >> > > http://shapeblue.com/cloudstack-software-engineering/
> > >> > > > >
> > >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > >> > > > > > > > > > > > > > <
> > >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> > >> > > > > >
> > >> > > > > > |
> > >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > >> > > > > > > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > --
> > >> > > > > > > > > > > > *Mike Tutkowski*
> > >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > o: 303.746.7302
> > >> > > > > > > > > > > > Advancing the way the world uses the cloud
> > >> > > > > > > > > > > > <
> > http://solidfire.com/solution/overview/?video=play
> > >> > >*™*
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > Find out more about ShapeBlue and our range of
> > >> CloudStack
> > >> > > > > related
> > >> > > > > > > > > > services:
> > >> > > > > > > > > > > IaaS Cloud Design & Build
> > >> > > > > > > > > > > <
> http://shapeblue.com/iaas-cloud-design-and-build//
> > >
> > >> |
> > >> > > > > CSForge –
> > >> > > > > > > > rapid
> > >> > > > > > > > > > > IaaS deployment framework <
> > >> http://shapeblue.com/csforge/
> > >> > >
> > >> > > > > > > CloudStack
> > >> > > > > > > > > > > Consulting <
> > >> http://shapeblue.com/cloudstack-consultancy/
> > >> > >
> > >> > > |
> > >> > > > > > > > CloudStack
> > >> > > > > > > > > > > Software Engineering
> > >> > > > > > > > > > > <
> > >> http://shapeblue.com/cloudstack-software-engineering/>
> > >> > > > > > > > > > > CloudStack Infrastructure Support
> > >> > > > > > > > > > > <
> > >> http://shapeblue.com/cloudstack-infrastructure-support/
> > >> > >
> > >> > > |
> > >> > > > > > > > CloudStack
> > >> > > > > > > > > > > Bootcamp Training Courses <
> > >> > > > > > > http://shapeblue.com/cloudstack-training/
> > >> > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > Find out more about ShapeBlue and our range of
> > >> CloudStack
> > >> > > > related
> > >> > > > > > > > > services:
> > >> > > > > > > > > > IaaS Cloud Design & Build
> > >> > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//
> >
> > |
> > >> > > > CSForge –
> > >> > > > > > > rapid
> > >> > > > > > > > > > IaaS deployment framework <
> > >> http://shapeblue.com/csforge/>
> > >> > > > > > > > > > CloudStack Consulting <
> > >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > >> > > > > > > |
> > >> > > > > > > > > CloudStack
> > >> > > > > > > > > > Software Engineering
> > >> > > > > > > > > > <
> > http://shapeblue.com/cloudstack-software-engineering/>
> > >> > > > > > > > > > CloudStack Infrastructure Support
> > >> > > > > > > > > > <
> > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > >> > |
> > >> > > > > > > CloudStack
> > >> > > > > > > > > > Bootcamp Training Courses <
> > >> > > > > > http://shapeblue.com/cloudstack-training/
> > >> > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > *Mike Tutkowski*
> > >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>
> > >> > > > > > > o: 303.746.7302
> > >> > > > > > > Advancing the way the world uses the cloud
> > >> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > *Mike Tutkowski*
> > >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > >> > > > > o: 303.746.7302
> > >> > > > > Advancing the way the world uses the cloud
> > >> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > Ian Rae
> > >> > > > CEO | PDG
> > >> > > > c: 514.944.4008
> > >> > > >
> > >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> > >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > *Mike Tutkowski*
> > >> > *Senior CloudStack Developer, SolidFire Inc.*
> > >> > e: mike.tutkowski@solidfire.com
> > >> > o: 303.746.7302
> > >> > Advancing the way the world uses the cloud
> > >> > <http://solidfire.com/solution/overview/?video=play>*™*
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

So, right now in 4.6, 4.7, and 4.8, the behavior for a managed storage
volume snapshot is for the data to remain on the SAN (not secondary
storage).

It was simply designed as a space-efficient and fast alternative to copying
data to NFS (secondary storage).

What we need to do is somehow maintain that original behavior, but augment
it with the option of backing up the data to secondary storage (and, if
doing that, then delete the SAN snapshot).

On Mon, Feb 8, 2016 at 11:55 AM, Ian Rae <ir...@cloudops.com> wrote:

> I would hope that default behaviour in CloudStack is the traditional volume
> snapshot moved to secondary storage. A global setting to change that
> behaviour is probably ok if it is not default, but the user would want to
> in certain cases make copies of those snapshots to secondary storage in
> addition to taking the snapshot on the primary storage.
>
> On Monday, February 8, 2016, Will Stevens <ws...@cloudops.com> wrote:
>
> > I don't think a global setting is a good option because we need both
> > functionality to be available at the same time and for different use
> cases
> > to be able to pick which they choose.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com <javascript:;>
> > > wrote:
> >
> > > Now that I re-read your e-mail, it dawned on me: The end-user doesn't
> > care
> > > where the snapshot is.
> > >
> > > If that's true, then we should perhaps control this via Global Settings
> > or
> > > something.
> > >
> > > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com> wrote:
> > >
> > > > It's not ideal - true, but it does allow us to be backward
> compatible.
> > > >
> > > > If you have other ideas, though, about how to maintain backward
> > > > compatibility, I'm definitely open to hear them.
> > > >
> > > > Thanks!
> > > >
> > > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <
> syed1.mushtaq@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi Mike,
> > > >>
> > > >> Adding a flag to createSnapshot was the first and the most obvious
> > thing
> > > >> that came to our minds. The problem that I had with this was that:
> > > >>
> > > >> 1) I feel it is exposing something to the end user that is internal
> to
> > > the
> > > >> cloud.
> > > >>
> > > >> 2) We have to follow two different ways of restore/deletion in the
> > same
> > > >> code path depending on where the Snapshot resides which I find kind
> > of a
> > > >> bad design.
> > > >>
> > > >> But if exposing a archive flag to the end user is acceptable then we
> > can
> > > >> definitely use this instead of adding the StorageSnapshot API
> > > >>
> > > >> Thanks,
> > > >> -Syed
> > > >>
> > > >>
> > > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > > >> mike.tutkowski@solidfire.com
> > > >> > wrote:
> > > >>
> > > >> > Hi Pierre-Luc,
> > > >> >
> > > >> > My recommendation would be this:
> > > >> >
> > > >> > Add an "archive" flag to the current volume-snapshot API. Its
> > default
> > > >> would
> > > >> > be "false" because that would be backward compatible with how 4.6
> > has
> > > >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6,
> 4.7,
> > > and
> > > >> > 4.8).
> > > >> >
> > > >> > If you set archive=true, then we would perform a background
> > migration
> > > of
> > > >> > the snapshot from the SAN to the secondary storage (then delete
> the
> > > SAN
> > > >> > snapshot).
> > > >> >
> > > >> > That archive parameter would only be valid for managed storage.
> > > >> >
> > > >> > Sound reasonable?
> > > >> >
> > > >> > Also, a VM snapshot that includes disks provided by managed
> storage
> > > >> should
> > > >> > work.
> > > >> >
> > > >> > Talk to you later,
> > > >> > Mike
> > > >> >
> > > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <
> pdion@cloudops.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Mike,
> > > >> > >
> > > >> > > In terms of API's, would you prefer introducing a parameter to
> the
> > > >> > existing
> > > >> > > VolumeSnapshot, example:   extract={true|false}  with a default
> > > value
> > > >> of
> > > >> > > true  which would extract snapshot into the secondary storage,
> > which
> > > >> is
> > > >> > the
> > > >> > > current default behavior. Then for SAN snapshot that remain on
> the
> > > >> SAN we
> > > >> > > would just set "extract=false" ?  as oppose to create a new
> > > >> > >  StorageSnapshot API ?
> > > >> > >
> > > >> > >
> > > >> > > Paul,
> > > >> > >
> > > >> > > From what I'm seeing so far, we can't do a VM-snapshot when
> using
> > > >> managed
> > > >> > > storage for VM having more than one Volume. For the reason that
> > > >> snapshot
> > > >> > > are performed outside of the hypervisor awareness and
> > > asynchronously.
> > > >> If
> > > >> > > someone have a way to address that, it would make thinks much
> more
> > > >> > > attractive.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com>
> > wrote:
> > > >> > >
> > > >> > > > I think a service provider backup scenario is more likely to
> > take
> > > >> > > advantage
> > > >> > > > of SAN snapshot. There are a few reasons for this. Traditional
> > > >> backups
> > > >> > > > involve access to the file system, and there is an expectation
> > > that
> > > >> > this
> > > >> > > > can be done with reasonably short time frames without
> negatively
> > > >> > > impacting
> > > >> > > > VM performance, and that the backup orchestrator can apply
> > various
> > > >> > logic
> > > >> > > > and or transformations to the data (compress, encrypt, deltas
> > > >> etc...).
> > > >> > > > While it is true that one could apply a backup process to a
> > cloud
> > > >> > > snapshot,
> > > >> > > > this would be slow and inefficient requiring the data to be
> > moved
> > > >> > several
> > > >> > > > times and there are some major bottlenecks with cloud
> snapshots.
> > > >> With a
> > > >> > > > cloud snapshot - there seems to be no reasonable expectation
> of
> > > >> being
> > > >> > > able
> > > >> > > > to do differential snapshots (I think this depends on the
> > > >> hypervisor)
> > > >> > and
> > > >> > > > if you do differential snapshots this will make file backups
> > from
> > > >> those
> > > >> > > > snapshots even more complicated to orchestrate.
> > > >> > > >
> > > >> > > > Suspect there needs to be a different thread on how to better
> > > enable
> > > >> > > > backups, as a service. As per Paul's suggestion, but it is a
> > > related
> > > >> > > > workflow so relevant to this discussion.
> > > >> > > >
> > > >> > > > Ian
> > > >> > > >
> > > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > >> > > mike.tutkowski@solidfire.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > To me it sounds like number two and number three are
> different
> > > >> uses
> > > >> > for
> > > >> > > > the
> > > >> > > > > same "thing"(which is totally fine).
> > > >> > > > >
> > > >> > > > > As for taking a fast SAN snapshot and exporting it
> > > >> asynchronously, do
> > > >> > > we
> > > >> > > > > see the SSVM as performing the export?
> > > >> > > > >
> > > >> > > > > To be backwards compatible with what we have in 4.6 and
> later
> > > for
> > > >> > > volume
> > > >> > > > > snapshots for managed storage, I think it might be easier if
> > we
> > > >> pass
> > > >> > > in a
> > > >> > > > > flag that says whether or not to archive the SAN snapshot
> > > (which,
> > > >> I
> > > >> > > > think,
> > > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > > >> > > > >
> > > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > > pdion@cloudops.com
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > >
> > > >> > > > > > Hi Mike,
> > > >> > > > > >
> > > >> > > > > > The reason behind the creation of a SAN snapshot which is
> > > >> exported
> > > >> > > into
> > > >> > > > > > secondary storage, is because creating a copy of the .VHD
> > > >> directly
> > > >> > > > would
> > > >> > > > > > impact uptime of the VM as creating that copy take lots of
> > > time.
> > > >> > Has
> > > >> > > > > oppose
> > > >> > > > > > to a SAN snapshot that is close to instantaneous which can
> > > >> > afterward
> > > >> > > be
> > > >> > > > > > clone into Secondary Storage asynchronously.
> > > >> > > > > >
> > > >> > > > > > I would suspect an extracted VolumeSnapshot taken from a
> SAN
> > > >> > snapshot
> > > >> > > > > could
> > > >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> > > >> consumption
> > > >> > > on
> > > >> > > > > the
> > > >> > > > > > Primary Storage side.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > I see 3 definitions in our current discussion regarding
> the
> > > term
> > > >> > > > snapshot
> > > >> > > > > > (these are not official terminology but by own
> > interpretation
> > > of
> > > >> > > them):
> > > >> > > > > >
> > > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition
> of
> > a
> > > >> > > > snapshot):
> > > >> > > > > > it's a volume snapshot at the storage level, point in time
> > of
> > > >> your
> > > >> > > > data.
> > > >> > > > > it
> > > >> > > > > > reside on the primary storage. Useful and efficient for
> > > software
> > > >> > side
> > > >> > > > > > incident.
> > > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/
> cloud
> > > >> backup
> > > >> > > > aws-S3
> > > >> > > > > > style ): Point in time copy of the Virtual Disk that
> reside
> > > on a
> > > >> > > > > different
> > > >> > > > > > storage array then the original Volume. Facilitate data
> > > >> migration
> > > >> > > > between
> > > >> > > > > > clusters and, in case of primary storage incident, Volume
> > > >> snapshots
> > > >> > > are
> > > >> > > > > not
> > > >> > > > > > impacted and can be reuse.
> > > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that
> > also
> > > >> > > validate
> > > >> > > > > data
> > > >> > > > > > integrity, provide a storage efficient archiving method
> and
> > an
> > > >> > > > > independent
> > > >> > > > > > way to restore your data in case of an major
> infrastructure
> > > >> > disaster.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Regards,
> > > >> > > > > >
> > > >> > > > > > PL
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > >> > > > > > mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> > > >> > > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > So, let's see if I currently follow the requirements:
> > > >> > > > > > >
> > > >> > > > > > > * Augment volume snapshots for managed storage to
> > > >> conditionally
> > > >> > > > export
> > > >> > > > > > data
> > > >> > > > > > > to NFS. The current process of taking a snapshot on the
> > SAN
> > > is
> > > >> > > fine,
> > > >> > > > > but
> > > >> > > > > > > we'd like the option to export the data to NFS, as well.
> > > >> > > > > > >
> > > >> > > > > > > Questions:
> > > >> > > > > > >
> > > >> > > > > > > Once the data has been exported to NFS, do we keep the
> SAN
> > > >> > snapshot
> > > >> > > > or
> > > >> > > > > > > delete it?
> > > >> > > > > > >
> > > >> > > > > > > If we are deleting the SAN snapshot, then why don't we
> > just
> > > >> copy
> > > >> > > the
> > > >> > > > > VHD
> > > >> > > > > > > from primary to secondary the way we do today for
> > > non-managed
> > > >> > (i.e.
> > > >> > > > > > > traditional) storage? Why create a SAN snapshot in that
> > > >> scenario?
> > > >> > > > > Perhaps
> > > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> > secondary
> > > >> > > storage
> > > >> > > > > > > instead of a XenServer host?
> > > >> > > > > > >
> > > >> > > > > > > Thanks for the clarification.
> > > >> > > > > > >
> > > >> > > > > > > By the way, to me a backup is when you copy data from
> one
> > > >> storage
> > > >> > > > > system
> > > >> > > > > > to
> > > >> > > > > > > another (regardless of features, if any, to restore the
> > data
> > > >> in
> > > >> > the
> > > >> > > > > > > future). A snapshot is a point-in-time view of the data
> > of a
> > > >> > volume
> > > >> > > > and
> > > >> > > > > > > it's stored on the same storage system as the volume.
> > > >> > > > > > >
> > > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > >> > > pdion@cloudops.com
> > > >> > > > > <javascript:;>
> > > >> > > > > > <javascript:;>>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > That's fun to see that discussion happening. I 100%
> > agree
> > > >> with
> > > >> > > > Paul's
> > > >> > > > > > > > points of view. VolumeSnapshot are not a backup, but I
> > do
> > > >> > > consider
> > > >> > > > > them
> > > >> > > > > > > as
> > > >> > > > > > > > a safety vest against Primary Storage failure, because
> > > >> failure
> > > >> > > > append
> > > >> > > > > > > :-( .
> > > >> > > > > > > >
> > > >> > > > > > > > The current proposal around snapshots that reside on
> the
> > > >> > primary
> > > >> > > > > > storage
> > > >> > > > > > > or
> > > >> > > > > > > > snapshots that end in the Secondary Storage  is not to
> > > >> address
> > > >> > > any
> > > >> > > > > kind
> > > >> > > > > > > of
> > > >> > > > > > > > backups requirement because a snapshot is not a
> backup,
> > > >> event
> > > >> > an
> > > >> > > > > > > extracted
> > > >> > > > > > > > VM snapshot.
> > > >> > > > > > > >
> > > >> > > > > > > > The main idea, and again this is for managed storage;
> > > >> > > > > > > >
> > > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> > > >> capability
> > > >> > > for
> > > >> > > > > > fast
> > > >> > > > > > > > response time that support rollback to previous
> > timestamp,
> > > >> > create
> > > >> > > > new
> > > >> > > > > > > > volume and maybe create template.
> > > >> > > > > > > >     not required to be a new API if the work is
> already
> > > >> done, I
> > > >> > > > think
> > > >> > > > > > > this
> > > >> > > > > > > > is a different behaviors than the user expectation of
> a
> > > >> > > > > > volume-snapshot.
> > > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> > behavior
> > > >> that
> > > >> > > > create
> > > >> > > > > > an
> > > >> > > > > > > > extraction of a volume into SecondaryStorage which can
> > be
> > > >> reuse
> > > >> > > to
> > > >> > > > > > > create a
> > > >> > > > > > > > new volume into another Primary Storage. This type of
> > > >> snapshot
> > > >> > > is a
> > > >> > > > > > slow
> > > >> > > > > > > > job since yes it would have to copy the full volume
> size
> > > on
> > > >> the
> > > >> > > > > > Secondary
> > > >> > > > > > > > storage.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > PL
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > >> > > > > > <javascript:;>>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > I think I share you view on the 'Ideal world'.
> Backup
> > > (via
> > > >> > > Volume
> > > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This
> is
> > > >> > > amplified
> > > >> > > > > > > > especially
> > > >> > > > > > > > > when you have a object storage as your secondary
> > storage
> > > >> > > because
> > > >> > > > it
> > > >> > > > > > > > > requires two copies (one to an NFS staging area and
> > from
> > > >> > there
> > > >> > > to
> > > >> > > > > > > object
> > > >> > > > > > > > > storage). And not to mention that all these copies
> are
> > > >> > > consuming
> > > >> > > > > > > > hypervisor
> > > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge
> bottleneck
> > as
> > > >> all
> > > >> > > the
> > > >> > > > > > > Network
> > > >> > > > > > > > > and I/O flow through it. So our intention of
> proposing
> > > the
> > > >> > > > "Storage
> > > >> > > > > > > > > Snapshots" is to give a better way of achiving
> > snapshots
> > > >> > while
> > > >> > > > > still
> > > >> > > > > > > > > keeping the original definition of volume snpashots
> > (ie
> > > >> > upload
> > > >> > > to
> > > >> > > > > sec
> > > >> > > > > > > > > storage).
> > > >> > > > > > > > >
> > > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > > backups.
> > > >> > They
> > > >> > > > > don't
> > > >> > > > > > > work
> > > >> > > > > > > > > form multi-disk LVM volume groups and dynamic
> disks. I
> > > am
> > > >> all
> > > >> > > in
> > > >> > > > > for
> > > >> > > > > > a
> > > >> > > > > > > > > better backup solution which handles these use cases
> > and
> > > >> > > utilizes
> > > >> > > > > the
> > > >> > > > > > > > > storage's advanced features.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > >> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > In the beginning... there were CloudStack
> snapshots
> > > and
> > > >> > they
> > > >> > > > were
> > > >> > > > > > > > > actually
> > > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > > snapshots.
> > > >> > > > > > > > > > Then VM snapshots were created (which are
> > > point-in-time
> > > >> > > > > hypervisor
> > > >> > > > > > > > > > snapshots) and we started referring to the
> original
> > > >> > snapshots
> > > >> > > > as
> > > >> > > > > > > volume
> > > >> > > > > > > > > > snapshots.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > CloudStack does not offer 'backups', but many
> people
> > > use
> > > >> > > volume
> > > >> > > > > > > > snapshots
> > > >> > > > > > > > > > as backups. However you can't in-place restore
> > volume
> > > >> > > snapshots
> > > >> > > > > and
> > > >> > > > > > > if
> > > >> > > > > > > > > you
> > > >> > > > > > > > > > have a VM with multiple volumes, the volume
> > snapshots
> > > >> must
> > > >> > be
> > > >> > > > > done
> > > >> > > > > > in
> > > >> > > > > > > > > > series, meaning that the state across all of the
> > > >> volumes is
> > > >> > > > > > unlikely
> > > >> > > > > > > to
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > consistent.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> > > options
> > > >> > > which
> > > >> > > > > > users
> > > >> > > > > > > > > might
> > > >> > > > > > > > > > expect as well options as to where they might be
> > > >> stored. In
> > > >> > > my
> > > >> > > > > > ideal
> > > >> > > > > > > > > world
> > > >> > > > > > > > > > they would also be able to leverage back-end
> > hardware
> > > >> (such
> > > >> > > as
> > > >> > > > > > > > Solidfire,
> > > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> > Commvault
> > > >> etc
> > > >> > to
> > > >> > > > > > > accelerate
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > process.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > >> > > > > > > > > > Paul Angus
> > > >> > > > > > > > > > VP Technology ,  ShapeBlue
> > > >> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > >> > > > > > > > > >
> > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> > > >> > > > > m:
> > > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > >> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> > > >> > <javascript:;>
> > > >> > > |
> > > >> > > > > t: @cloudyangus*
> > > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > >> > > > > <ja...@cloudyangus>
> > > >> > > > > > |  w:
> > > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > >> > > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N
> 4HS
> > UK
> > > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in
> England
> > &
> > > >> > Wales.
> > > >> > > > > > > ShapeBlue
> > > >> > > > > > > > > > Services India LLP is a company incorporated in
> > India
> > > >> and
> > > >> > is
> > > >> > > > > > operated
> > > >> > > > > > > > > under
> > > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > >> Consultoria
> > > >> > > Ltda
> > > >> > > > > is
> > > >> > > > > > a
> > > >> > > > > > > > > > company incorporated in Brasil and is operated
> under
> > > >> > license
> > > >> > > > from
> > > >> > > > > > > Shape
> > > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > registered
> > > >> by
> > > >> > The
> > > >> > > > > > > Republic
> > > >> > > > > > > > of
> > > >> > > > > > > > > > South Africa and is traded under license from
> Shape
> > > Blue
> > > >> > Ltd.
> > > >> > > > > > > ShapeBlue
> > > >> > > > > > > > > is
> > > >> > > > > > > > > > a registered trademark.
> > > >> > > > > > > > > > This email and any attachments to it may be
> > > confidential
> > > >> > and
> > > >> > > > are
> > > >> > > > > > > > intended
> > > >> > > > > > > > > > solely for the use of the individual to whom it is
> > > >> > addressed.
> > > >> > > > Any
> > > >> > > > > > > views
> > > >> > > > > > > > > or
> > > >> > > > > > > > > > opinions expressed are solely those of the author
> > and
> > > do
> > > >> > not
> > > >> > > > > > > > necessarily
> > > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > > companies.
> > > >> If
> > > >> > > you
> > > >> > > > > are
> > > >> > > > > > > not
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > intended recipient of this email, you must neither
> > > take
> > > >> any
> > > >> > > > > action
> > > >> > > > > > > > based
> > > >> > > > > > > > > > upon its contents, nor copy or show it to anyone.
> > > Please
> > > >> > > > contact
> > > >> > > > > > the
> > > >> > > > > > > > > sender
> > > >> > > > > > > > > > if you believe you have received this email in
> > error.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > -----Original Message-----
> > > >> > > > > > > > > > From: Syed Mushtaq [mailto:
> syed1.mushtaq@gmail.com
> > > >> > > > > <javascript:;> <javascript:;>]
> > > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > >> > <javascript:;>
> > > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > Snapshots
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Paul,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > When you say actual backups, how would it be
> > different
> > > >> from
> > > >> > > the
> > > >> > > > > > > Volume
> > > >> > > > > > > > > > Snapshots that exist currently. My understanding
> is
> > > that
> > > >> > > > Backups
> > > >> > > > > > end
> > > >> > > > > > > up
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> > point-in-time
> > > >> > state
> > > >> > > of
> > > >> > > > > > your
> > > >> > > > > > > > > volume
> > > >> > > > > > > > > > which can be restored back correct?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > -Syed
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > >> > > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi Syed,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > As I understand it, the SolidFire plugin will
> > export
> > > >> the
> > > >> > > > > snapshot
> > > >> > > > > > > to
> > > >> > > > > > > > > > > secondary storage if the user requests a
> template
> > > from
> > > >> > the
> > > >> > > > > > snapshot
> > > >> > > > > > > > or
> > > >> > > > > > > > > > > wants to download the snapshot from the cloud.
> > This
> > > >> is a
> > > >> > > > good,
> > > >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> > > storage
> > > >> is
> > > >> > > > super
> > > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays take
> up
> > > >> next
> > > >> > to
> > > >> > > no
> > > >> > > > > > > space.
> > > >> > > > > > > > > > > BUT I think that we are talking about a more
> > general
> > > >> > > purpose
> > > >> > > > > API,
> > > >> > > > > > > and
> > > >> > > > > > > > > > > other storage systems may not be as awesome as
> > > Mike's.
> > > >> > > That's
> > > >> > > > > my
> > > >> > > > > > > > > > > concern. Also, the time to transfer for say 1TB
> to
> > > >> move
> > > >> > > from
> > > >> > > > > > > primary
> > > >> > > > > > > > > > > to sec storage and then create a VM template out
> > of
> > > it
> > > >> > may
> > > >> > > be
> > > >> > > > > too
> > > >> > > > > > > > long
> > > >> > > > > > > > > > for users.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > > snapshot
> > > >> for
> > > >> > > > > backup,
> > > >> > > > > > > > it's
> > > >> > > > > > > > > > > just that users want to do backups and a volume
> > > >> snapshot
> > > >> > is
> > > >> > > > the
> > > >> > > > > > > only
> > > >> > > > > > > > > > > type of snapshot that copies the disk elsewhere
> > and
> > > >> can
> > > >> > be
> > > >> > > > > > > scheduled.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> > actual
> > > >> > backups
> > > >> > > > > > > (through
> > > >> > > > > > > > > > > recognised backup providers) and the user
> > > requirements
> > > >> > > around
> > > >> > > > > > them
> > > >> > > > > > > > > > > (particularly restoration use cases) as a
> separate
> > > >> thread
> > > >> > > of
> > > >> > > > > > work.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > Paul
> > > >> Angus
> > > >> > > VP
> > > >> > > > > > > > Technology
> > > >> > > > > > > > > > > , ShapeBlue
> > > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > >> > > > > > > > > > >
> > > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > >> > > > |
> > > >> > > > > m:
> > > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > >> > > <javascript:;> |
> > > >> > > > > t: @cloudyangus*
> > > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > >> <javascript:;>
> > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> 4HS
> > > UK
> > > >> > Shape
> > > >> > > > > Blue
> > > >> > > > > > > Ltd
> > > >> > > > > > > > > > > is a company incorporated in England & Wales.
> > > >> ShapeBlue
> > > >> > > > > Services
> > > >> > > > > > > > India
> > > >> > > > > > > > > > > LLP is a company incorporated in India and is
> > > operated
> > > >> > > under
> > > >> > > > > > > license
> > > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil
> Consultoria
> > > >> Ltda
> > > >> > is
> > > >> > > a
> > > >> > > > > > > company
> > > >> > > > > > > > > > > incorporated in Brasil and is operated under
> > license
> > > >> from
> > > >> > > > Shape
> > > >> > > > > > > Blue
> > > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company
> registered
> > by
> > > >> The
> > > >> > > > > Republic
> > > >> > > > > > > of
> > > >> > > > > > > > > > > South Africa and is traded under license from
> > Shape
> > > >> Blue
> > > >> > > Ltd.
> > > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > >> > > > > > > > > > > This email and any attachments to it may be
> > > >> confidential
> > > >> > > and
> > > >> > > > > are
> > > >> > > > > > > > > > > intended solely for the use of the individual to
> > > whom
> > > >> it
> > > >> > is
> > > >> > > > > > > > addressed.
> > > >> > > > > > > > > > > Any views or opinions expressed are solely those
> > of
> > > >> the
> > > >> > > > author
> > > >> > > > > > and
> > > >> > > > > > > do
> > > >> > > > > > > > > > > not necessarily represent those of Shape Blue
> Ltd
> > or
> > > >> > > related
> > > >> > > > > > > > > > > companies. If you are not the intended recipient
> > of
> > > >> this
> > > >> > > > email,
> > > >> > > > > > you
> > > >> > > > > > > > > > > must neither take any action based upon its
> > > contents,
> > > >> nor
> > > >> > > > copy
> > > >> > > > > or
> > > >> > > > > > > > show
> > > >> > > > > > > > > > > it to anyone. Please contact the sender if you
> > > believe
> > > >> > you
> > > >> > > > have
> > > >> > > > > > > > > received
> > > >> > > > > > > > > > this email in error.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > -----Original Message-----
> > > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> > syed1.mushtaq@gmail.com
> > > >> > > > > <javascript:;>
> > > >> > > > > > <javascript:;>]
> > > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > >> > > <javascript:;>
> > > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > > Snapshots
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I think the terminology confusion comes from AWS
> > > where
> > > >> > they
> > > >> > > > do
> > > >> > > > > > EBS
> > > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
> > > >> followed
> > > >> > > > that.
> > > >> > > > > > And
> > > >> > > > > > > > as
> > > >> > > > > > > > > > > an end user who is oblivious to the internals of
> > my
> > > >> > > provider,
> > > >> > > > > my
> > > >> > > > > > > > > > > expectation would be something similar to what
> AWS
> > > as
> > > >> > that
> > > >> > > is
> > > >> > > > > my
> > > >> > > > > > > > > > > biggest reference point.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > To your point Mike, I agree that a Primary
> Storage
> > > >> > failure
> > > >> > > on
> > > >> > > > > > > > > > > SolidFire is unlikely, there are other
> motivations
> > > >> for us
> > > >> > > to
> > > >> > > > > push
> > > >> > > > > > > > data
> > > >> > > > > > > > > > > to secondary storage. Primary storage (atleast
> for
> > > us)
> > > >> > > costs
> > > >> > > > > > > around 3
> > > >> > > > > > > > > > > times as much as secondary storage and snapshots
> > on
> > > >> > primary
> > > >> > > > > > storage
> > > >> > > > > > > > > > > are rarely used (especially for some of our
> > > customers
> > > >> who
> > > >> > > do
> > > >> > > > > > daily
> > > >> > > > > > > > > > backups).
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski
> <
> > > >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > >> > > <javascript:;>>
> > > >> > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Some of the weirdness is around terminology.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > For most systems I've worked on, a snapshot
> and
> > a
> > > >> > backup
> > > >> > > > are
> > > >> > > > > > two
> > > >> > > > > > > > > > > > completely different things (but CloudStack
> has
> > > >> > > > traditionally
> > > >> > > > > > > used
> > > >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
> > > >> though,
> > > >> > > that
> > > >> > > > > if
> > > >> > > > > > > your
> > > >> > > > > > > > > > > > primary storage is running on SolidFire that
> it
> > is
> > > >> > > unlikely
> > > >> > > > > > > you'll
> > > >> > > > > > > > > > > > encounter an issue where your primary storage
> > goes
> > > >> > > offline
> > > >> > > > > (and
> > > >> > > > > > > > > > > > you'll even maintain your performance
> guarantees
> > > >> during
> > > >> > > > > failure
> > > >> > > > > > > > > > > > scenarios and upgrades, as well). That being
> the
> > > >> case,
> > > >> > it
> > > >> > > > is
> > > >> > > > > > less
> > > >> > > > > > > > > > > > useful to require a backup to Swift (but it's
> > > >> perfectly
> > > >> > > OK
> > > >> > > > if
> > > >> > > > > > > > that's
> > > >> > > > > > > > > > > > what we want to do
> > > >> > > > > > > > > > > here).
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > > >> > <javascript:;>>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Paul,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > I believe with the current implementation of
> > > >> > Snapshots
> > > >> > > on
> > > >> > > > > > > managed
> > > >> > > > > > > > > > > > > storage
> > > >> > > > > > > > > > > > > (SolidFire) the snapshots are never exported
> > to
> > > >> the
> > > >> > > > > secondary
> > > >> > > > > > > > > > storage.
> > > >> > > > > > > > > > > > > While this solves the problem of having
> > > snapshots
> > > >> > > taking
> > > >> > > > > > > forever
> > > >> > > > > > > > > > > > > to get to sec storage, this leaves us with a
> > > >> > > > > > > > > > > > huge
> > > >> > > > > > > > > > > > > liability if our primary storage goes down.
> We
> > > see
> > > >> > > > > snapshots
> > > >> > > > > > as
> > > >> > > > > > > > > > > > > our recovery path because we store them in
> > Swift
> > > >> > which
> > > >> > > is
> > > >> > > > > > > > reliable
> > > >> > > > > > > > > > > > > and resilient to failures.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> > > Volume
> > > >> > > > snapshots
> > > >> > > > > > > > always
> > > >> > > > > > > > > > > > > backed up to secondary storage and Storage
> > > >> Snapshots
> > > >> > > stay
> > > >> > > > > on
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > primary
> > > >> > > > > > > > > > > > storage.
> > > >> > > > > > > > > > > > > A provider could potentially mix both these
> > and
> > > >> solve
> > > >> > > the
> > > >> > > > > > > problem
> > > >> > > > > > > > > > > > > that you mentioned where you want to meet
> > user's
> > > >> > > > > expectation
> > > >> > > > > > > of a
> > > >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while
> > having
> > > >> an
> > > >> > > > ability
> > > >> > > > > > to
> > > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the
> device)
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Hope this clarifies things.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > -Syed
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > >> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > >> > > <javascript:;>>
> > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > HI guys,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Could someone point me to the Jira bug of
> FS
> > > for
> > > >> > the
> > > >> > > > > > > > > > > > > > SAN-snapshot
> > > >> > > > > > > > > > > > feature
> > > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > From my discussions with users and
> operators
> > > >> around
> > > >> > > > > > snapshots
> > > >> > > > > > > > > > > > > > I'd make
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > following observations:
> > > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
> > > >> long-term
> > > >> > > and
> > > >> > > > > > short
> > > >> > > > > > > > > > > > > > term)
> > > >> > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > the expectation that they can use them for
> > > >> recovery
> > > >> > > if
> > > >> > > > > > > > required.
> > > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > > something
> > > >> > has
> > > >> > > > gone
> > > >> > > > > > > wrong
> > > >> > > > > > > > > > > > > > with primary storage.
> > > >> > > > > > > > > > > > > > c. users sometimes want to be able to
> export
> > > >> > > snapshots
> > > >> > > > as
> > > >> > > > > > > well
> > > >> > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > create
> > > >> > > > > > > > > > > > > > new VMs from their snapshots
> > > >> > > > > > > > > > > > > > d. snapshots are a currently a massive
> pain
> > > for
> > > >> > > > > operators,
> > > >> > > > > > I
> > > >> > > > > > > > > > > > > > know at
> > > >> > > > > > > > > > > > > least
> > > >> > > > > > > > > > > > > > one public cloud who have snapshots which
> > > take 2
> > > >> > days
> > > >> > > > to
> > > >> > > > > > > > > complete.
> > > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used
> for
> > > >> > multiple
> > > >> > > > LVM
> > > >> > > > > > > > disks.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > I think the process Mike has used in the
> > > >> SolidFire
> > > >> > > > plugin
> > > >> > > > > > > (only
> > > >> > > > > > > > > > > > > > moving
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > disk image to secondary storage when you
> > > >> absolutely
> > > >> > > > have
> > > >> > > > > > to)
> > > >> > > > > > > is
> > > >> > > > > > > > > > > > > > a very
> > > >> > > > > > > > > > > > > good
> > > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> > problems
> > > >> an
> > > >> > > > > operator
> > > >> > > > > > > > might
> > > >> > > > > > > > > > > > > experience
> > > >> > > > > > > > > > > > > > if they have an issue with a given primary
> > > >> storage
> > > >> > > pool
> > > >> > > > > in
> > > >> > > > > > a
> > > >> > > > > > > > > > cluster.
> > > >> > > > > > > > > > > > (I
> > > >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> > > >> SolidFire
> > > >> > > case
> > > >> > > > > > Mike
> > > >> > > > > > > :)
> > > >> > > > > > > > > > > > > > ) And
> > > >> > > > > > > > > > > > if
> > > >> > > > > > > > > > > > > > the transfer from primary to secondary is
> > > slow,
> > > >> the
> > > >> > > > time
> > > >> > > > > to
> > > >> > > > > > > > > > > > > > being able
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > create a template or export the volume
> will
> > be
> > > >> > slow.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > So for me the issue is around making sure
> > that
> > > >> the
> > > >> > > end
> > > >> > > > > > users
> > > >> > > > > > > > > > > > expectations
> > > >> > > > > > > > > > > > > > are met (while improving the
> > speed/efficiency
> > > of
> > > >> > the
> > > >> > > > back
> > > >> > > > > > > end)
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> > http://www.shapeblue.com>
> > > >> Paul
> > > >> > > > Angus
> > > >> > > > > > VP
> > > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603
> 0540*
> > > >> > > > > > > > > > > > > >
> > > >> > > > > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > >> > > > > > > |
> > > >> > > > > > > > m:
> > > >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com
> <javascript:;>
> > > >> > > > > <javascript:;> | t:
> > > >> > > > > > @cloudyangus*
> > > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > >> > > > <javascript:;>
> > > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> > http://www.shapeblue.com
> > > >
> > > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> > WC2N
> > > >> 4HS
> > > >> > UK
> > > >> > > > > Shape
> > > >> > > > > > > > Blue
> > > >> > > > > > > > > > > > > > Ltd is a company incorporated in England &
> > > >> Wales.
> > > >> > > > > ShapeBlue
> > > >> > > > > > > > > > > > > > Services India LLP is a company
> incorporated
> > > in
> > > >> > India
> > > >> > > > and
> > > >> > > > > > is
> > > >> > > > > > > > > > > > > > operated
> > > >> > > > > > > > > > > > > under
> > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> > Brasil
> > > >> > > > > Consultoria
> > > >> > > > > > > Ltda
> > > >> > > > > > > > > > > > > > is a company incorporated in Brasil and is
> > > >> operated
> > > >> > > > under
> > > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA
> > Pty
> > > >> Ltd
> > > >> > is
> > > >> > > a
> > > >> > > > > > > company
> > > >> > > > > > > > > > > > > > registered by The Republic
> > > >> > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > South Africa and is traded under license
> > from
> > > >> Shape
> > > >> > > > Blue
> > > >> > > > > > Ltd.
> > > >> > > > > > > > > > > > > > ShapeBlue
> > > >> > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > a registered trademark.
> > > >> > > > > > > > > > > > > > This email and any attachments to it may
> be
> > > >> > > > confidential
> > > >> > > > > > and
> > > >> > > > > > > > are
> > > >> > > > > > > > > > > > intended
> > > >> > > > > > > > > > > > > > solely for the use of the individual to
> whom
> > > it
> > > >> is
> > > >> > > > > > addressed.
> > > >> > > > > > > > > > > > > > Any views
> > > >> > > > > > > > > > > > > or
> > > >> > > > > > > > > > > > > > opinions expressed are solely those of the
> > > >> author
> > > >> > and
> > > >> > > > do
> > > >> > > > > > not
> > > >> > > > > > > > > > > > necessarily
> > > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or
> related
> > > >> > > companies.
> > > >> > > > > If
> > > >> > > > > > > you
> > > >> > > > > > > > > > > > > > are not
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > intended recipient of this email, you must
> > > >> neither
> > > >> > > take
> > > >> > > > > any
> > > >> > > > > > > > > > > > > > action
> > > >> > > > > > > > > > > > based
> > > >> > > > > > > > > > > > > > upon its contents, nor copy or show it to
> > > >> anyone.
> > > >> > > > Please
> > > >> > > > > > > > contact
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > sender
> > > >> > > > > > > > > > > > > > if you believe you have received this
> email
> > in
> > > >> > error.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > -----Original Message-----
> > > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > > >> pdion@cloudops.com
> > > >> > > > > <javascript:;>
> > > >> > > > > > <javascript:;>]
> > > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> > <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature]
> Storage
> > > >> > Snapshots
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Hi Mike,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > > >> StorageSnapshot
> > > >> > > for
> > > >> > > > > > > managed
> > > >> > > > > > > > > > > > > > storage
> > > >> > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > > expected,
> > > >> > > > behavior
> > > >> > > > > > is
> > > >> > > > > > > to
> > > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> > Storage.
> > > >> So a
> > > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > > >> > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > snapshot that remain on the managed
> storage
> > > >> > > appliance.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Quickly looking at the API doc and I don't
> > > see a
> > > >> > > strong
> > > >> > > > > > > > > > > > > > requirement for volume snapshots to be
> moved
> > > >> into
> > > >> > > > > secondary
> > > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is
> > not
> > > >> > useful,
> > > >> > > > but
> > > >> > > > > > > both
> > > >> > > > > > > > > > > > > > use cases are required. A snapshot that
> > remain
> > > >> on
> > > >> > the
> > > >> > > > > > managed
> > > >> > > > > > > > > > > > > > storage, and another type of
> > > >> > > > > > > > > > > > snapshot
> > > >> > > > > > > > > > > > > > that end up into the secondary storage.
> > Since
> > > >> > you've
> > > >> > > > > done a
> > > >> > > > > > > lot
> > > >> > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > work,
> > > >> > > > > > > > > > > > > > might easier to just add a parameter to
> the
> > > >> current
> > > >> > > > > > snapshot
> > > >> > > > > > > > API
> > > >> > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > trigger an extraction of the storage
> > snapshot
> > > >> into
> > > >> > > the
> > > >> > > > > > > > secondary
> > > >> > > > > > > > > > > > storage?
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > PL
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > > Tutkowski <
> > > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com
> <javascript:;>
> > > >> > > > > <javascript:;>
> > > >> > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > I think that all sounds reasonable then
> -
> > > >> thanks!
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > > Mushtaq <
> > > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > >> <javascript:;>>
> > > >> > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> > > >> > requirements.
> > > >> > > > One
> > > >> > > > > > of
> > > >> > > > > > > > our
> > > >> > > > > > > > > > > > earlier
> > > >> > > > > > > > > > > > > > >> iterations on this was to have an
> > argument
> > > to
> > > >> > the
> > > >> > > > > create
> > > >> > > > > > > > > > > > > > >> snapshot
> > > >> > > > > > > > > > > > API
> > > >> > > > > > > > > > > > > > >> which decides whether to backup the
> > volume
> > > to
> > > >> > sec
> > > >> > > > > > storage
> > > >> > > > > > > > but
> > > >> > > > > > > > > > > > > > >> we realized it would make management of
> > > >> > snapshots
> > > >> > > > > quite
> > > >> > > > > > > > messy
> > > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > >> > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> > Tutkowski
> > > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > > <javascript:;>
> > > >> > > > > <javascript:;>>
> > > >> > > > > > > > > > > > > > >> wrote:
> > > >> > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > >>> Hi,
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> Just to make sure I understand all the
> > > >> > > requirements
> > > >> > > > > > here:
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 1) This relates only to managed
> storage
> > > (1:1
> > > >> > > > mapping
> > > >> > > > > > > > between
> > > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> > volume).
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> > (introduced
> > > >> in
> > > >> > > 4.6)
> > > >> > > > > > > > > > > > > > >>> functionality, which creates a
> snapshot
> > on
> > > >> the
> > > >> > > SAN,
> > > >> > > > > and
> > > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > > >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> > > >> snapshot,
> > > >> > but
> > > >> > > > to
> > > >> > > > > > copy
> > > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to
> > > NFS.
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken.
> > It's
> > > >> the
> > > >> > > > backup
> > > >> > > > > to
> > > >> > > > > > > NFS
> > > >> > > > > > > > > > > > > > >>> that is optional.
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> > > >> snapshot
> > > >> > > > that's
> > > >> > > > > on
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > > >>> SAN (already works).
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created
> > from
> > > >> the
> > > >> > > > > snapshot
> > > >> > > > > > > > > > > > > > >>> that's on
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > >>> SAN (already works as long as the new
> > > >> > CloudStack
> > > >> > > > > volume
> > > >> > > > > > > > ends
> > > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > > snapshot
> > > >> API
> > > >> > > > then
> > > >> > > > > or
> > > >> > > > > > > > > > > > > > >>> would that just be the standard volume
> > > >> snapshot
> > > >> > > > > without
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > > > >>> backup to
> > > >> > > > > > > > > > > NFS?
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> Thanks!
> > > >> > > > > > > > > > > > > > >>> Mike
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> > > Mushtaq
> > > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com
> <javascript:;>
> > > >> > > > > <javascript:;>>
> > > >> > > > > > > > > > > > > > >>> wrote:
> > > >> > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > > functionalities
> > > >> > > > > (snapshot
> > > >> > > > > > on
> > > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we
> > > would
> > > >> > like
> > > >> > > > to
> > > >> > > > > > have
> > > >> > > > > > > > > both.
> > > >> > > > > > > > > > > > > > >>>> For example, some of our customers
> want
> > > to
> > > >> > > > implement
> > > >> > > > > > > their
> > > >> > > > > > > > > > > > > > >>>> own backup strategies and do
> encryption
> > > to
> > > >> > their
> > > >> > > > > > backups
> > > >> > > > > > > > > > > > > > >>>> which is a perfect use case for
> Storage
> > > >> > Snapshot
> > > >> > > > > while
> > > >> > > > > > > our
> > > >> > > > > > > > > > > > > > >>>> other customers will still keep using
> > the
> > > >> > > standard
> > > >> > > > > > > volume
> > > >> > > > > > > > > > > snapshot.
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> To keep things backward compatible,
> we
> > > can
> > > >> > add a
> > > >> > > > > > setting
> > > >> > > > > > > > > > > > > > >>>> which
> > > >> > > > > > > > > > > > says
> > > >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> > > >> because,
> > > >> > > after
> > > >> > > > > > all,
> > > >> > > > > > > > you
> > > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when
> > > doing
> > > >> a
> > > >> > > > Volume
> > > >> > > > > > > > > Snapshot.
> > > >> > > > > > > > > > > > > > >>>> You could stop the process there and
> > not
> > > do
> > > >> > the
> > > >> > > > > > upload.
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> What do you think about this
> approach?
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> Thanks,
> > > >> > > > > > > > > > > > > > >>>> -Syed
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> > > >> > Tutkowski <
> > > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>
> > > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out
> load
> > > >> here,
> > > >> > but
> > > >> > > > if
> > > >> > > > > a
> > > >> > > > > > > > given
> > > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually
> need
> > > to
> > > >> > > provide
> > > >> > > > > > both
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > ability
> > > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it
> > to
> > > >> NFS
> > > >> > (if
> > > >> > > > > just
> > > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then
> we
> > > >> might
> > > >> > be
> > > >> > > > able
> > > >> > > > > > to
> > > >> > > > > > > > get
> > > >> > > > > > > > > > > > > > >>>>> away with no new
> > > >> > > > > > > > > > > > API
> > > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> > custom
> > > >> > > snapshot
> > > >> > > > > > > strategy
> > > >> > > > > > > > > > > > > > >>>>> and
> > > >> > > > > > > > > > > > data
> > > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case
> > where
> > > >> the
> > > >> > > > > > CloudStack
> > > >> > > > > > > > > > > > > > >>>>> cloud
> > > >> > > > > > > > > > > > does
> > > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > > >> exported-to-NFS
> > > >> > > > > backup.
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> In other words, the "default"
> behavior
> > > >> would
> > > >> > be
> > > >> > > > to
> > > >> > > > > > use
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> > > strategy
> > > >> > that
> > > >> > > > we
> > > >> > > > > > > > already
> > > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> > > >> > snapshot).
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however,
> > wants
> > > >> to
> > > >> > > take
> > > >> > > > a
> > > >> > > > > > SAN
> > > >> > > > > > > > > > > > > > >>>>> snapshot and have the data exported
> to
> > > >> NFS,
> > > >> > > then
> > > >> > > > we
> > > >> > > > > > > could
> > > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config
> > file
> > > to
> > > >> > make
> > > >> > > > use
> > > >> > > > > > of
> > > >> > > > > > > a
> > > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data
> motion
> > > >> > strategy
> > > >> > > > that
> > > >> > > > > > > > > > > > > > >>>>> performs both of these
> > > >> > > > > > > > > > > > > > activities.
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is still
> > the
> > > >> > default
> > > >> > > > for
> > > >> > > > > > > > users,
> > > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change
> this
> > > >> > behavior
> > > >> > > > via
> > > >> > > > > > > > > > > configuration.
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM,
> Mike
> > > >> > > Tutkowski <
> > > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > > >> <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>
> > > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to
> come
> > up
> > > >> > with a
> > > >> > > > > > viable
> > > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way
> > for
> > > >> them
> > > >> > > to
> > > >> > > > > move
> > > >> > > > > > > > from
> > > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and
> some
> > > >> obvious
> > > >> > > way
> > > >> > > > > > that
> > > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > do this).
> > > >> > > > > > > > > > > > > > >>>>>>
> > > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM,
> Syed
> > > >> > Mushtaq
> > > >> > > <
> > > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> > <javascript:;>
> > > >> > > > > <javascript:;>> wrote:
> > > >> > > > > > > > > > > > > > >>>>>>
> > > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> > upgrade
> > > >> path
> > > >> > > > > however,
> > > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and
> are
> > > on a
> > > >> > > > managed
> > > >> > > > > > > > storage
> > > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > > >> functionality
> > > >> > > > with
> > > >> > > > > > > Volume
> > > >> > > > > > > > >
>
>
>
> --
> Ian Rae
> CEO | PDG
> c: 514.944.4008
>
> CloudOps | Cloud Infrastructure and Networking Solutions
> www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Ian Rae <ir...@cloudops.com>]

I would hope that default behaviour in CloudStack is the traditional volume
snapshot moved to secondary storage. A global setting to change that
behaviour is probably ok if it is not default, but the user would want to
in certain cases make copies of those snapshots to secondary storage in
addition to taking the snapshot on the primary storage.

On Monday, February 8, 2016, Will Stevens <ws...@cloudops.com> wrote:

> I don't think a global setting is a good option because we need both
> functionality to be available at the same time and for different use cases
> to be able to pick which they choose.
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com <javascript:;>
> > wrote:
>
> > Now that I re-read your e-mail, it dawned on me: The end-user doesn't
> care
> > where the snapshot is.
> >
> > If that's true, then we should perhaps control this via Global Settings
> or
> > something.
> >
> > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com> wrote:
> >
> > > It's not ideal - true, but it does allow us to be backward compatible.
> > >
> > > If you have other ideas, though, about how to maintain backward
> > > compatibility, I'm definitely open to hear them.
> > >
> > > Thanks!
> > >
> > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <syed1.mushtaq@gmail.com
> >
> > > wrote:
> > >
> > >> Hi Mike,
> > >>
> > >> Adding a flag to createSnapshot was the first and the most obvious
> thing
> > >> that came to our minds. The problem that I had with this was that:
> > >>
> > >> 1) I feel it is exposing something to the end user that is internal to
> > the
> > >> cloud.
> > >>
> > >> 2) We have to follow two different ways of restore/deletion in the
> same
> > >> code path depending on where the Snapshot resides which I find kind
> of a
> > >> bad design.
> > >>
> > >> But if exposing a archive flag to the end user is acceptable then we
> can
> > >> definitely use this instead of adding the StorageSnapshot API
> > >>
> > >> Thanks,
> > >> -Syed
> > >>
> > >>
> > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > >> mike.tutkowski@solidfire.com
> > >> > wrote:
> > >>
> > >> > Hi Pierre-Luc,
> > >> >
> > >> > My recommendation would be this:
> > >> >
> > >> > Add an "archive" flag to the current volume-snapshot API. Its
> default
> > >> would
> > >> > be "false" because that would be backward compatible with how 4.6
> has
> > >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6, 4.7,
> > and
> > >> > 4.8).
> > >> >
> > >> > If you set archive=true, then we would perform a background
> migration
> > of
> > >> > the snapshot from the SAN to the secondary storage (then delete the
> > SAN
> > >> > snapshot).
> > >> >
> > >> > That archive parameter would only be valid for managed storage.
> > >> >
> > >> > Sound reasonable?
> > >> >
> > >> > Also, a VM snapshot that includes disks provided by managed storage
> > >> should
> > >> > work.
> > >> >
> > >> > Talk to you later,
> > >> > Mike
> > >> >
> > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <pdion@cloudops.com
> >
> > >> > wrote:
> > >> >
> > >> > > Mike,
> > >> > >
> > >> > > In terms of API's, would you prefer introducing a parameter to the
> > >> > existing
> > >> > > VolumeSnapshot, example:   extract={true|false}  with a default
> > value
> > >> of
> > >> > > true  which would extract snapshot into the secondary storage,
> which
> > >> is
> > >> > the
> > >> > > current default behavior. Then for SAN snapshot that remain on the
> > >> SAN we
> > >> > > would just set "extract=false" ?  as oppose to create a new
> > >> > >  StorageSnapshot API ?
> > >> > >
> > >> > >
> > >> > > Paul,
> > >> > >
> > >> > > From what I'm seeing so far, we can't do a VM-snapshot when using
> > >> managed
> > >> > > storage for VM having more than one Volume. For the reason that
> > >> snapshot
> > >> > > are performed outside of the hypervisor awareness and
> > asynchronously.
> > >> If
> > >> > > someone have a way to address that, it would make thinks much more
> > >> > > attractive.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com>
> wrote:
> > >> > >
> > >> > > > I think a service provider backup scenario is more likely to
> take
> > >> > > advantage
> > >> > > > of SAN snapshot. There are a few reasons for this. Traditional
> > >> backups
> > >> > > > involve access to the file system, and there is an expectation
> > that
> > >> > this
> > >> > > > can be done with reasonably short time frames without negatively
> > >> > > impacting
> > >> > > > VM performance, and that the backup orchestrator can apply
> various
> > >> > logic
> > >> > > > and or transformations to the data (compress, encrypt, deltas
> > >> etc...).
> > >> > > > While it is true that one could apply a backup process to a
> cloud
> > >> > > snapshot,
> > >> > > > this would be slow and inefficient requiring the data to be
> moved
> > >> > several
> > >> > > > times and there are some major bottlenecks with cloud snapshots.
> > >> With a
> > >> > > > cloud snapshot - there seems to be no reasonable expectation of
> > >> being
> > >> > > able
> > >> > > > to do differential snapshots (I think this depends on the
> > >> hypervisor)
> > >> > and
> > >> > > > if you do differential snapshots this will make file backups
> from
> > >> those
> > >> > > > snapshots even more complicated to orchestrate.
> > >> > > >
> > >> > > > Suspect there needs to be a different thread on how to better
> > enable
> > >> > > > backups, as a service. As per Paul's suggestion, but it is a
> > related
> > >> > > > workflow so relevant to this discussion.
> > >> > > >
> > >> > > > Ian
> > >> > > >
> > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > >> > > mike.tutkowski@solidfire.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > To me it sounds like number two and number three are different
> > >> uses
> > >> > for
> > >> > > > the
> > >> > > > > same "thing"(which is totally fine).
> > >> > > > >
> > >> > > > > As for taking a fast SAN snapshot and exporting it
> > >> asynchronously, do
> > >> > > we
> > >> > > > > see the SSVM as performing the export?
> > >> > > > >
> > >> > > > > To be backwards compatible with what we have in 4.6 and later
> > for
> > >> > > volume
> > >> > > > > snapshots for managed storage, I think it might be easier if
> we
> > >> pass
> > >> > > in a
> > >> > > > > flag that says whether or not to archive the SAN snapshot
> > (which,
> > >> I
> > >> > > > think,
> > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > >> > > > >
> > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > pdion@cloudops.com
> > >> > > > > <javascript:;>> wrote:
> > >> > > > >
> > >> > > > > > Hi Mike,
> > >> > > > > >
> > >> > > > > > The reason behind the creation of a SAN snapshot which is
> > >> exported
> > >> > > into
> > >> > > > > > secondary storage, is because creating a copy of the .VHD
> > >> directly
> > >> > > > would
> > >> > > > > > impact uptime of the VM as creating that copy take lots of
> > time.
> > >> > Has
> > >> > > > > oppose
> > >> > > > > > to a SAN snapshot that is close to instantaneous which can
> > >> > afterward
> > >> > > be
> > >> > > > > > clone into Secondary Storage asynchronously.
> > >> > > > > >
> > >> > > > > > I would suspect an extracted VolumeSnapshot taken from a SAN
> > >> > snapshot
> > >> > > > > could
> > >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> > >> consumption
> > >> > > on
> > >> > > > > the
> > >> > > > > > Primary Storage side.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > I see 3 definitions in our current discussion regarding the
> > term
> > >> > > > snapshot
> > >> > > > > > (these are not official terminology but by own
> interpretation
> > of
> > >> > > them):
> > >> > > > > >
> > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of
> a
> > >> > > > snapshot):
> > >> > > > > > it's a volume snapshot at the storage level, point in time
> of
> > >> your
> > >> > > > data.
> > >> > > > > it
> > >> > > > > > reside on the primary storage. Useful and efficient for
> > software
> > >> > side
> > >> > > > > > incident.
> > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud
> > >> backup
> > >> > > > aws-S3
> > >> > > > > > style ): Point in time copy of the Virtual Disk that reside
> > on a
> > >> > > > > different
> > >> > > > > > storage array then the original Volume. Facilitate data
> > >> migration
> > >> > > > between
> > >> > > > > > clusters and, in case of primary storage incident, Volume
> > >> snapshots
> > >> > > are
> > >> > > > > not
> > >> > > > > > impacted and can be reuse.
> > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that
> also
> > >> > > validate
> > >> > > > > data
> > >> > > > > > integrity, provide a storage efficient archiving method and
> an
> > >> > > > > independent
> > >> > > > > > way to restore your data in case of an major infrastructure
> > >> > disaster.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Regards,
> > >> > > > > >
> > >> > > > > > PL
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > >> > > > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > >> > > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > So, let's see if I currently follow the requirements:
> > >> > > > > > >
> > >> > > > > > > * Augment volume snapshots for managed storage to
> > >> conditionally
> > >> > > > export
> > >> > > > > > data
> > >> > > > > > > to NFS. The current process of taking a snapshot on the
> SAN
> > is
> > >> > > fine,
> > >> > > > > but
> > >> > > > > > > we'd like the option to export the data to NFS, as well.
> > >> > > > > > >
> > >> > > > > > > Questions:
> > >> > > > > > >
> > >> > > > > > > Once the data has been exported to NFS, do we keep the SAN
> > >> > snapshot
> > >> > > > or
> > >> > > > > > > delete it?
> > >> > > > > > >
> > >> > > > > > > If we are deleting the SAN snapshot, then why don't we
> just
> > >> copy
> > >> > > the
> > >> > > > > VHD
> > >> > > > > > > from primary to secondary the way we do today for
> > non-managed
> > >> > (i.e.
> > >> > > > > > > traditional) storage? Why create a SAN snapshot in that
> > >> scenario?
> > >> > > > > Perhaps
> > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> secondary
> > >> > > storage
> > >> > > > > > > instead of a XenServer host?
> > >> > > > > > >
> > >> > > > > > > Thanks for the clarification.
> > >> > > > > > >
> > >> > > > > > > By the way, to me a backup is when you copy data from one
> > >> storage
> > >> > > > > system
> > >> > > > > > to
> > >> > > > > > > another (regardless of features, if any, to restore the
> data
> > >> in
> > >> > the
> > >> > > > > > > future). A snapshot is a point-in-time view of the data
> of a
> > >> > volume
> > >> > > > and
> > >> > > > > > > it's stored on the same storage system as the volume.
> > >> > > > > > >
> > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > >> > > pdion@cloudops.com
> > >> > > > > <javascript:;>
> > >> > > > > > <javascript:;>>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > That's fun to see that discussion happening. I 100%
> agree
> > >> with
> > >> > > > Paul's
> > >> > > > > > > > points of view. VolumeSnapshot are not a backup, but I
> do
> > >> > > consider
> > >> > > > > them
> > >> > > > > > > as
> > >> > > > > > > > a safety vest against Primary Storage failure, because
> > >> failure
> > >> > > > append
> > >> > > > > > > :-( .
> > >> > > > > > > >
> > >> > > > > > > > The current proposal around snapshots that reside on the
> > >> > primary
> > >> > > > > > storage
> > >> > > > > > > or
> > >> > > > > > > > snapshots that end in the Secondary Storage  is not to
> > >> address
> > >> > > any
> > >> > > > > kind
> > >> > > > > > > of
> > >> > > > > > > > backups requirement because a snapshot is not a backup,
> > >> event
> > >> > an
> > >> > > > > > > extracted
> > >> > > > > > > > VM snapshot.
> > >> > > > > > > >
> > >> > > > > > > > The main idea, and again this is for managed storage;
> > >> > > > > > > >
> > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> > >> capability
> > >> > > for
> > >> > > > > > fast
> > >> > > > > > > > response time that support rollback to previous
> timestamp,
> > >> > create
> > >> > > > new
> > >> > > > > > > > volume and maybe create template.
> > >> > > > > > > >     not required to be a new API if the work is already
> > >> done, I
> > >> > > > think
> > >> > > > > > > this
> > >> > > > > > > > is a different behaviors than the user expectation of a
> > >> > > > > > volume-snapshot.
> > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> behavior
> > >> that
> > >> > > > create
> > >> > > > > > an
> > >> > > > > > > > extraction of a volume into SecondaryStorage which can
> be
> > >> reuse
> > >> > > to
> > >> > > > > > > create a
> > >> > > > > > > > new volume into another Primary Storage. This type of
> > >> snapshot
> > >> > > is a
> > >> > > > > > slow
> > >> > > > > > > > job since yes it would have to copy the full volume size
> > on
> > >> the
> > >> > > > > > Secondary
> > >> > > > > > > > storage.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > PL
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > >> > > > > > <javascript:;>>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > I think I share you view on the 'Ideal world'. Backup
> > (via
> > >> > > Volume
> > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is
> > >> > > amplified
> > >> > > > > > > > especially
> > >> > > > > > > > > when you have a object storage as your secondary
> storage
> > >> > > because
> > >> > > > it
> > >> > > > > > > > > requires two copies (one to an NFS staging area and
> from
> > >> > there
> > >> > > to
> > >> > > > > > > object
> > >> > > > > > > > > storage). And not to mention that all these copies are
> > >> > > consuming
> > >> > > > > > > > hypervisor
> > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge bottleneck
> as
> > >> all
> > >> > > the
> > >> > > > > > > Network
> > >> > > > > > > > > and I/O flow through it. So our intention of proposing
> > the
> > >> > > > "Storage
> > >> > > > > > > > > Snapshots" is to give a better way of achiving
> snapshots
> > >> > while
> > >> > > > > still
> > >> > > > > > > > > keeping the original definition of volume snpashots
> (ie
> > >> > upload
> > >> > > to
> > >> > > > > sec
> > >> > > > > > > > > storage).
> > >> > > > > > > > >
> > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > backups.
> > >> > They
> > >> > > > > don't
> > >> > > > > > > work
> > >> > > > > > > > > form multi-disk LVM volume groups and dynamic disks. I
> > am
> > >> all
> > >> > > in
> > >> > > > > for
> > >> > > > > > a
> > >> > > > > > > > > better backup solution which handles these use cases
> and
> > >> > > utilizes
> > >> > > > > the
> > >> > > > > > > > > storage's advanced features.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > >> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > In the beginning... there were CloudStack snapshots
> > and
> > >> > they
> > >> > > > were
> > >> > > > > > > > > actually
> > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > snapshots.
> > >> > > > > > > > > > Then VM snapshots were created (which are
> > point-in-time
> > >> > > > > hypervisor
> > >> > > > > > > > > > snapshots) and we started referring to the original
> > >> > snapshots
> > >> > > > as
> > >> > > > > > > volume
> > >> > > > > > > > > > snapshots.
> > >> > > > > > > > > >
> > >> > > > > > > > > > CloudStack does not offer 'backups', but many people
> > use
> > >> > > volume
> > >> > > > > > > > snapshots
> > >> > > > > > > > > > as backups. However you can't in-place restore
> volume
> > >> > > snapshots
> > >> > > > > and
> > >> > > > > > > if
> > >> > > > > > > > > you
> > >> > > > > > > > > > have a VM with multiple volumes, the volume
> snapshots
> > >> must
> > >> > be
> > >> > > > > done
> > >> > > > > > in
> > >> > > > > > > > > > series, meaning that the state across all of the
> > >> volumes is
> > >> > > > > > unlikely
> > >> > > > > > > to
> > >> > > > > > > > > be
> > >> > > > > > > > > > consistent.
> > >> > > > > > > > > >
> > >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> > options
> > >> > > which
> > >> > > > > > users
> > >> > > > > > > > > might
> > >> > > > > > > > > > expect as well options as to where they might be
> > >> stored. In
> > >> > > my
> > >> > > > > > ideal
> > >> > > > > > > > > world
> > >> > > > > > > > > > they would also be able to leverage back-end
> hardware
> > >> (such
> > >> > > as
> > >> > > > > > > > Solidfire,
> > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> Commvault
> > >> etc
> > >> > to
> > >> > > > > > > accelerate
> > >> > > > > > > > > the
> > >> > > > > > > > > > process.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > >> > > > > > > > > > Paul Angus
> > >> > > > > > > > > > VP Technology ,  ShapeBlue
> > >> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > >> > > > > > > > > >
> > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> > >> > > > > m:
> > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > >> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> > >> > <javascript:;>
> > >> > > |
> > >> > > > > t: @cloudyangus*
> > >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > >> > > > > <ja...@cloudyangus>
> > >> > > > > > |  w:
> > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > >> > > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS
> UK
> > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in England
> &
> > >> > Wales.
> > >> > > > > > > ShapeBlue
> > >> > > > > > > > > > Services India LLP is a company incorporated in
> India
> > >> and
> > >> > is
> > >> > > > > > operated
> > >> > > > > > > > > under
> > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > >> Consultoria
> > >> > > Ltda
> > >> > > > > is
> > >> > > > > > a
> > >> > > > > > > > > > company incorporated in Brasil and is operated under
> > >> > license
> > >> > > > from
> > >> > > > > > > Shape
> > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> registered
> > >> by
> > >> > The
> > >> > > > > > > Republic
> > >> > > > > > > > of
> > >> > > > > > > > > > South Africa and is traded under license from Shape
> > Blue
> > >> > Ltd.
> > >> > > > > > > ShapeBlue
> > >> > > > > > > > > is
> > >> > > > > > > > > > a registered trademark.
> > >> > > > > > > > > > This email and any attachments to it may be
> > confidential
> > >> > and
> > >> > > > are
> > >> > > > > > > > intended
> > >> > > > > > > > > > solely for the use of the individual to whom it is
> > >> > addressed.
> > >> > > > Any
> > >> > > > > > > views
> > >> > > > > > > > > or
> > >> > > > > > > > > > opinions expressed are solely those of the author
> and
> > do
> > >> > not
> > >> > > > > > > > necessarily
> > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > companies.
> > >> If
> > >> > > you
> > >> > > > > are
> > >> > > > > > > not
> > >> > > > > > > > > the
> > >> > > > > > > > > > intended recipient of this email, you must neither
> > take
> > >> any
> > >> > > > > action
> > >> > > > > > > > based
> > >> > > > > > > > > > upon its contents, nor copy or show it to anyone.
> > Please
> > >> > > > contact
> > >> > > > > > the
> > >> > > > > > > > > sender
> > >> > > > > > > > > > if you believe you have received this email in
> error.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > -----Original Message-----
> > >> > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > >> > > > > <javascript:;> <javascript:;>]
> > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > >> > <javascript:;>
> > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> Snapshots
> > >> > > > > > > > > >
> > >> > > > > > > > > > Paul,
> > >> > > > > > > > > >
> > >> > > > > > > > > > When you say actual backups, how would it be
> different
> > >> from
> > >> > > the
> > >> > > > > > > Volume
> > >> > > > > > > > > > Snapshots that exist currently. My understanding is
> > that
> > >> > > > Backups
> > >> > > > > > end
> > >> > > > > > > up
> > >> > > > > > > > > in
> > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> point-in-time
> > >> > state
> > >> > > of
> > >> > > > > > your
> > >> > > > > > > > > volume
> > >> > > > > > > > > > which can be restored back correct?
> > >> > > > > > > > > >
> > >> > > > > > > > > > -Syed
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > >> > > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Syed,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > As I understand it, the SolidFire plugin will
> export
> > >> the
> > >> > > > > snapshot
> > >> > > > > > > to
> > >> > > > > > > > > > > secondary storage if the user requests a template
> > from
> > >> > the
> > >> > > > > > snapshot
> > >> > > > > > > > or
> > >> > > > > > > > > > > wants to download the snapshot from the cloud.
> This
> > >> is a
> > >> > > > good,
> > >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> > storage
> > >> is
> > >> > > > super
> > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays take up
> > >> next
> > >> > to
> > >> > > no
> > >> > > > > > > space.
> > >> > > > > > > > > > > BUT I think that we are talking about a more
> general
> > >> > > purpose
> > >> > > > > API,
> > >> > > > > > > and
> > >> > > > > > > > > > > other storage systems may not be as awesome as
> > Mike's.
> > >> > > That's
> > >> > > > > my
> > >> > > > > > > > > > > concern. Also, the time to transfer for say 1TB to
> > >> move
> > >> > > from
> > >> > > > > > > primary
> > >> > > > > > > > > > > to sec storage and then create a VM template out
> of
> > it
> > >> > may
> > >> > > be
> > >> > > > > too
> > >> > > > > > > > long
> > >> > > > > > > > > > for users.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > snapshot
> > >> for
> > >> > > > > backup,
> > >> > > > > > > > it's
> > >> > > > > > > > > > > just that users want to do backups and a volume
> > >> snapshot
> > >> > is
> > >> > > > the
> > >> > > > > > > only
> > >> > > > > > > > > > > type of snapshot that copies the disk elsewhere
> and
> > >> can
> > >> > be
> > >> > > > > > > scheduled.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> actual
> > >> > backups
> > >> > > > > > > (through
> > >> > > > > > > > > > > recognised backup providers) and the user
> > requirements
> > >> > > around
> > >> > > > > > them
> > >> > > > > > > > > > > (particularly restoration use cases) as a separate
> > >> thread
> > >> > > of
> > >> > > > > > work.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> Paul
> > >> Angus
> > >> > > VP
> > >> > > > > > > > Technology
> > >> > > > > > > > > > > , ShapeBlue
> > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > >> > > > > > > > > > >
> > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > >> > > > |
> > >> > > > > m:
> > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > >> > > <javascript:;> |
> > >> > > > > t: @cloudyangus*
> > >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > >> <javascript:;>
> > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS
> > UK
> > >> > Shape
> > >> > > > > Blue
> > >> > > > > > > Ltd
> > >> > > > > > > > > > > is a company incorporated in England & Wales.
> > >> ShapeBlue
> > >> > > > > Services
> > >> > > > > > > > India
> > >> > > > > > > > > > > LLP is a company incorporated in India and is
> > operated
> > >> > > under
> > >> > > > > > > license
> > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria
> > >> Ltda
> > >> > is
> > >> > > a
> > >> > > > > > > company
> > >> > > > > > > > > > > incorporated in Brasil and is operated under
> license
> > >> from
> > >> > > > Shape
> > >> > > > > > > Blue
> > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered
> by
> > >> The
> > >> > > > > Republic
> > >> > > > > > > of
> > >> > > > > > > > > > > South Africa and is traded under license from
> Shape
> > >> Blue
> > >> > > Ltd.
> > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > >> > > > > > > > > > > This email and any attachments to it may be
> > >> confidential
> > >> > > and
> > >> > > > > are
> > >> > > > > > > > > > > intended solely for the use of the individual to
> > whom
> > >> it
> > >> > is
> > >> > > > > > > > addressed.
> > >> > > > > > > > > > > Any views or opinions expressed are solely those
> of
> > >> the
> > >> > > > author
> > >> > > > > > and
> > >> > > > > > > do
> > >> > > > > > > > > > > not necessarily represent those of Shape Blue Ltd
> or
> > >> > > related
> > >> > > > > > > > > > > companies. If you are not the intended recipient
> of
> > >> this
> > >> > > > email,
> > >> > > > > > you
> > >> > > > > > > > > > > must neither take any action based upon its
> > contents,
> > >> nor
> > >> > > > copy
> > >> > > > > or
> > >> > > > > > > > show
> > >> > > > > > > > > > > it to anyone. Please contact the sender if you
> > believe
> > >> > you
> > >> > > > have
> > >> > > > > > > > > received
> > >> > > > > > > > > > this email in error.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > -----Original Message-----
> > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> syed1.mushtaq@gmail.com
> > >> > > > > <javascript:;>
> > >> > > > > > <javascript:;>]
> > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > >> > > <javascript:;>
> > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > Snapshots
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I think the terminology confusion comes from AWS
> > where
> > >> > they
> > >> > > > do
> > >> > > > > > EBS
> > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
> > >> followed
> > >> > > > that.
> > >> > > > > > And
> > >> > > > > > > > as
> > >> > > > > > > > > > > an end user who is oblivious to the internals of
> my
> > >> > > provider,
> > >> > > > > my
> > >> > > > > > > > > > > expectation would be something similar to what AWS
> > as
> > >> > that
> > >> > > is
> > >> > > > > my
> > >> > > > > > > > > > > biggest reference point.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > To your point Mike, I agree that a Primary Storage
> > >> > failure
> > >> > > on
> > >> > > > > > > > > > > SolidFire is unlikely, there are other motivations
> > >> for us
> > >> > > to
> > >> > > > > push
> > >> > > > > > > > data
> > >> > > > > > > > > > > to secondary storage. Primary storage (atleast for
> > us)
> > >> > > costs
> > >> > > > > > > around 3
> > >> > > > > > > > > > > times as much as secondary storage and snapshots
> on
> > >> > primary
> > >> > > > > > storage
> > >> > > > > > > > > > > are rarely used (especially for some of our
> > customers
> > >> who
> > >> > > do
> > >> > > > > > daily
> > >> > > > > > > > > > backups).
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > >> > > <javascript:;>>
> > >> > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Some of the weirdness is around terminology.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > For most systems I've worked on, a snapshot and
> a
> > >> > backup
> > >> > > > are
> > >> > > > > > two
> > >> > > > > > > > > > > > completely different things (but CloudStack has
> > >> > > > traditionally
> > >> > > > > > > used
> > >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
> > >> though,
> > >> > > that
> > >> > > > > if
> > >> > > > > > > your
> > >> > > > > > > > > > > > primary storage is running on SolidFire that it
> is
> > >> > > unlikely
> > >> > > > > > > you'll
> > >> > > > > > > > > > > > encounter an issue where your primary storage
> goes
> > >> > > offline
> > >> > > > > (and
> > >> > > > > > > > > > > > you'll even maintain your performance guarantees
> > >> during
> > >> > > > > failure
> > >> > > > > > > > > > > > scenarios and upgrades, as well). That being the
> > >> case,
> > >> > it
> > >> > > > is
> > >> > > > > > less
> > >> > > > > > > > > > > > useful to require a backup to Swift (but it's
> > >> perfectly
> > >> > > OK
> > >> > > > if
> > >> > > > > > > > that's
> > >> > > > > > > > > > > > what we want to do
> > >> > > > > > > > > > > here).
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > >> > <javascript:;>>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Paul,
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > I believe with the current implementation of
> > >> > Snapshots
> > >> > > on
> > >> > > > > > > managed
> > >> > > > > > > > > > > > > storage
> > >> > > > > > > > > > > > > (SolidFire) the snapshots are never exported
> to
> > >> the
> > >> > > > > secondary
> > >> > > > > > > > > > storage.
> > >> > > > > > > > > > > > > While this solves the problem of having
> > snapshots
> > >> > > taking
> > >> > > > > > > forever
> > >> > > > > > > > > > > > > to get to sec storage, this leaves us with a
> > >> > > > > > > > > > > > huge
> > >> > > > > > > > > > > > > liability if our primary storage goes down. We
> > see
> > >> > > > > snapshots
> > >> > > > > > as
> > >> > > > > > > > > > > > > our recovery path because we store them in
> Swift
> > >> > which
> > >> > > is
> > >> > > > > > > > reliable
> > >> > > > > > > > > > > > > and resilient to failures.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> > Volume
> > >> > > > snapshots
> > >> > > > > > > > always
> > >> > > > > > > > > > > > > backed up to secondary storage and Storage
> > >> Snapshots
> > >> > > stay
> > >> > > > > on
> > >> > > > > > > the
> > >> > > > > > > > > > > > > primary
> > >> > > > > > > > > > > > storage.
> > >> > > > > > > > > > > > > A provider could potentially mix both these
> and
> > >> solve
> > >> > > the
> > >> > > > > > > problem
> > >> > > > > > > > > > > > > that you mentioned where you want to meet
> user's
> > >> > > > > expectation
> > >> > > > > > > of a
> > >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while
> having
> > >> an
> > >> > > > ability
> > >> > > > > > to
> > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the device)
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hope this clarifies things.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > -Syed
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > >> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > >> > > <javascript:;>>
> > >> > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > HI guys,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Could someone point me to the Jira bug of FS
> > for
> > >> > the
> > >> > > > > > > > > > > > > > SAN-snapshot
> > >> > > > > > > > > > > > feature
> > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > From my discussions with users and operators
> > >> around
> > >> > > > > > snapshots
> > >> > > > > > > > > > > > > > I'd make
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > following observations:
> > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
> > >> long-term
> > >> > > and
> > >> > > > > > short
> > >> > > > > > > > > > > > > > term)
> > >> > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > the expectation that they can use them for
> > >> recovery
> > >> > > if
> > >> > > > > > > > required.
> > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > something
> > >> > has
> > >> > > > gone
> > >> > > > > > > wrong
> > >> > > > > > > > > > > > > > with primary storage.
> > >> > > > > > > > > > > > > > c. users sometimes want to be able to export
> > >> > > snapshots
> > >> > > > as
> > >> > > > > > > well
> > >> > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > create
> > >> > > > > > > > > > > > > > new VMs from their snapshots
> > >> > > > > > > > > > > > > > d. snapshots are a currently a massive pain
> > for
> > >> > > > > operators,
> > >> > > > > > I
> > >> > > > > > > > > > > > > > know at
> > >> > > > > > > > > > > > > least
> > >> > > > > > > > > > > > > > one public cloud who have snapshots which
> > take 2
> > >> > days
> > >> > > > to
> > >> > > > > > > > > complete.
> > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used for
> > >> > multiple
> > >> > > > LVM
> > >> > > > > > > > disks.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I think the process Mike has used in the
> > >> SolidFire
> > >> > > > plugin
> > >> > > > > > > (only
> > >> > > > > > > > > > > > > > moving
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > disk image to secondary storage when you
> > >> absolutely
> > >> > > > have
> > >> > > > > > to)
> > >> > > > > > > is
> > >> > > > > > > > > > > > > > a very
> > >> > > > > > > > > > > > > good
> > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> problems
> > >> an
> > >> > > > > operator
> > >> > > > > > > > might
> > >> > > > > > > > > > > > > experience
> > >> > > > > > > > > > > > > > if they have an issue with a given primary
> > >> storage
> > >> > > pool
> > >> > > > > in
> > >> > > > > > a
> > >> > > > > > > > > > cluster.
> > >> > > > > > > > > > > > (I
> > >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> > >> SolidFire
> > >> > > case
> > >> > > > > > Mike
> > >> > > > > > > :)
> > >> > > > > > > > > > > > > > ) And
> > >> > > > > > > > > > > > if
> > >> > > > > > > > > > > > > > the transfer from primary to secondary is
> > slow,
> > >> the
> > >> > > > time
> > >> > > > > to
> > >> > > > > > > > > > > > > > being able
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > create a template or export the volume will
> be
> > >> > slow.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > So for me the issue is around making sure
> that
> > >> the
> > >> > > end
> > >> > > > > > users
> > >> > > > > > > > > > > > expectations
> > >> > > > > > > > > > > > > > are met (while improving the
> speed/efficiency
> > of
> > >> > the
> > >> > > > back
> > >> > > > > > > end)
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> http://www.shapeblue.com>
> > >> Paul
> > >> > > > Angus
> > >> > > > > > VP
> > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > >> > > > > > > > > > > > > >
> > >> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > >> > > > > > > |
> > >> > > > > > > > m:
> > >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > >> > > > > <javascript:;> | t:
> > >> > > > > > @cloudyangus*
> > >> > > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > >> > > > <javascript:;>
> > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> http://www.shapeblue.com
> > >
> > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> WC2N
> > >> 4HS
> > >> > UK
> > >> > > > > Shape
> > >> > > > > > > > Blue
> > >> > > > > > > > > > > > > > Ltd is a company incorporated in England &
> > >> Wales.
> > >> > > > > ShapeBlue
> > >> > > > > > > > > > > > > > Services India LLP is a company incorporated
> > in
> > >> > India
> > >> > > > and
> > >> > > > > > is
> > >> > > > > > > > > > > > > > operated
> > >> > > > > > > > > > > > > under
> > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> Brasil
> > >> > > > > Consultoria
> > >> > > > > > > Ltda
> > >> > > > > > > > > > > > > > is a company incorporated in Brasil and is
> > >> operated
> > >> > > > under
> > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA
> Pty
> > >> Ltd
> > >> > is
> > >> > > a
> > >> > > > > > > company
> > >> > > > > > > > > > > > > > registered by The Republic
> > >> > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > South Africa and is traded under license
> from
> > >> Shape
> > >> > > > Blue
> > >> > > > > > Ltd.
> > >> > > > > > > > > > > > > > ShapeBlue
> > >> > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > a registered trademark.
> > >> > > > > > > > > > > > > > This email and any attachments to it may be
> > >> > > > confidential
> > >> > > > > > and
> > >> > > > > > > > are
> > >> > > > > > > > > > > > intended
> > >> > > > > > > > > > > > > > solely for the use of the individual to whom
> > it
> > >> is
> > >> > > > > > addressed.
> > >> > > > > > > > > > > > > > Any views
> > >> > > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > opinions expressed are solely those of the
> > >> author
> > >> > and
> > >> > > > do
> > >> > > > > > not
> > >> > > > > > > > > > > > necessarily
> > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or related
> > >> > > companies.
> > >> > > > > If
> > >> > > > > > > you
> > >> > > > > > > > > > > > > > are not
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > intended recipient of this email, you must
> > >> neither
> > >> > > take
> > >> > > > > any
> > >> > > > > > > > > > > > > > action
> > >> > > > > > > > > > > > based
> > >> > > > > > > > > > > > > > upon its contents, nor copy or show it to
> > >> anyone.
> > >> > > > Please
> > >> > > > > > > > contact
> > >> > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > sender
> > >> > > > > > > > > > > > > > if you believe you have received this email
> in
> > >> > error.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > -----Original Message-----
> > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > >> pdion@cloudops.com
> > >> > > > > <javascript:;>
> > >> > > > > > <javascript:;>]
> > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > >> > Snapshots
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Hi Mike,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > >> StorageSnapshot
> > >> > > for
> > >> > > > > > > managed
> > >> > > > > > > > > > > > > > storage
> > >> > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > expected,
> > >> > > > behavior
> > >> > > > > > is
> > >> > > > > > > to
> > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> Storage.
> > >> So a
> > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > >> > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > snapshot that remain on the managed storage
> > >> > > appliance.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Quickly looking at the API doc and I don't
> > see a
> > >> > > strong
> > >> > > > > > > > > > > > > > requirement for volume snapshots to be moved
> > >> into
> > >> > > > > secondary
> > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is
> not
> > >> > useful,
> > >> > > > but
> > >> > > > > > > both
> > >> > > > > > > > > > > > > > use cases are required. A snapshot that
> remain
> > >> on
> > >> > the
> > >> > > > > > managed
> > >> > > > > > > > > > > > > > storage, and another type of
> > >> > > > > > > > > > > > snapshot
> > >> > > > > > > > > > > > > > that end up into the secondary storage.
> Since
> > >> > you've
> > >> > > > > done a
> > >> > > > > > > lot
> > >> > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > work,
> > >> > > > > > > > > > > > > > might easier to just add a parameter to the
> > >> current
> > >> > > > > > snapshot
> > >> > > > > > > > API
> > >> > > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > would
> > >> > > > > > > > > > > > > > trigger an extraction of the storage
> snapshot
> > >> into
> > >> > > the
> > >> > > > > > > > secondary
> > >> > > > > > > > > > > > storage?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > PL
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > Tutkowski <
> > >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > I think that all sounds reasonable then -
> > >> thanks!
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > Mushtaq <
> > >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> > >> <javascript:;>>
> > >> > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> > >> > requirements.
> > >> > > > One
> > >> > > > > > of
> > >> > > > > > > > our
> > >> > > > > > > > > > > > earlier
> > >> > > > > > > > > > > > > > >> iterations on this was to have an
> argument
> > to
> > >> > the
> > >> > > > > create
> > >> > > > > > > > > > > > > > >> snapshot
> > >> > > > > > > > > > > > API
> > >> > > > > > > > > > > > > > >> which decides whether to backup the
> volume
> > to
> > >> > sec
> > >> > > > > > storage
> > >> > > > > > > > but
> > >> > > > > > > > > > > > > > >> we realized it would make management of
> > >> > snapshots
> > >> > > > > quite
> > >> > > > > > > > messy
> > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > >> > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> Tutkowski
> > >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> > <javascript:;>
> > >> > > > > <javascript:;>>
> > >> > > > > > > > > > > > > > >> wrote:
> > >> > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > >>> Hi,
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> Just to make sure I understand all the
> > >> > > requirements
> > >> > > > > > here:
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 1) This relates only to managed storage
> > (1:1
> > >> > > > mapping
> > >> > > > > > > > between
> > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> volume).
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> (introduced
> > >> in
> > >> > > 4.6)
> > >> > > > > > > > > > > > > > >>> functionality, which creates a snapshot
> on
> > >> the
> > >> > > SAN,
> > >> > > > > and
> > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> > >> snapshot,
> > >> > but
> > >> > > > to
> > >> > > > > > copy
> > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to
> > NFS.
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken.
> It's
> > >> the
> > >> > > > backup
> > >> > > > > to
> > >> > > > > > > NFS
> > >> > > > > > > > > > > > > > >>> that is optional.
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> > >> snapshot
> > >> > > > that's
> > >> > > > > on
> > >> > > > > > > the
> > >> > > > > > > > > > > > > > >>> SAN (already works).
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created
> from
> > >> the
> > >> > > > > snapshot
> > >> > > > > > > > > > > > > > >>> that's on
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > >>> SAN (already works as long as the new
> > >> > CloudStack
> > >> > > > > volume
> > >> > > > > > > > ends
> > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > snapshot
> > >> API
> > >> > > > then
> > >> > > > > or
> > >> > > > > > > > > > > > > > >>> would that just be the standard volume
> > >> snapshot
> > >> > > > > without
> > >> > > > > > > the
> > >> > > > > > > > > > > > > > >>> backup to
> > >> > > > > > > > > > > NFS?
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> Thanks!
> > >> > > > > > > > > > > > > > >>> Mike
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> > Mushtaq
> > >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
> > >> > > > > <javascript:;>>
> > >> > > > > > > > > > > > > > >>> wrote:
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > functionalities
> > >> > > > > (snapshot
> > >> > > > > > on
> > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we
> > would
> > >> > like
> > >> > > > to
> > >> > > > > > have
> > >> > > > > > > > > both.
> > >> > > > > > > > > > > > > > >>>> For example, some of our customers want
> > to
> > >> > > > implement
> > >> > > > > > > their
> > >> > > > > > > > > > > > > > >>>> own backup strategies and do encryption
> > to
> > >> > their
> > >> > > > > > backups
> > >> > > > > > > > > > > > > > >>>> which is a perfect use case for Storage
> > >> > Snapshot
> > >> > > > > while
> > >> > > > > > > our
> > >> > > > > > > > > > > > > > >>>> other customers will still keep using
> the
> > >> > > standard
> > >> > > > > > > volume
> > >> > > > > > > > > > > snapshot.
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> To keep things backward compatible, we
> > can
> > >> > add a
> > >> > > > > > setting
> > >> > > > > > > > > > > > > > >>>> which
> > >> > > > > > > > > > > > says
> > >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> > >> because,
> > >> > > after
> > >> > > > > > all,
> > >> > > > > > > > you
> > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when
> > doing
> > >> a
> > >> > > > Volume
> > >> > > > > > > > > Snapshot.
> > >> > > > > > > > > > > > > > >>>> You could stop the process there and
> not
> > do
> > >> > the
> > >> > > > > > upload.
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> What do you think about this approach?
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> Thanks,
> > >> > > > > > > > > > > > > > >>>> -Syed
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> > >> > Tutkowski <
> > >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out load
> > >> here,
> > >> > but
> > >> > > > if
> > >> > > > > a
> > >> > > > > > > > given
> > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need
> > to
> > >> > > provide
> > >> > > > > > both
> > >> > > > > > > > the
> > >> > > > > > > > > > > > ability
> > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it
> to
> > >> NFS
> > >> > (if
> > >> > > > > just
> > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we
> > >> might
> > >> > be
> > >> > > > able
> > >> > > > > > to
> > >> > > > > > > > get
> > >> > > > > > > > > > > > > > >>>>> away with no new
> > >> > > > > > > > > > > > API
> > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> custom
> > >> > > snapshot
> > >> > > > > > > strategy
> > >> > > > > > > > > > > > > > >>>>> and
> > >> > > > > > > > > > > > data
> > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case
> where
> > >> the
> > >> > > > > > CloudStack
> > >> > > > > > > > > > > > > > >>>>> cloud
> > >> > > > > > > > > > > > does
> > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > >> exported-to-NFS
> > >> > > > > backup.
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> In other words, the "default" behavior
> > >> would
> > >> > be
> > >> > > > to
> > >> > > > > > use
> > >> > > > > > > > the
> > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> > strategy
> > >> > that
> > >> > > > we
> > >> > > > > > > > already
> > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> > >> > snapshot).
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however,
> wants
> > >> to
> > >> > > take
> > >> > > > a
> > >> > > > > > SAN
> > >> > > > > > > > > > > > > > >>>>> snapshot and have the data exported to
> > >> NFS,
> > >> > > then
> > >> > > > we
> > >> > > > > > > could
> > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config
> file
> > to
> > >> > make
> > >> > > > use
> > >> > > > > > of
> > >> > > > > > > a
> > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data motion
> > >> > strategy
> > >> > > > that
> > >> > > > > > > > > > > > > > >>>>> performs both of these
> > >> > > > > > > > > > > > > > activities.
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is still
> the
> > >> > default
> > >> > > > for
> > >> > > > > > > > users,
> > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change this
> > >> > behavior
> > >> > > > via
> > >> > > > > > > > > > > configuration.
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike
> > >> > > Tutkowski <
> > >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to come
> up
> > >> > with a
> > >> > > > > > viable
> > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way
> for
> > >> them
> > >> > > to
> > >> > > > > move
> > >> > > > > > > > from
> > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and some
> > >> obvious
> > >> > > way
> > >> > > > > > that
> > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > do this).
> > >> > > > > > > > > > > > > > >>>>>>
> > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed
> > >> > Mushtaq
> > >> > > <
> > >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com
> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>>
> > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> upgrade
> > >> path
> > >> > > > > however,
> > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and are
> > on a
> > >> > > > managed
> > >> > > > > > > > storage
> > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > >> functionality
> > >> > > > with
> > >> > > > > > > Volume
> > >> > > > > > > > >



-- 
Ian Rae
CEO | PDG
c: 514.944.4008

CloudOps | Cloud Infrastructure and Networking Solutions
www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6

Re: [Propose][New Feature] Storage Snapshots

[Will Stevens <ws...@cloudops.com>]

I don't think a global setting is a good option because we need both
functionality to be available at the same time and for different use cases
to be able to pick which they choose.

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> Now that I re-read your e-mail, it dawned on me: The end-user doesn't care
> where the snapshot is.
>
> If that's true, then we should perhaps control this via Global Settings or
> something.
>
> On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
> > It's not ideal - true, but it does allow us to be backward compatible.
> >
> > If you have other ideas, though, about how to maintain backward
> > compatibility, I'm definitely open to hear them.
> >
> > Thanks!
> >
> > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <sy...@gmail.com>
> > wrote:
> >
> >> Hi Mike,
> >>
> >> Adding a flag to createSnapshot was the first and the most obvious thing
> >> that came to our minds. The problem that I had with this was that:
> >>
> >> 1) I feel it is exposing something to the end user that is internal to
> the
> >> cloud.
> >>
> >> 2) We have to follow two different ways of restore/deletion in the same
> >> code path depending on where the Snapshot resides which I find kind of a
> >> bad design.
> >>
> >> But if exposing a archive flag to the end user is acceptable then we can
> >> definitely use this instead of adding the StorageSnapshot API
> >>
> >> Thanks,
> >> -Syed
> >>
> >>
> >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> >> mike.tutkowski@solidfire.com
> >> > wrote:
> >>
> >> > Hi Pierre-Luc,
> >> >
> >> > My recommendation would be this:
> >> >
> >> > Add an "archive" flag to the current volume-snapshot API. Its default
> >> would
> >> > be "false" because that would be backward compatible with how 4.6 has
> >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6, 4.7,
> and
> >> > 4.8).
> >> >
> >> > If you set archive=true, then we would perform a background migration
> of
> >> > the snapshot from the SAN to the secondary storage (then delete the
> SAN
> >> > snapshot).
> >> >
> >> > That archive parameter would only be valid for managed storage.
> >> >
> >> > Sound reasonable?
> >> >
> >> > Also, a VM snapshot that includes disks provided by managed storage
> >> should
> >> > work.
> >> >
> >> > Talk to you later,
> >> > Mike
> >> >
> >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <pd...@cloudops.com>
> >> > wrote:
> >> >
> >> > > Mike,
> >> > >
> >> > > In terms of API's, would you prefer introducing a parameter to the
> >> > existing
> >> > > VolumeSnapshot, example:   extract={true|false}  with a default
> value
> >> of
> >> > > true  which would extract snapshot into the secondary storage, which
> >> is
> >> > the
> >> > > current default behavior. Then for SAN snapshot that remain on the
> >> SAN we
> >> > > would just set "extract=false" ?  as oppose to create a new
> >> > >  StorageSnapshot API ?
> >> > >
> >> > >
> >> > > Paul,
> >> > >
> >> > > From what I'm seeing so far, we can't do a VM-snapshot when using
> >> managed
> >> > > storage for VM having more than one Volume. For the reason that
> >> snapshot
> >> > > are performed outside of the hypervisor awareness and
> asynchronously.
> >> If
> >> > > someone have a way to address that, it would make thinks much more
> >> > > attractive.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com> wrote:
> >> > >
> >> > > > I think a service provider backup scenario is more likely to take
> >> > > advantage
> >> > > > of SAN snapshot. There are a few reasons for this. Traditional
> >> backups
> >> > > > involve access to the file system, and there is an expectation
> that
> >> > this
> >> > > > can be done with reasonably short time frames without negatively
> >> > > impacting
> >> > > > VM performance, and that the backup orchestrator can apply various
> >> > logic
> >> > > > and or transformations to the data (compress, encrypt, deltas
> >> etc...).
> >> > > > While it is true that one could apply a backup process to a cloud
> >> > > snapshot,
> >> > > > this would be slow and inefficient requiring the data to be moved
> >> > several
> >> > > > times and there are some major bottlenecks with cloud snapshots.
> >> With a
> >> > > > cloud snapshot - there seems to be no reasonable expectation of
> >> being
> >> > > able
> >> > > > to do differential snapshots (I think this depends on the
> >> hypervisor)
> >> > and
> >> > > > if you do differential snapshots this will make file backups from
> >> those
> >> > > > snapshots even more complicated to orchestrate.
> >> > > >
> >> > > > Suspect there needs to be a different thread on how to better
> enable
> >> > > > backups, as a service. As per Paul's suggestion, but it is a
> related
> >> > > > workflow so relevant to this discussion.
> >> > > >
> >> > > > Ian
> >> > > >
> >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> >> > > mike.tutkowski@solidfire.com>
> >> > > > wrote:
> >> > > >
> >> > > > > To me it sounds like number two and number three are different
> >> uses
> >> > for
> >> > > > the
> >> > > > > same "thing"(which is totally fine).
> >> > > > >
> >> > > > > As for taking a fast SAN snapshot and exporting it
> >> asynchronously, do
> >> > > we
> >> > > > > see the SSVM as performing the export?
> >> > > > >
> >> > > > > To be backwards compatible with what we have in 4.6 and later
> for
> >> > > volume
> >> > > > > snapshots for managed storage, I think it might be easier if we
> >> pass
> >> > > in a
> >> > > > > flag that says whether or not to archive the SAN snapshot
> (which,
> >> I
> >> > > > think,
> >> > > > > is something that you suggested, as well, Pierre-Luc).
> >> > > > >
> >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> pdion@cloudops.com
> >> > > > > <javascript:;>> wrote:
> >> > > > >
> >> > > > > > Hi Mike,
> >> > > > > >
> >> > > > > > The reason behind the creation of a SAN snapshot which is
> >> exported
> >> > > into
> >> > > > > > secondary storage, is because creating a copy of the .VHD
> >> directly
> >> > > > would
> >> > > > > > impact uptime of the VM as creating that copy take lots of
> time.
> >> > Has
> >> > > > > oppose
> >> > > > > > to a SAN snapshot that is close to instantaneous which can
> >> > afterward
> >> > > be
> >> > > > > > clone into Secondary Storage asynchronously.
> >> > > > > >
> >> > > > > > I would suspect an extracted VolumeSnapshot taken from a SAN
> >> > snapshot
> >> > > > > could
> >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> >> consumption
> >> > > on
> >> > > > > the
> >> > > > > > Primary Storage side.
> >> > > > > >
> >> > > > > >
> >> > > > > > I see 3 definitions in our current discussion regarding the
> term
> >> > > > snapshot
> >> > > > > > (these are not official terminology but by own interpretation
> of
> >> > > them):
> >> > > > > >
> >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a
> >> > > > snapshot):
> >> > > > > > it's a volume snapshot at the storage level, point in time of
> >> your
> >> > > > data.
> >> > > > > it
> >> > > > > > reside on the primary storage. Useful and efficient for
> software
> >> > side
> >> > > > > > incident.
> >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud
> >> backup
> >> > > > aws-S3
> >> > > > > > style ): Point in time copy of the Virtual Disk that reside
> on a
> >> > > > > different
> >> > > > > > storage array then the original Volume. Facilitate data
> >> migration
> >> > > > between
> >> > > > > > clusters and, in case of primary storage incident, Volume
> >> snapshots
> >> > > are
> >> > > > > not
> >> > > > > > impacted and can be reuse.
> >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that also
> >> > > validate
> >> > > > > data
> >> > > > > > integrity, provide a storage efficient archiving method and an
> >> > > > > independent
> >> > > > > > way to restore your data in case of an major infrastructure
> >> > disaster.
> >> > > > > >
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > >
> >> > > > > > PL
> >> > > > > >
> >> > > > > >
> >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> >> > > > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> >> > > > > > > wrote:
> >> > > > > >
> >> > > > > > > So, let's see if I currently follow the requirements:
> >> > > > > > >
> >> > > > > > > * Augment volume snapshots for managed storage to
> >> conditionally
> >> > > > export
> >> > > > > > data
> >> > > > > > > to NFS. The current process of taking a snapshot on the SAN
> is
> >> > > fine,
> >> > > > > but
> >> > > > > > > we'd like the option to export the data to NFS, as well.
> >> > > > > > >
> >> > > > > > > Questions:
> >> > > > > > >
> >> > > > > > > Once the data has been exported to NFS, do we keep the SAN
> >> > snapshot
> >> > > > or
> >> > > > > > > delete it?
> >> > > > > > >
> >> > > > > > > If we are deleting the SAN snapshot, then why don't we just
> >> copy
> >> > > the
> >> > > > > VHD
> >> > > > > > > from primary to secondary the way we do today for
> non-managed
> >> > (i.e.
> >> > > > > > > traditional) storage? Why create a SAN snapshot in that
> >> scenario?
> >> > > > > Perhaps
> >> > > > > > > to have the SSVM mount and perform the VHD copy to secondary
> >> > > storage
> >> > > > > > > instead of a XenServer host?
> >> > > > > > >
> >> > > > > > > Thanks for the clarification.
> >> > > > > > >
> >> > > > > > > By the way, to me a backup is when you copy data from one
> >> storage
> >> > > > > system
> >> > > > > > to
> >> > > > > > > another (regardless of features, if any, to restore the data
> >> in
> >> > the
> >> > > > > > > future). A snapshot is a point-in-time view of the data of a
> >> > volume
> >> > > > and
> >> > > > > > > it's stored on the same storage system as the volume.
> >> > > > > > >
> >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> >> > > pdion@cloudops.com
> >> > > > > <javascript:;>
> >> > > > > > <javascript:;>>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > That's fun to see that discussion happening. I 100% agree
> >> with
> >> > > > Paul's
> >> > > > > > > > points of view. VolumeSnapshot are not a backup, but I do
> >> > > consider
> >> > > > > them
> >> > > > > > > as
> >> > > > > > > > a safety vest against Primary Storage failure, because
> >> failure
> >> > > > append
> >> > > > > > > :-( .
> >> > > > > > > >
> >> > > > > > > > The current proposal around snapshots that reside on the
> >> > primary
> >> > > > > > storage
> >> > > > > > > or
> >> > > > > > > > snapshots that end in the Secondary Storage  is not to
> >> address
> >> > > any
> >> > > > > kind
> >> > > > > > > of
> >> > > > > > > > backups requirement because a snapshot is not a backup,
> >> event
> >> > an
> >> > > > > > > extracted
> >> > > > > > > > VM snapshot.
> >> > > > > > > >
> >> > > > > > > > The main idea, and again this is for managed storage;
> >> > > > > > > >
> >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> >> capability
> >> > > for
> >> > > > > > fast
> >> > > > > > > > response time that support rollback to previous timestamp,
> >> > create
> >> > > > new
> >> > > > > > > > volume and maybe create template.
> >> > > > > > > >     not required to be a new API if the work is already
> >> done, I
> >> > > > think
> >> > > > > > > this
> >> > > > > > > > is a different behaviors than the user expectation of a
> >> > > > > > volume-snapshot.
> >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior
> >> that
> >> > > > create
> >> > > > > > an
> >> > > > > > > > extraction of a volume into SecondaryStorage which can be
> >> reuse
> >> > > to
> >> > > > > > > create a
> >> > > > > > > > new volume into another Primary Storage. This type of
> >> snapshot
> >> > > is a
> >> > > > > > slow
> >> > > > > > > > job since yes it would have to copy the full volume size
> on
> >> the
> >> > > > > > Secondary
> >> > > > > > > > storage.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > PL
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> >> > > > > syed1.mushtaq@gmail.com <javascript:;>
> >> > > > > > <javascript:;>>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > I think I share you view on the 'Ideal world'. Backup
> (via
> >> > > Volume
> >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is
> >> > > amplified
> >> > > > > > > > especially
> >> > > > > > > > > when you have a object storage as your secondary storage
> >> > > because
> >> > > > it
> >> > > > > > > > > requires two copies (one to an NFS staging area and from
> >> > there
> >> > > to
> >> > > > > > > object
> >> > > > > > > > > storage). And not to mention that all these copies are
> >> > > consuming
> >> > > > > > > > hypervisor
> >> > > > > > > > > resources. Xenserver's Dom0 is also a huge bottleneck as
> >> all
> >> > > the
> >> > > > > > > Network
> >> > > > > > > > > and I/O flow through it. So our intention of proposing
> the
> >> > > > "Storage
> >> > > > > > > > > Snapshots" is to give a better way of achiving snapshots
> >> > while
> >> > > > > still
> >> > > > > > > > > keeping the original definition of volume snpashots (ie
> >> > upload
> >> > > to
> >> > > > > sec
> >> > > > > > > > > storage).
> >> > > > > > > > >
> >> > > > > > > > > But as Erik pointed out volume snapshots are not
> backups.
> >> > They
> >> > > > > don't
> >> > > > > > > work
> >> > > > > > > > > form multi-disk LVM volume groups and dynamic disks. I
> am
> >> all
> >> > > in
> >> > > > > for
> >> > > > > > a
> >> > > > > > > > > better backup solution which handles these use cases and
> >> > > utilizes
> >> > > > > the
> >> > > > > > > > > storage's advanced features.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> >> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > In the beginning... there were CloudStack snapshots
> and
> >> > they
> >> > > > were
> >> > > > > > > > > actually
> >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> snapshots.
> >> > > > > > > > > > Then VM snapshots were created (which are
> point-in-time
> >> > > > > hypervisor
> >> > > > > > > > > > snapshots) and we started referring to the original
> >> > snapshots
> >> > > > as
> >> > > > > > > volume
> >> > > > > > > > > > snapshots.
> >> > > > > > > > > >
> >> > > > > > > > > > CloudStack does not offer 'backups', but many people
> use
> >> > > volume
> >> > > > > > > > snapshots
> >> > > > > > > > > > as backups. However you can't in-place restore volume
> >> > > snapshots
> >> > > > > and
> >> > > > > > > if
> >> > > > > > > > > you
> >> > > > > > > > > > have a VM with multiple volumes, the volume snapshots
> >> must
> >> > be
> >> > > > > done
> >> > > > > > in
> >> > > > > > > > > > series, meaning that the state across all of the
> >> volumes is
> >> > > > > > unlikely
> >> > > > > > > to
> >> > > > > > > > > be
> >> > > > > > > > > > consistent.
> >> > > > > > > > > >
> >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> options
> >> > > which
> >> > > > > > users
> >> > > > > > > > > might
> >> > > > > > > > > > expect as well options as to where they might be
> >> stored. In
> >> > > my
> >> > > > > > ideal
> >> > > > > > > > > world
> >> > > > > > > > > > they would also be able to leverage back-end hardware
> >> (such
> >> > > as
> >> > > > > > > > Solidfire,
> >> > > > > > > > > > NetApp etc :) ) and software such as Veeam, Commvault
> >> etc
> >> > to
> >> > > > > > > accelerate
> >> > > > > > > > > the
> >> > > > > > > > > > process.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> >> > > > > > > > > > Paul Angus
> >> > > > > > > > > > VP Technology ,  ShapeBlue
> >> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> >> > > > > > > > > >
> >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> >> > > > > m:
> >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> >> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> >> > <javascript:;>
> >> > > |
> >> > > > > t: @cloudyangus*
> >> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> >> > > > > <ja...@cloudyangus>
> >> > > > > > |  w:
> >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> >> > > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> >> > > > > > > > > > Shape Blue Ltd is a company incorporated in England &
> >> > Wales.
> >> > > > > > > ShapeBlue
> >> > > > > > > > > > Services India LLP is a company incorporated in India
> >> and
> >> > is
> >> > > > > > operated
> >> > > > > > > > > under
> >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> >> Consultoria
> >> > > Ltda
> >> > > > > is
> >> > > > > > a
> >> > > > > > > > > > company incorporated in Brasil and is operated under
> >> > license
> >> > > > from
> >> > > > > > > Shape
> >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> >> by
> >> > The
> >> > > > > > > Republic
> >> > > > > > > > of
> >> > > > > > > > > > South Africa and is traded under license from Shape
> Blue
> >> > Ltd.
> >> > > > > > > ShapeBlue
> >> > > > > > > > > is
> >> > > > > > > > > > a registered trademark.
> >> > > > > > > > > > This email and any attachments to it may be
> confidential
> >> > and
> >> > > > are
> >> > > > > > > > intended
> >> > > > > > > > > > solely for the use of the individual to whom it is
> >> > addressed.
> >> > > > Any
> >> > > > > > > views
> >> > > > > > > > > or
> >> > > > > > > > > > opinions expressed are solely those of the author and
> do
> >> > not
> >> > > > > > > > necessarily
> >> > > > > > > > > > represent those of Shape Blue Ltd or related
> companies.
> >> If
> >> > > you
> >> > > > > are
> >> > > > > > > not
> >> > > > > > > > > the
> >> > > > > > > > > > intended recipient of this email, you must neither
> take
> >> any
> >> > > > > action
> >> > > > > > > > based
> >> > > > > > > > > > upon its contents, nor copy or show it to anyone.
> Please
> >> > > > contact
> >> > > > > > the
> >> > > > > > > > > sender
> >> > > > > > > > > > if you believe you have received this email in error.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > -----Original Message-----
> >> > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> >> > > > > <javascript:;> <javascript:;>]
> >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> >> > <javascript:;>
> >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> >> > > > > > > > > >
> >> > > > > > > > > > Paul,
> >> > > > > > > > > >
> >> > > > > > > > > > When you say actual backups, how would it be different
> >> from
> >> > > the
> >> > > > > > > Volume
> >> > > > > > > > > > Snapshots that exist currently. My understanding is
> that
> >> > > > Backups
> >> > > > > > end
> >> > > > > > > up
> >> > > > > > > > > in
> >> > > > > > > > > > Sec Storage whereas Snapshots are just a point-in-time
> >> > state
> >> > > of
> >> > > > > > your
> >> > > > > > > > > volume
> >> > > > > > > > > > which can be restored back correct?
> >> > > > > > > > > >
> >> > > > > > > > > > -Syed
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> >> > > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Syed,
> >> > > > > > > > > > >
> >> > > > > > > > > > > As I understand it, the SolidFire plugin will export
> >> the
> >> > > > > snapshot
> >> > > > > > > to
> >> > > > > > > > > > > secondary storage if the user requests a template
> from
> >> > the
> >> > > > > > snapshot
> >> > > > > > > > or
> >> > > > > > > > > > > wants to download the snapshot from the cloud. This
> >> is a
> >> > > > good,
> >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> storage
> >> is
> >> > > > super
> >> > > > > > > > > > > reliable and snapshots on SolidFire arrays take up
> >> next
> >> > to
> >> > > no
> >> > > > > > > space.
> >> > > > > > > > > > > BUT I think that we are talking about a more general
> >> > > purpose
> >> > > > > API,
> >> > > > > > > and
> >> > > > > > > > > > > other storage systems may not be as awesome as
> Mike's.
> >> > > That's
> >> > > > > my
> >> > > > > > > > > > > concern. Also, the time to transfer for say 1TB to
> >> move
> >> > > from
> >> > > > > > > primary
> >> > > > > > > > > > > to sec storage and then create a VM template out of
> it
> >> > may
> >> > > be
> >> > > > > too
> >> > > > > > > > long
> >> > > > > > > > > > for users.
> >> > > > > > > > > > >
> >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> snapshot
> >> for
> >> > > > > backup,
> >> > > > > > > > it's
> >> > > > > > > > > > > just that users want to do backups and a volume
> >> snapshot
> >> > is
> >> > > > the
> >> > > > > > > only
> >> > > > > > > > > > > type of snapshot that copies the disk elsewhere and
> >> can
> >> > be
> >> > > > > > > scheduled.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I'm 'pondering' the implications of enabling actual
> >> > backups
> >> > > > > > > (through
> >> > > > > > > > > > > recognised backup providers) and the user
> requirements
> >> > > around
> >> > > > > > them
> >> > > > > > > > > > > (particularly restoration use cases) as a separate
> >> thread
> >> > > of
> >> > > > > > work.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul
> >> Angus
> >> > > VP
> >> > > > > > > > Technology
> >> > > > > > > > > > > , ShapeBlue
> >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> >> > > > > > > > > > >
> >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> >> > > > |
> >> > > > > m:
> >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> >> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> >> > > <javascript:;> |
> >> > > > > t: @cloudyangus*
> >> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> >> <javascript:;>
> >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS
> UK
> >> > Shape
> >> > > > > Blue
> >> > > > > > > Ltd
> >> > > > > > > > > > > is a company incorporated in England & Wales.
> >> ShapeBlue
> >> > > > > Services
> >> > > > > > > > India
> >> > > > > > > > > > > LLP is a company incorporated in India and is
> operated
> >> > > under
> >> > > > > > > license
> >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria
> >> Ltda
> >> > is
> >> > > a
> >> > > > > > > company
> >> > > > > > > > > > > incorporated in Brasil and is operated under license
> >> from
> >> > > > Shape
> >> > > > > > > Blue
> >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by
> >> The
> >> > > > > Republic
> >> > > > > > > of
> >> > > > > > > > > > > South Africa and is traded under license from Shape
> >> Blue
> >> > > Ltd.
> >> > > > > > > > > > > ShapeBlue is a registered trademark.
> >> > > > > > > > > > > This email and any attachments to it may be
> >> confidential
> >> > > and
> >> > > > > are
> >> > > > > > > > > > > intended solely for the use of the individual to
> whom
> >> it
> >> > is
> >> > > > > > > > addressed.
> >> > > > > > > > > > > Any views or opinions expressed are solely those of
> >> the
> >> > > > author
> >> > > > > > and
> >> > > > > > > do
> >> > > > > > > > > > > not necessarily represent those of Shape Blue Ltd or
> >> > > related
> >> > > > > > > > > > > companies. If you are not the intended recipient of
> >> this
> >> > > > email,
> >> > > > > > you
> >> > > > > > > > > > > must neither take any action based upon its
> contents,
> >> nor
> >> > > > copy
> >> > > > > or
> >> > > > > > > > show
> >> > > > > > > > > > > it to anyone. Please contact the sender if you
> believe
> >> > you
> >> > > > have
> >> > > > > > > > > received
> >> > > > > > > > > > this email in error.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > -----Original Message-----
> >> > > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> >> > > > > <javascript:;>
> >> > > > > > <javascript:;>]
> >> > > > > > > > > > > Sent: 05 February 2016 15:31
> >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> >> > > <javascript:;>
> >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> Snapshots
> >> > > > > > > > > > >
> >> > > > > > > > > > > I think the terminology confusion comes from AWS
> where
> >> > they
> >> > > > do
> >> > > > > > EBS
> >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
> >> followed
> >> > > > that.
> >> > > > > > And
> >> > > > > > > > as
> >> > > > > > > > > > > an end user who is oblivious to the internals of my
> >> > > provider,
> >> > > > > my
> >> > > > > > > > > > > expectation would be something similar to what AWS
> as
> >> > that
> >> > > is
> >> > > > > my
> >> > > > > > > > > > > biggest reference point.
> >> > > > > > > > > > >
> >> > > > > > > > > > > To your point Mike, I agree that a Primary Storage
> >> > failure
> >> > > on
> >> > > > > > > > > > > SolidFire is unlikely, there are other motivations
> >> for us
> >> > > to
> >> > > > > push
> >> > > > > > > > data
> >> > > > > > > > > > > to secondary storage. Primary storage (atleast for
> us)
> >> > > costs
> >> > > > > > > around 3
> >> > > > > > > > > > > times as much as secondary storage and snapshots on
> >> > primary
> >> > > > > > storage
> >> > > > > > > > > > > are rarely used (especially for some of our
> customers
> >> who
> >> > > do
> >> > > > > > daily
> >> > > > > > > > > > backups).
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> >> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> >> > > <javascript:;>>
> >> > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Some of the weirdness is around terminology.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > For most systems I've worked on, a snapshot and a
> >> > backup
> >> > > > are
> >> > > > > > two
> >> > > > > > > > > > > > completely different things (but CloudStack has
> >> > > > traditionally
> >> > > > > > > used
> >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
> >> though,
> >> > > that
> >> > > > > if
> >> > > > > > > your
> >> > > > > > > > > > > > primary storage is running on SolidFire that it is
> >> > > unlikely
> >> > > > > > > you'll
> >> > > > > > > > > > > > encounter an issue where your primary storage goes
> >> > > offline
> >> > > > > (and
> >> > > > > > > > > > > > you'll even maintain your performance guarantees
> >> during
> >> > > > > failure
> >> > > > > > > > > > > > scenarios and upgrades, as well). That being the
> >> case,
> >> > it
> >> > > > is
> >> > > > > > less
> >> > > > > > > > > > > > useful to require a backup to Swift (but it's
> >> perfectly
> >> > > OK
> >> > > > if
> >> > > > > > > > that's
> >> > > > > > > > > > > > what we want to do
> >> > > > > > > > > > > here).
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> >> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> >> > <javascript:;>>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Paul,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > I believe with the current implementation of
> >> > Snapshots
> >> > > on
> >> > > > > > > managed
> >> > > > > > > > > > > > > storage
> >> > > > > > > > > > > > > (SolidFire) the snapshots are never exported to
> >> the
> >> > > > > secondary
> >> > > > > > > > > > storage.
> >> > > > > > > > > > > > > While this solves the problem of having
> snapshots
> >> > > taking
> >> > > > > > > forever
> >> > > > > > > > > > > > > to get to sec storage, this leaves us with a
> >> > > > > > > > > > > > huge
> >> > > > > > > > > > > > > liability if our primary storage goes down. We
> see
> >> > > > > snapshots
> >> > > > > > as
> >> > > > > > > > > > > > > our recovery path because we store them in Swift
> >> > which
> >> > > is
> >> > > > > > > > reliable
> >> > > > > > > > > > > > > and resilient to failures.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> Volume
> >> > > > snapshots
> >> > > > > > > > always
> >> > > > > > > > > > > > > backed up to secondary storage and Storage
> >> Snapshots
> >> > > stay
> >> > > > > on
> >> > > > > > > the
> >> > > > > > > > > > > > > primary
> >> > > > > > > > > > > > storage.
> >> > > > > > > > > > > > > A provider could potentially mix both these and
> >> solve
> >> > > the
> >> > > > > > > problem
> >> > > > > > > > > > > > > that you mentioned where you want to meet user's
> >> > > > > expectation
> >> > > > > > > of a
> >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while having
> >> an
> >> > > > ability
> >> > > > > > to
> >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the device)
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Hope this clarifies things.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > -Syed
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> >> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> >> > > <javascript:;>>
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > HI guys,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Could someone point me to the Jira bug of FS
> for
> >> > the
> >> > > > > > > > > > > > > > SAN-snapshot
> >> > > > > > > > > > > > feature
> >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > From my discussions with users and operators
> >> around
> >> > > > > > snapshots
> >> > > > > > > > > > > > > > I'd make
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > following observations:
> >> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
> >> long-term
> >> > > and
> >> > > > > > short
> >> > > > > > > > > > > > > > term)
> >> > > > > > > > > > > > with
> >> > > > > > > > > > > > > > the expectation that they can use them for
> >> recovery
> >> > > if
> >> > > > > > > > required.
> >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> something
> >> > has
> >> > > > gone
> >> > > > > > > wrong
> >> > > > > > > > > > > > > > with primary storage.
> >> > > > > > > > > > > > > > c. users sometimes want to be able to export
> >> > > snapshots
> >> > > > as
> >> > > > > > > well
> >> > > > > > > > > > > > > > as
> >> > > > > > > > > > > > create
> >> > > > > > > > > > > > > > new VMs from their snapshots
> >> > > > > > > > > > > > > > d. snapshots are a currently a massive pain
> for
> >> > > > > operators,
> >> > > > > > I
> >> > > > > > > > > > > > > > know at
> >> > > > > > > > > > > > > least
> >> > > > > > > > > > > > > > one public cloud who have snapshots which
> take 2
> >> > days
> >> > > > to
> >> > > > > > > > > complete.
> >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used for
> >> > multiple
> >> > > > LVM
> >> > > > > > > > disks.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I think the process Mike has used in the
> >> SolidFire
> >> > > > plugin
> >> > > > > > > (only
> >> > > > > > > > > > > > > > moving
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > disk image to secondary storage when you
> >> absolutely
> >> > > > have
> >> > > > > > to)
> >> > > > > > > is
> >> > > > > > > > > > > > > > a very
> >> > > > > > > > > > > > > good
> >> > > > > > > > > > > > > > and pragmatic solution. I wonder what problems
> >> an
> >> > > > > operator
> >> > > > > > > > might
> >> > > > > > > > > > > > > experience
> >> > > > > > > > > > > > > > if they have an issue with a given primary
> >> storage
> >> > > pool
> >> > > > > in
> >> > > > > > a
> >> > > > > > > > > > cluster.
> >> > > > > > > > > > > > (I
> >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> >> SolidFire
> >> > > case
> >> > > > > > Mike
> >> > > > > > > :)
> >> > > > > > > > > > > > > > ) And
> >> > > > > > > > > > > > if
> >> > > > > > > > > > > > > > the transfer from primary to secondary is
> slow,
> >> the
> >> > > > time
> >> > > > > to
> >> > > > > > > > > > > > > > being able
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > create a template or export the volume will be
> >> > slow.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > So for me the issue is around making sure that
> >> the
> >> > > end
> >> > > > > > users
> >> > > > > > > > > > > > expectations
> >> > > > > > > > > > > > > > are met (while improving the speed/efficiency
> of
> >> > the
> >> > > > back
> >> > > > > > > end)
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> >> Paul
> >> > > > Angus
> >> > > > > > VP
> >> > > > > > > > > > > > > > Technology , ShapeBlue
> >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> >> > > > > > > > > > > > > >
> >> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> >> > > > > > > |
> >> > > > > > > > m:
> >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> >> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> >> > > > > <javascript:;> | t:
> >> > > > > > @cloudyangus*
> >> > > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> >> > > > <javascript:;>
> >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> >> > > > > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com
> >
> >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> >> 4HS
> >> > UK
> >> > > > > Shape
> >> > > > > > > > Blue
> >> > > > > > > > > > > > > > Ltd is a company incorporated in England &
> >> Wales.
> >> > > > > ShapeBlue
> >> > > > > > > > > > > > > > Services India LLP is a company incorporated
> in
> >> > India
> >> > > > and
> >> > > > > > is
> >> > > > > > > > > > > > > > operated
> >> > > > > > > > > > > > > under
> >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> >> > > > > Consultoria
> >> > > > > > > Ltda
> >> > > > > > > > > > > > > > is a company incorporated in Brasil and is
> >> operated
> >> > > > under
> >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty
> >> Ltd
> >> > is
> >> > > a
> >> > > > > > > company
> >> > > > > > > > > > > > > > registered by The Republic
> >> > > > > > > > > > > > of
> >> > > > > > > > > > > > > > South Africa and is traded under license from
> >> Shape
> >> > > > Blue
> >> > > > > > Ltd.
> >> > > > > > > > > > > > > > ShapeBlue
> >> > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > a registered trademark.
> >> > > > > > > > > > > > > > This email and any attachments to it may be
> >> > > > confidential
> >> > > > > > and
> >> > > > > > > > are
> >> > > > > > > > > > > > intended
> >> > > > > > > > > > > > > > solely for the use of the individual to whom
> it
> >> is
> >> > > > > > addressed.
> >> > > > > > > > > > > > > > Any views
> >> > > > > > > > > > > > > or
> >> > > > > > > > > > > > > > opinions expressed are solely those of the
> >> author
> >> > and
> >> > > > do
> >> > > > > > not
> >> > > > > > > > > > > > necessarily
> >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or related
> >> > > companies.
> >> > > > > If
> >> > > > > > > you
> >> > > > > > > > > > > > > > are not
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > intended recipient of this email, you must
> >> neither
> >> > > take
> >> > > > > any
> >> > > > > > > > > > > > > > action
> >> > > > > > > > > > > > based
> >> > > > > > > > > > > > > > upon its contents, nor copy or show it to
> >> anyone.
> >> > > > Please
> >> > > > > > > > contact
> >> > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > sender
> >> > > > > > > > > > > > > > if you believe you have received this email in
> >> > error.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > -----Original Message-----
> >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> >> pdion@cloudops.com
> >> > > > > <javascript:;>
> >> > > > > > <javascript:;>]
> >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> >> > Snapshots
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Hi Mike,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > The idea of introducing a new API:
> >> StorageSnapshot
> >> > > for
> >> > > > > > > managed
> >> > > > > > > > > > > > > > storage
> >> > > > > > > > > > > > is
> >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> expected,
> >> > > > behavior
> >> > > > > > is
> >> > > > > > > to
> >> > > > > > > > > > > > > > archive snapshots into the Secondary Storage.
> >> So a
> >> > > > > > > > > > > > > > StorageSnapshot API would be
> >> > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > snapshot that remain on the managed storage
> >> > > appliance.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Quickly looking at the API doc and I don't
> see a
> >> > > strong
> >> > > > > > > > > > > > > > requirement for volume snapshots to be moved
> >> into
> >> > > > > secondary
> >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is not
> >> > useful,
> >> > > > but
> >> > > > > > > both
> >> > > > > > > > > > > > > > use cases are required. A snapshot that remain
> >> on
> >> > the
> >> > > > > > managed
> >> > > > > > > > > > > > > > storage, and another type of
> >> > > > > > > > > > > > snapshot
> >> > > > > > > > > > > > > > that end up into the secondary storage. Since
> >> > you've
> >> > > > > done a
> >> > > > > > > lot
> >> > > > > > > > > > > > > > of
> >> > > > > > > > > > > > work,
> >> > > > > > > > > > > > > > might easier to just add a parameter to the
> >> current
> >> > > > > > snapshot
> >> > > > > > > > API
> >> > > > > > > > > > > > > > that
> >> > > > > > > > > > > > > would
> >> > > > > > > > > > > > > > trigger an extraction of the storage snapshot
> >> into
> >> > > the
> >> > > > > > > > secondary
> >> > > > > > > > > > > > storage?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > PL
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> Tutkowski <
> >> > > > > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > I think that all sounds reasonable then -
> >> thanks!
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> Mushtaq <
> >> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> >> <javascript:;>>
> >> > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> >> > requirements.
> >> > > > One
> >> > > > > > of
> >> > > > > > > > our
> >> > > > > > > > > > > > earlier
> >> > > > > > > > > > > > > > >> iterations on this was to have an argument
> to
> >> > the
> >> > > > > create
> >> > > > > > > > > > > > > > >> snapshot
> >> > > > > > > > > > > > API
> >> > > > > > > > > > > > > > >> which decides whether to backup the volume
> to
> >> > sec
> >> > > > > > storage
> >> > > > > > > > but
> >> > > > > > > > > > > > > > >> we realized it would make management of
> >> > snapshots
> >> > > > > quite
> >> > > > > > > > messy
> >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> >> > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> >> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com
> <javascript:;>
> >> > > > > <javascript:;>>
> >> > > > > > > > > > > > > > >> wrote:
> >> > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >>> Hi,
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> Just to make sure I understand all the
> >> > > requirements
> >> > > > > > here:
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 1) This relates only to managed storage
> (1:1
> >> > > > mapping
> >> > > > > > > > between
> >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN volume).
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 2) We want to take the current (introduced
> >> in
> >> > > 4.6)
> >> > > > > > > > > > > > > > >>> functionality, which creates a snapshot on
> >> the
> >> > > SAN,
> >> > > > > and
> >> > > > > > > > > > > > > > >>> extend it via a config option (or
> >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> >> snapshot,
> >> > but
> >> > > > to
> >> > > > > > copy
> >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to
> NFS.
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken. It's
> >> the
> >> > > > backup
> >> > > > > to
> >> > > > > > > NFS
> >> > > > > > > > > > > > > > >>> that is optional.
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> >> snapshot
> >> > > > that's
> >> > > > > on
> >> > > > > > > the
> >> > > > > > > > > > > > > > >>> SAN (already works).
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created from
> >> the
> >> > > > > snapshot
> >> > > > > > > > > > > > > > >>> that's on
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > >>> SAN (already works as long as the new
> >> > CloudStack
> >> > > > > volume
> >> > > > > > > > ends
> >> > > > > > > > > > > > > > >>> up on the same primary storage).
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> snapshot
> >> API
> >> > > > then
> >> > > > > or
> >> > > > > > > > > > > > > > >>> would that just be the standard volume
> >> snapshot
> >> > > > > without
> >> > > > > > > the
> >> > > > > > > > > > > > > > >>> backup to
> >> > > > > > > > > > > NFS?
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> Thanks!
> >> > > > > > > > > > > > > > >>> Mike
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> Mushtaq
> >> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
> >> > > > > <javascript:;>>
> >> > > > > > > > > > > > > > >>> wrote:
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>>> Is it possible to have both
> functionalities
> >> > > > > (snapshot
> >> > > > > > on
> >> > > > > > > > > > > > > > >>>> SAN & Sec
> >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we
> would
> >> > like
> >> > > > to
> >> > > > > > have
> >> > > > > > > > > both.
> >> > > > > > > > > > > > > > >>>> For example, some of our customers want
> to
> >> > > > implement
> >> > > > > > > their
> >> > > > > > > > > > > > > > >>>> own backup strategies and do encryption
> to
> >> > their
> >> > > > > > backups
> >> > > > > > > > > > > > > > >>>> which is a perfect use case for Storage
> >> > Snapshot
> >> > > > > while
> >> > > > > > > our
> >> > > > > > > > > > > > > > >>>> other customers will still keep using the
> >> > > standard
> >> > > > > > > volume
> >> > > > > > > > > > > snapshot.
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>> To keep things backward compatible, we
> can
> >> > add a
> >> > > > > > setting
> >> > > > > > > > > > > > > > >>>> which
> >> > > > > > > > > > > > says
> >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> >> because,
> >> > > after
> >> > > > > > all,
> >> > > > > > > > you
> >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when
> doing
> >> a
> >> > > > Volume
> >> > > > > > > > > Snapshot.
> >> > > > > > > > > > > > > > >>>> You could stop the process there and not
> do
> >> > the
> >> > > > > > upload.
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>> What do you think about this approach?
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>> Thanks,
> >> > > > > > > > > > > > > > >>>> -Syed
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> >> > Tutkowski <
> >> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
> >> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out load
> >> here,
> >> > but
> >> > > > if
> >> > > > > a
> >> > > > > > > > given
> >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need
> to
> >> > > provide
> >> > > > > > both
> >> > > > > > > > the
> >> > > > > > > > > > > > ability
> >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it to
> >> NFS
> >> > (if
> >> > > > > just
> >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we
> >> might
> >> > be
> >> > > > able
> >> > > > > > to
> >> > > > > > > > get
> >> > > > > > > > > > > > > > >>>>> away with no new
> >> > > > > > > > > > > > API
> >> > > > > > > > > > > > > > >>>>> calls and simply implement a new custom
> >> > > snapshot
> >> > > > > > > strategy
> >> > > > > > > > > > > > > > >>>>> and
> >> > > > > > > > > > > > data
> >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case where
> >> the
> >> > > > > > CloudStack
> >> > > > > > > > > > > > > > >>>>> cloud
> >> > > > > > > > > > > > does
> >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> >> exported-to-NFS
> >> > > > > backup.
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> In other words, the "default" behavior
> >> would
> >> > be
> >> > > > to
> >> > > > > > use
> >> > > > > > > > the
> >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> strategy
> >> > that
> >> > > > we
> >> > > > > > > > already
> >> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> >> > snapshot).
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however, wants
> >> to
> >> > > take
> >> > > > a
> >> > > > > > SAN
> >> > > > > > > > > > > > > > >>>>> snapshot and have the data exported to
> >> NFS,
> >> > > then
> >> > > > we
> >> > > > > > > could
> >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config file
> to
> >> > make
> >> > > > use
> >> > > > > > of
> >> > > > > > > a
> >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data motion
> >> > strategy
> >> > > > that
> >> > > > > > > > > > > > > > >>>>> performs both of these
> >> > > > > > > > > > > > > > activities.
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> This way, the old behavior is still the
> >> > default
> >> > > > for
> >> > > > > > > > users,
> >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change this
> >> > behavior
> >> > > > via
> >> > > > > > > > > > > configuration.
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> Thoughts?
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike
> >> > > Tutkowski <
> >> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> >> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to come up
> >> > with a
> >> > > > > > viable
> >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way for
> >> them
> >> > > to
> >> > > > > move
> >> > > > > > > > from
> >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and some
> >> obvious
> >> > > way
> >> > > > > > that
> >> > > > > > > > > > > > > > >>>>>> they will know they need
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > do this).
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed
> >> > Mushtaq
> >> > > <
> >> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the upgrade
> >> path
> >> > > > > however,
> >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and are
> on a
> >> > > > managed
> >> > > > > > > > storage
> >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> >> functionality
> >> > > > with
> >> > > > > > > Volume
> >> > > > > > > > > > > Snapshots.
> >> > > > > > > > > > > > > > >>>>>>>
> >> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed
> >> > Mushtaq
> >> > > <
> >> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com
> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly,
> currently
> >> > > > taking a
> >> > > > > > > > Volume
> >> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a managed
> >> storage
> >> > > > keeps
> >> > > > > > it
> >> > > > > > > on
> >> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part of this
> >> > > feature,
> >> > > > we
> >> > > > > > can
> >> > > > > > > > > > > > > > >>>>>>>> make sure
> >> > > > > > > > > > > > that
> >> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed storage
> are
> >> > > > uploaded
> >> > > > > > to
> >> > > > > > > > the
> >> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would make
> the
> >> > > Volume
> >> > > > > > > Snapshot
> >> > > > > > > > > > > > > > >>>>>>>> feature behave the same regardless of
> >> the
> >> > > > > storage
> >> > > > > > > > > > > > > > >>>>>>>> (managed or
> >> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing the
> >> > > efficient
> >> > > > > > > backend
> >> > > > > > > > > > > > > > >>>>>>>> storage
> >> > > > > > > > > > > > > > capabilities, we can use the new storage
> >> snapshots
> >> > > API.
> >> > > > > > > > > > > > > > >>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike
> >> > > > Tutkowski <
> >> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> >> > <javascript:;>
> >> > > > > <javascript:;>>
> >> > > > > > wrote:
> >> > > > > > > > > > > > > > >>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need to
> have a
> >> > plan
> >> > > > to
> >> > > > > > deal
> >> > > > > > > > > > > > > > >>>>>>>>> with the fact that we already have a
> >> > > feature
> >> > > > > (in
> >> > > > > > > 4.6
> >> > > > > > > > > > > > > > >>>>>>>>> and
> >> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to use the
> >> > existing
> >> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a
> >> volume
> >> > > > > snapshot
> >> > > > > > > (for
> >> > > > > > > > > > > > > > >>>>>>>>> managed
> >> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a backend
> SAN
> >> > > > (using a
> >> > > > > > > > custom
> >> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom data
> >> > motion
> >> > > > > > > strategy).
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then how
> >> should
> >> > > the
> >> > > > > > > original
> >> > > > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6 and
> >> later)
> >> > > be
> >> > > > > > > changed?
> >> > > > > > > > > > > > > > >>>>>>>>> If it
> >> > > > > > > > > > > > is
> >> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support customers
> >> who
> >> > > are
> >> > > > > > > already
> >> > > > > > > > > > > > > > >>>>>>>>> using
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API to take
> >> > > > snapshots
> >> > > > > > on a
> >> > > > > > > > > > > > > > >>>>>>>>> backend
> >> > > > > > > > > > > > > SAN?
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> >> > > > > > > > > > > > > > >>>>>>>>> Mike
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM,
> Will
> >> > > > Stevens <
> >> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com
> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create a
> Template
> >> > > from a
> >> > > > > > > > > > > > StorageSnapshot?
> >> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template be stored
> >> in
> >> > the
> >> > > > > > > secondary
> >> > > > > > > > > > > > > > >>>>>>>>>> storage like normal templates or
> will
> >> > that
> >> > > > be
> >> > > > > > > > handled
> >> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> >> > > > > > > > > > > > > > vendor side?
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> >> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions
> >> Experts
> >> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec
> >> *|*
> >> > > H3J
> >> > > > > 1S6
> >> > > > > > w
> >> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM,
> Syed
> >> > > > Mushtaq <
> >> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> >> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> >> > > > > > > > > > > > > > >>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM,
> Will
> >> > > > Stevens
> >> > > > > <
> >> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
> >> <javascript:;>
> >> > > > > <javascript:;>> wrote:
> >> > > > > > > > > > > > > > >>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design
> >> Spec in
> >> > > the
> >> > > > > > Jira
> >> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not clear
> in
> >> the
> >> > > > > > 'mention'
> >> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows as a
> page
> >> > 'you
> >> > > do
> >> > > > > not
> >> > > > > > > > have
> >> > > > > > > > > > > permission to'.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> >> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions
> >> Experts
> >> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|*
> Quebec
> >> > *|*
> >> > > > H3J
> >> > > > > > 1S6
> >> > > > > > > w
> >> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM,
> >> Syed
> >> > > Ahmed
> >> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> >> <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > >
> >> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of
> >> APIs
> >> > to
> >> > > > do
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage
> >> backends
> >> > > > like
> >> > > > > > > > > SolidFire.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed
> >> storage
> >> > > stay
> >> > > > > on
> >> > > > > > > the
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary to what
> >> > > > CloudStack
> >> > > > > > > calls
> >> > > > > > > > > > > snpshots.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage
> >> and
> >> > > > keeping
> >> > > > > > it
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> >> > > > > > > > > > > > and
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to have
> both
> >> > ways
> >> > > > of
> >> > > > > > > doing
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4
> >> new
> >> > > APIs
> >> > > > to
> >> > > > > > > > create
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of this
> >> > > feature? I
> >> > > > > > would
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am
> >> > > working
> >> > > > on
> >> > > > > > > > making
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> >> > > > > > > > > > > > spec
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to
> have a
> >> > high
> >> > > > > level
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before starting
> to
> >> > work
> >> > > on
> >> > > > > it.
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> >> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>> --
> >> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> >> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer,
> >> SolidFire
> >> > > Inc.*
> >> > > > > > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> >> > > > <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> >> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world uses the
> >> > cloud
> >> > > > > > > > > > > > > > >>>>>>>>> <
> >> > > > > > > http://solidfire.com/solution/overview/?video=play
> >> > > > > > > > >*™
> >> > > > > > > > > > > > > > >>>>>>>>> *
> >> > > > > > > > > > > > > > >>>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>>
> >> > > > > > > > > > > > > > >>>>>>>
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>> --
> >> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> >> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire
> >> > Inc.*
> >> > > > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> >> > > <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> >> > > > > > > > > > > > > > >>>>>> Advancing the way the world uses the
> >> cloud
> >> > > > > > > > > > > > > > >>>>>> <
> >> > > > > http://solidfire.com/solution/overview/?video=play
> >> > > > > > > >*™*
> >> > > > > > > > > > > > > > >>>>>>
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>> --
> >> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> >> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire
> >> Inc.*
> >> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> >> > <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> >> > > > > > > > > > > > > > >>>>> Advancing the way the world uses the
> cloud
> >> > > > > > > > > > > > > > >>>>> <
> >> > > > > http://solidfire.com/solution/overview/?video=play
> >> > > > > > > >*™*
> >> > > > > > > > > > > > > > >>>>>
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>>
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>> --
> >> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> >> > > > > > > > > > > > > > >>> *Senior CloudStack Developer, SolidFire
> >> Inc.*
> >> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> >> <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > >>> o: 303.746.7302
> >> > > > > > > > > > > > > > >>> Advancing the way the world uses the cloud
> >> > > > > > > > > > > > > > >>> <
> >> > > > http://solidfire.com/solution/overview/?video=play
> >> > > > > > >*™*
> >> > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > --
> >> > > > > > > > > > > > > > > *Mike Tutkowski*
> >> > > > > > > > > > > > > > > *Senior CloudStack Developer, SolidFire
> Inc.*
> >> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
> >> <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > > > > o: 303.746.7302
> >> > > > > > > > > > > > > > > Advancing the way the world uses the cloud
> >> > > > > > > > > > > > > > > <
> >> > > http://solidfire.com/solution/overview/?video=play
> >> > > > > >*™*
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Find out more about ShapeBlue and our range of
> >> > > > CloudStack
> >> > > > > > > > > > > > > > related
> >> > > > > > > > > > > > > services:
> >> > > > > > > > > > > > > > IaaS Cloud Design & Build
> >> > > > > > > > > > > > > > <
> >> > http://shapeblue.com/iaas-cloud-design-and-build//>
> >> > > |
> >> > > > > > > > CSForge –
> >> > > > > > > > > > > > > > rapid IaaS deployment framework <
> >> > > > > > > http://shapeblue.com/csforge/
> >> > > > > > > > >
> >> > > > > > > > > > > > > > CloudStack Consulting
> >> > > > > > > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/
> >
> >> |
> >> > > > > > > > > > > > > CloudStack
> >> > > > > > > > > > > > > > Software Engineering
> >> > > > > > > > > > > > > > <
> >> > > http://shapeblue.com/cloudstack-software-engineering/
> >> > > > >
> >> > > > > > > > > > > > > > CloudStack Infrastructure Support
> >> > > > > > > > > > > > > > <
> >> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> >> > > > > >
> >> > > > > > |
> >> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> >> > > > > > > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > --
> >> > > > > > > > > > > > *Mike Tutkowski*
> >> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> >> > > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> >> > > > > <javascript:;>
> >> > > > > > > > > > > > o: 303.746.7302
> >> > > > > > > > > > > > Advancing the way the world uses the cloud
> >> > > > > > > > > > > > <
> http://solidfire.com/solution/overview/?video=play
> >> > >*™*
> >> > > > > > > > > > > >
> >> > > > > > > > > > > Find out more about ShapeBlue and our range of
> >> CloudStack
> >> > > > > related
> >> > > > > > > > > > services:
> >> > > > > > > > > > > IaaS Cloud Design & Build
> >> > > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//
> >
> >> |
> >> > > > > CSForge –
> >> > > > > > > > rapid
> >> > > > > > > > > > > IaaS deployment framework <
> >> http://shapeblue.com/csforge/
> >> > >
> >> > > > > > > CloudStack
> >> > > > > > > > > > > Consulting <
> >> http://shapeblue.com/cloudstack-consultancy/
> >> > >
> >> > > |
> >> > > > > > > > CloudStack
> >> > > > > > > > > > > Software Engineering
> >> > > > > > > > > > > <
> >> http://shapeblue.com/cloudstack-software-engineering/>
> >> > > > > > > > > > > CloudStack Infrastructure Support
> >> > > > > > > > > > > <
> >> http://shapeblue.com/cloudstack-infrastructure-support/
> >> > >
> >> > > |
> >> > > > > > > > CloudStack
> >> > > > > > > > > > > Bootcamp Training Courses <
> >> > > > > > > http://shapeblue.com/cloudstack-training/
> >> > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > Find out more about ShapeBlue and our range of
> >> CloudStack
> >> > > > related
> >> > > > > > > > > services:
> >> > > > > > > > > > IaaS Cloud Design & Build
> >> > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//>
> |
> >> > > > CSForge –
> >> > > > > > > rapid
> >> > > > > > > > > > IaaS deployment framework <
> >> http://shapeblue.com/csforge/>
> >> > > > > > > > > > CloudStack Consulting <
> >> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> >> > > > > > > |
> >> > > > > > > > > CloudStack
> >> > > > > > > > > > Software Engineering
> >> > > > > > > > > > <
> http://shapeblue.com/cloudstack-software-engineering/>
> >> > > > > > > > > > CloudStack Infrastructure Support
> >> > > > > > > > > > <
> >> http://shapeblue.com/cloudstack-infrastructure-support/>
> >> > |
> >> > > > > > > CloudStack
> >> > > > > > > > > > Bootcamp Training Courses <
> >> > > > > > http://shapeblue.com/cloudstack-training/
> >> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > *Mike Tutkowski*
> >> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> >> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> >> > > > > > > o: 303.746.7302
> >> > > > > > > Advancing the way the world uses the cloud
> >> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > *Mike Tutkowski*
> >> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> >> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> >> > > > > o: 303.746.7302
> >> > > > > Advancing the way the world uses the cloud
> >> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Ian Rae
> >> > > > CEO | PDG
> >> > > > c: 514.944.4008
> >> > > >
> >> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> >> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > *Mike Tutkowski*
> >> > *Senior CloudStack Developer, SolidFire Inc.*
> >> > e: mike.tutkowski@solidfire.com
> >> > o: 303.746.7302
> >> > Advancing the way the world uses the cloud
> >> > <http://solidfire.com/solution/overview/?video=play>*™*
> >> >
> >>
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Now that I re-read your e-mail, it dawned on me: The end-user doesn't care
where the snapshot is.

If that's true, then we should perhaps control this via Global Settings or
something.

On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
mike.tutkowski@solidfire.com> wrote:

> It's not ideal - true, but it does allow us to be backward compatible.
>
> If you have other ideas, though, about how to maintain backward
> compatibility, I'm definitely open to hear them.
>
> Thanks!
>
> On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
>> Hi Mike,
>>
>> Adding a flag to createSnapshot was the first and the most obvious thing
>> that came to our minds. The problem that I had with this was that:
>>
>> 1) I feel it is exposing something to the end user that is internal to the
>> cloud.
>>
>> 2) We have to follow two different ways of restore/deletion in the same
>> code path depending on where the Snapshot resides which I find kind of a
>> bad design.
>>
>> But if exposing a archive flag to the end user is acceptable then we can
>> definitely use this instead of adding the StorageSnapshot API
>>
>> Thanks,
>> -Syed
>>
>>
>> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
>> mike.tutkowski@solidfire.com
>> > wrote:
>>
>> > Hi Pierre-Luc,
>> >
>> > My recommendation would be this:
>> >
>> > Add an "archive" flag to the current volume-snapshot API. Its default
>> would
>> > be "false" because that would be backward compatible with how 4.6 has
>> > volume snapshots implemented (i.e. they stay on the SAN in 4.6, 4.7, and
>> > 4.8).
>> >
>> > If you set archive=true, then we would perform a background migration of
>> > the snapshot from the SAN to the secondary storage (then delete the SAN
>> > snapshot).
>> >
>> > That archive parameter would only be valid for managed storage.
>> >
>> > Sound reasonable?
>> >
>> > Also, a VM snapshot that includes disks provided by managed storage
>> should
>> > work.
>> >
>> > Talk to you later,
>> > Mike
>> >
>> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <pd...@cloudops.com>
>> > wrote:
>> >
>> > > Mike,
>> > >
>> > > In terms of API's, would you prefer introducing a parameter to the
>> > existing
>> > > VolumeSnapshot, example:   extract={true|false}  with a default value
>> of
>> > > true  which would extract snapshot into the secondary storage, which
>> is
>> > the
>> > > current default behavior. Then for SAN snapshot that remain on the
>> SAN we
>> > > would just set "extract=false" ?  as oppose to create a new
>> > >  StorageSnapshot API ?
>> > >
>> > >
>> > > Paul,
>> > >
>> > > From what I'm seeing so far, we can't do a VM-snapshot when using
>> managed
>> > > storage for VM having more than one Volume. For the reason that
>> snapshot
>> > > are performed outside of the hypervisor awareness and asynchronously.
>> If
>> > > someone have a way to address that, it would make thinks much more
>> > > attractive.
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com> wrote:
>> > >
>> > > > I think a service provider backup scenario is more likely to take
>> > > advantage
>> > > > of SAN snapshot. There are a few reasons for this. Traditional
>> backups
>> > > > involve access to the file system, and there is an expectation that
>> > this
>> > > > can be done with reasonably short time frames without negatively
>> > > impacting
>> > > > VM performance, and that the backup orchestrator can apply various
>> > logic
>> > > > and or transformations to the data (compress, encrypt, deltas
>> etc...).
>> > > > While it is true that one could apply a backup process to a cloud
>> > > snapshot,
>> > > > this would be slow and inefficient requiring the data to be moved
>> > several
>> > > > times and there are some major bottlenecks with cloud snapshots.
>> With a
>> > > > cloud snapshot - there seems to be no reasonable expectation of
>> being
>> > > able
>> > > > to do differential snapshots (I think this depends on the
>> hypervisor)
>> > and
>> > > > if you do differential snapshots this will make file backups from
>> those
>> > > > snapshots even more complicated to orchestrate.
>> > > >
>> > > > Suspect there needs to be a different thread on how to better enable
>> > > > backups, as a service. As per Paul's suggestion, but it is a related
>> > > > workflow so relevant to this discussion.
>> > > >
>> > > > Ian
>> > > >
>> > > > On Monday, February 8, 2016, Mike Tutkowski <
>> > > mike.tutkowski@solidfire.com>
>> > > > wrote:
>> > > >
>> > > > > To me it sounds like number two and number three are different
>> uses
>> > for
>> > > > the
>> > > > > same "thing"(which is totally fine).
>> > > > >
>> > > > > As for taking a fast SAN snapshot and exporting it
>> asynchronously, do
>> > > we
>> > > > > see the SSVM as performing the export?
>> > > > >
>> > > > > To be backwards compatible with what we have in 4.6 and later for
>> > > volume
>> > > > > snapshots for managed storage, I think it might be easier if we
>> pass
>> > > in a
>> > > > > flag that says whether or not to archive the SAN snapshot (which,
>> I
>> > > > think,
>> > > > > is something that you suggested, as well, Pierre-Luc).
>> > > > >
>> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <pdion@cloudops.com
>> > > > > <javascript:;>> wrote:
>> > > > >
>> > > > > > Hi Mike,
>> > > > > >
>> > > > > > The reason behind the creation of a SAN snapshot which is
>> exported
>> > > into
>> > > > > > secondary storage, is because creating a copy of the .VHD
>> directly
>> > > > would
>> > > > > > impact uptime of the VM as creating that copy take lots of time.
>> > Has
>> > > > > oppose
>> > > > > > to a SAN snapshot that is close to instantaneous which can
>> > afterward
>> > > be
>> > > > > > clone into Secondary Storage asynchronously.
>> > > > > >
>> > > > > > I would suspect an extracted VolumeSnapshot taken from a SAN
>> > snapshot
>> > > > > could
>> > > > > > have is SAN snapshot deleted to avoid duplica and space
>> consumption
>> > > on
>> > > > > the
>> > > > > > Primary Storage side.
>> > > > > >
>> > > > > >
>> > > > > > I see 3 definitions in our current discussion regarding the term
>> > > > snapshot
>> > > > > > (these are not official terminology but by own interpretation of
>> > > them):
>> > > > > >
>> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a
>> > > > snapshot):
>> > > > > > it's a volume snapshot at the storage level, point in time of
>> your
>> > > > data.
>> > > > > it
>> > > > > > reside on the primary storage. Useful and efficient for software
>> > side
>> > > > > > incident.
>> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud
>> backup
>> > > > aws-S3
>> > > > > > style ): Point in time copy of the Virtual Disk that reside on a
>> > > > > different
>> > > > > > storage array then the original Volume. Facilitate data
>> migration
>> > > > between
>> > > > > > clusters and, in case of primary storage incident, Volume
>> snapshots
>> > > are
>> > > > > not
>> > > > > > impacted and can be reuse.
>> > > > > > 3. *Backup*: Archival of your Virtual-machines data that also
>> > > validate
>> > > > > data
>> > > > > > integrity, provide a storage efficient archiving method and an
>> > > > > independent
>> > > > > > way to restore your data in case of an major infrastructure
>> > disaster.
>> > > > > >
>> > > > > >
>> > > > > > Regards,
>> > > > > >
>> > > > > > PL
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
>> > > > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
>> > > > > > > wrote:
>> > > > > >
>> > > > > > > So, let's see if I currently follow the requirements:
>> > > > > > >
>> > > > > > > * Augment volume snapshots for managed storage to
>> conditionally
>> > > > export
>> > > > > > data
>> > > > > > > to NFS. The current process of taking a snapshot on the SAN is
>> > > fine,
>> > > > > but
>> > > > > > > we'd like the option to export the data to NFS, as well.
>> > > > > > >
>> > > > > > > Questions:
>> > > > > > >
>> > > > > > > Once the data has been exported to NFS, do we keep the SAN
>> > snapshot
>> > > > or
>> > > > > > > delete it?
>> > > > > > >
>> > > > > > > If we are deleting the SAN snapshot, then why don't we just
>> copy
>> > > the
>> > > > > VHD
>> > > > > > > from primary to secondary the way we do today for non-managed
>> > (i.e.
>> > > > > > > traditional) storage? Why create a SAN snapshot in that
>> scenario?
>> > > > > Perhaps
>> > > > > > > to have the SSVM mount and perform the VHD copy to secondary
>> > > storage
>> > > > > > > instead of a XenServer host?
>> > > > > > >
>> > > > > > > Thanks for the clarification.
>> > > > > > >
>> > > > > > > By the way, to me a backup is when you copy data from one
>> storage
>> > > > > system
>> > > > > > to
>> > > > > > > another (regardless of features, if any, to restore the data
>> in
>> > the
>> > > > > > > future). A snapshot is a point-in-time view of the data of a
>> > volume
>> > > > and
>> > > > > > > it's stored on the same storage system as the volume.
>> > > > > > >
>> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
>> > > pdion@cloudops.com
>> > > > > <javascript:;>
>> > > > > > <javascript:;>>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > That's fun to see that discussion happening. I 100% agree
>> with
>> > > > Paul's
>> > > > > > > > points of view. VolumeSnapshot are not a backup, but I do
>> > > consider
>> > > > > them
>> > > > > > > as
>> > > > > > > > a safety vest against Primary Storage failure, because
>> failure
>> > > > append
>> > > > > > > :-( .
>> > > > > > > >
>> > > > > > > > The current proposal around snapshots that reside on the
>> > primary
>> > > > > > storage
>> > > > > > > or
>> > > > > > > > snapshots that end in the Secondary Storage  is not to
>> address
>> > > any
>> > > > > kind
>> > > > > > > of
>> > > > > > > > backups requirement because a snapshot is not a backup,
>> event
>> > an
>> > > > > > > extracted
>> > > > > > > > VM snapshot.
>> > > > > > > >
>> > > > > > > > The main idea, and again this is for managed storage;
>> > > > > > > >
>> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
>> capability
>> > > for
>> > > > > > fast
>> > > > > > > > response time that support rollback to previous timestamp,
>> > create
>> > > > new
>> > > > > > > > volume and maybe create template.
>> > > > > > > >     not required to be a new API if the work is already
>> done, I
>> > > > think
>> > > > > > > this
>> > > > > > > > is a different behaviors than the user expectation of a
>> > > > > > volume-snapshot.
>> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior
>> that
>> > > > create
>> > > > > > an
>> > > > > > > > extraction of a volume into SecondaryStorage which can be
>> reuse
>> > > to
>> > > > > > > create a
>> > > > > > > > new volume into another Primary Storage. This type of
>> snapshot
>> > > is a
>> > > > > > slow
>> > > > > > > > job since yes it would have to copy the full volume size on
>> the
>> > > > > > Secondary
>> > > > > > > > storage.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > PL
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
>> > > > > syed1.mushtaq@gmail.com <javascript:;>
>> > > > > > <javascript:;>>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I think I share you view on the 'Ideal world'. Backup (via
>> > > Volume
>> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is
>> > > amplified
>> > > > > > > > especially
>> > > > > > > > > when you have a object storage as your secondary storage
>> > > because
>> > > > it
>> > > > > > > > > requires two copies (one to an NFS staging area and from
>> > there
>> > > to
>> > > > > > > object
>> > > > > > > > > storage). And not to mention that all these copies are
>> > > consuming
>> > > > > > > > hypervisor
>> > > > > > > > > resources. Xenserver's Dom0 is also a huge bottleneck as
>> all
>> > > the
>> > > > > > > Network
>> > > > > > > > > and I/O flow through it. So our intention of proposing the
>> > > > "Storage
>> > > > > > > > > Snapshots" is to give a better way of achiving snapshots
>> > while
>> > > > > still
>> > > > > > > > > keeping the original definition of volume snpashots (ie
>> > upload
>> > > to
>> > > > > sec
>> > > > > > > > > storage).
>> > > > > > > > >
>> > > > > > > > > But as Erik pointed out volume snapshots are not backups.
>> > They
>> > > > > don't
>> > > > > > > work
>> > > > > > > > > form multi-disk LVM volume groups and dynamic disks. I am
>> all
>> > > in
>> > > > > for
>> > > > > > a
>> > > > > > > > > better backup solution which handles these use cases and
>> > > utilizes
>> > > > > the
>> > > > > > > > > storage's advanced features.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
>> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > In the beginning... there were CloudStack snapshots and
>> > they
>> > > > were
>> > > > > > > > > actually
>> > > > > > > > > > volume snapshots not hypervisor point-in-time snapshots.
>> > > > > > > > > > Then VM snapshots were created (which are point-in-time
>> > > > > hypervisor
>> > > > > > > > > > snapshots) and we started referring to the original
>> > snapshots
>> > > > as
>> > > > > > > volume
>> > > > > > > > > > snapshots.
>> > > > > > > > > >
>> > > > > > > > > > CloudStack does not offer 'backups', but many people use
>> > > volume
>> > > > > > > > snapshots
>> > > > > > > > > > as backups. However you can't in-place restore volume
>> > > snapshots
>> > > > > and
>> > > > > > > if
>> > > > > > > > > you
>> > > > > > > > > > have a VM with multiple volumes, the volume snapshots
>> must
>> > be
>> > > > > done
>> > > > > > in
>> > > > > > > > > > series, meaning that the state across all of the
>> volumes is
>> > > > > > unlikely
>> > > > > > > to
>> > > > > > > > > be
>> > > > > > > > > > consistent.
>> > > > > > > > > >
>> > > > > > > > > > 'Actual Backups' would enable all of the restore options
>> > > which
>> > > > > > users
>> > > > > > > > > might
>> > > > > > > > > > expect as well options as to where they might be
>> stored. In
>> > > my
>> > > > > > ideal
>> > > > > > > > > world
>> > > > > > > > > > they would also be able to leverage back-end hardware
>> (such
>> > > as
>> > > > > > > > Solidfire,
>> > > > > > > > > > NetApp etc :) ) and software such as Veeam, Commvault
>> etc
>> > to
>> > > > > > > accelerate
>> > > > > > > > > the
>> > > > > > > > > > process.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
>> > > > > > > > > > Paul Angus
>> > > > > > > > > > VP Technology ,  ShapeBlue
>> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
>> > > > > > > > > >
>> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
>> > > > > m:
>> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
>> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
>> > <javascript:;>
>> > > |
>> > > > > t: @cloudyangus*
>> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
>> > > > > <ja...@cloudyangus>
>> > > > > > |  w:
>> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
>> > > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
>> > > > > > > > > > Shape Blue Ltd is a company incorporated in England &
>> > Wales.
>> > > > > > > ShapeBlue
>> > > > > > > > > > Services India LLP is a company incorporated in India
>> and
>> > is
>> > > > > > operated
>> > > > > > > > > under
>> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
>> Consultoria
>> > > Ltda
>> > > > > is
>> > > > > > a
>> > > > > > > > > > company incorporated in Brasil and is operated under
>> > license
>> > > > from
>> > > > > > > Shape
>> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>> by
>> > The
>> > > > > > > Republic
>> > > > > > > > of
>> > > > > > > > > > South Africa and is traded under license from Shape Blue
>> > Ltd.
>> > > > > > > ShapeBlue
>> > > > > > > > > is
>> > > > > > > > > > a registered trademark.
>> > > > > > > > > > This email and any attachments to it may be confidential
>> > and
>> > > > are
>> > > > > > > > intended
>> > > > > > > > > > solely for the use of the individual to whom it is
>> > addressed.
>> > > > Any
>> > > > > > > views
>> > > > > > > > > or
>> > > > > > > > > > opinions expressed are solely those of the author and do
>> > not
>> > > > > > > > necessarily
>> > > > > > > > > > represent those of Shape Blue Ltd or related companies.
>> If
>> > > you
>> > > > > are
>> > > > > > > not
>> > > > > > > > > the
>> > > > > > > > > > intended recipient of this email, you must neither take
>> any
>> > > > > action
>> > > > > > > > based
>> > > > > > > > > > upon its contents, nor copy or show it to anyone. Please
>> > > > contact
>> > > > > > the
>> > > > > > > > > sender
>> > > > > > > > > > if you believe you have received this email in error.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > -----Original Message-----
>> > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
>> > > > > <javascript:;> <javascript:;>]
>> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
>> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
>> > <javascript:;>
>> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
>> > > > > > > > > >
>> > > > > > > > > > Paul,
>> > > > > > > > > >
>> > > > > > > > > > When you say actual backups, how would it be different
>> from
>> > > the
>> > > > > > > Volume
>> > > > > > > > > > Snapshots that exist currently. My understanding is that
>> > > > Backups
>> > > > > > end
>> > > > > > > up
>> > > > > > > > > in
>> > > > > > > > > > Sec Storage whereas Snapshots are just a point-in-time
>> > state
>> > > of
>> > > > > > your
>> > > > > > > > > volume
>> > > > > > > > > > which can be restored back correct?
>> > > > > > > > > >
>> > > > > > > > > > -Syed
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
>> > > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Syed,
>> > > > > > > > > > >
>> > > > > > > > > > > As I understand it, the SolidFire plugin will export
>> the
>> > > > > snapshot
>> > > > > > > to
>> > > > > > > > > > > secondary storage if the user requests a template from
>> > the
>> > > > > > snapshot
>> > > > > > > > or
>> > > > > > > > > > > wants to download the snapshot from the cloud. This
>> is a
>> > > > good,
>> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire storage
>> is
>> > > > super
>> > > > > > > > > > > reliable and snapshots on SolidFire arrays take up
>> next
>> > to
>> > > no
>> > > > > > > space.
>> > > > > > > > > > > BUT I think that we are talking about a more general
>> > > purpose
>> > > > > API,
>> > > > > > > and
>> > > > > > > > > > > other storage systems may not be as awesome as Mike's.
>> > > That's
>> > > > > my
>> > > > > > > > > > > concern. Also, the time to transfer for say 1TB to
>> move
>> > > from
>> > > > > > > primary
>> > > > > > > > > > > to sec storage and then create a VM template out of it
>> > may
>> > > be
>> > > > > too
>> > > > > > > > long
>> > > > > > > > > > for users.
>> > > > > > > > > > >
>> > > > > > > > > > > @Mike I don’t think 'we' use the term volume snapshot
>> for
>> > > > > backup,
>> > > > > > > > it's
>> > > > > > > > > > > just that users want to do backups and a volume
>> snapshot
>> > is
>> > > > the
>> > > > > > > only
>> > > > > > > > > > > type of snapshot that copies the disk elsewhere and
>> can
>> > be
>> > > > > > > scheduled.
>> > > > > > > > > > >
>> > > > > > > > > > > I'm 'pondering' the implications of enabling actual
>> > backups
>> > > > > > > (through
>> > > > > > > > > > > recognised backup providers) and the user requirements
>> > > around
>> > > > > > them
>> > > > > > > > > > > (particularly restoration use cases) as a separate
>> thread
>> > > of
>> > > > > > work.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul
>> Angus
>> > > VP
>> > > > > > > > Technology
>> > > > > > > > > > > , ShapeBlue
>> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
>> > > > > > > > > > >
>> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
>> > > > |
>> > > > > m:
>> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
>> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
>> > > <javascript:;> |
>> > > > > t: @cloudyangus*
>> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
>> <javascript:;>
>> > > > > > %20%7C%20t:%20@cloudyangus> | w:
>> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
>> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
>> > Shape
>> > > > > Blue
>> > > > > > > Ltd
>> > > > > > > > > > > is a company incorporated in England & Wales.
>> ShapeBlue
>> > > > > Services
>> > > > > > > > India
>> > > > > > > > > > > LLP is a company incorporated in India and is operated
>> > > under
>> > > > > > > license
>> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria
>> Ltda
>> > is
>> > > a
>> > > > > > > company
>> > > > > > > > > > > incorporated in Brasil and is operated under license
>> from
>> > > > Shape
>> > > > > > > Blue
>> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by
>> The
>> > > > > Republic
>> > > > > > > of
>> > > > > > > > > > > South Africa and is traded under license from Shape
>> Blue
>> > > Ltd.
>> > > > > > > > > > > ShapeBlue is a registered trademark.
>> > > > > > > > > > > This email and any attachments to it may be
>> confidential
>> > > and
>> > > > > are
>> > > > > > > > > > > intended solely for the use of the individual to whom
>> it
>> > is
>> > > > > > > > addressed.
>> > > > > > > > > > > Any views or opinions expressed are solely those of
>> the
>> > > > author
>> > > > > > and
>> > > > > > > do
>> > > > > > > > > > > not necessarily represent those of Shape Blue Ltd or
>> > > related
>> > > > > > > > > > > companies. If you are not the intended recipient of
>> this
>> > > > email,
>> > > > > > you
>> > > > > > > > > > > must neither take any action based upon its contents,
>> nor
>> > > > copy
>> > > > > or
>> > > > > > > > show
>> > > > > > > > > > > it to anyone. Please contact the sender if you believe
>> > you
>> > > > have
>> > > > > > > > > received
>> > > > > > > > > > this email in error.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > -----Original Message-----
>> > > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
>> > > > > <javascript:;>
>> > > > > > <javascript:;>]
>> > > > > > > > > > > Sent: 05 February 2016 15:31
>> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
>> > > <javascript:;>
>> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
>> > > > > > > > > > >
>> > > > > > > > > > > I think the terminology confusion comes from AWS where
>> > they
>> > > > do
>> > > > > > EBS
>> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
>> followed
>> > > > that.
>> > > > > > And
>> > > > > > > > as
>> > > > > > > > > > > an end user who is oblivious to the internals of my
>> > > provider,
>> > > > > my
>> > > > > > > > > > > expectation would be something similar to what AWS as
>> > that
>> > > is
>> > > > > my
>> > > > > > > > > > > biggest reference point.
>> > > > > > > > > > >
>> > > > > > > > > > > To your point Mike, I agree that a Primary Storage
>> > failure
>> > > on
>> > > > > > > > > > > SolidFire is unlikely, there are other motivations
>> for us
>> > > to
>> > > > > push
>> > > > > > > > data
>> > > > > > > > > > > to secondary storage. Primary storage (atleast for us)
>> > > costs
>> > > > > > > around 3
>> > > > > > > > > > > times as much as secondary storage and snapshots on
>> > primary
>> > > > > > storage
>> > > > > > > > > > > are rarely used (especially for some of our customers
>> who
>> > > do
>> > > > > > daily
>> > > > > > > > > > backups).
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
>> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
>> > > <javascript:;>>
>> > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Some of the weirdness is around terminology.
>> > > > > > > > > > > >
>> > > > > > > > > > > > For most systems I've worked on, a snapshot and a
>> > backup
>> > > > are
>> > > > > > two
>> > > > > > > > > > > > completely different things (but CloudStack has
>> > > > traditionally
>> > > > > > > used
>> > > > > > > > > > > > the term "volume snapshot" to mean backup).
>> > > > > > > > > > > >
>> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
>> though,
>> > > that
>> > > > > if
>> > > > > > > your
>> > > > > > > > > > > > primary storage is running on SolidFire that it is
>> > > unlikely
>> > > > > > > you'll
>> > > > > > > > > > > > encounter an issue where your primary storage goes
>> > > offline
>> > > > > (and
>> > > > > > > > > > > > you'll even maintain your performance guarantees
>> during
>> > > > > failure
>> > > > > > > > > > > > scenarios and upgrades, as well). That being the
>> case,
>> > it
>> > > > is
>> > > > > > less
>> > > > > > > > > > > > useful to require a backup to Swift (but it's
>> perfectly
>> > > OK
>> > > > if
>> > > > > > > > that's
>> > > > > > > > > > > > what we want to do
>> > > > > > > > > > > here).
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
>> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
>> > <javascript:;>>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Paul,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I believe with the current implementation of
>> > Snapshots
>> > > on
>> > > > > > > managed
>> > > > > > > > > > > > > storage
>> > > > > > > > > > > > > (SolidFire) the snapshots are never exported to
>> the
>> > > > > secondary
>> > > > > > > > > > storage.
>> > > > > > > > > > > > > While this solves the problem of having snapshots
>> > > taking
>> > > > > > > forever
>> > > > > > > > > > > > > to get to sec storage, this leaves us with a
>> > > > > > > > > > > > huge
>> > > > > > > > > > > > > liability if our primary storage goes down. We see
>> > > > > snapshots
>> > > > > > as
>> > > > > > > > > > > > > our recovery path because we store them in Swift
>> > which
>> > > is
>> > > > > > > > reliable
>> > > > > > > > > > > > > and resilient to failures.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > With Storage snpashots our goal is to have Volume
>> > > > snapshots
>> > > > > > > > always
>> > > > > > > > > > > > > backed up to secondary storage and Storage
>> Snapshots
>> > > stay
>> > > > > on
>> > > > > > > the
>> > > > > > > > > > > > > primary
>> > > > > > > > > > > > storage.
>> > > > > > > > > > > > > A provider could potentially mix both these and
>> solve
>> > > the
>> > > > > > > problem
>> > > > > > > > > > > > > that you mentioned where you want to meet user's
>> > > > > expectation
>> > > > > > > of a
>> > > > > > > > > > > > > snapshot (ie backup to sec storage) while having
>> an
>> > > > ability
>> > > > > > to
>> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the device)
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Hope this clarifies things.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > -Syed
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
>> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
>> > > <javascript:;>>
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > HI guys,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Could someone point me to the Jira bug of FS for
>> > the
>> > > > > > > > > > > > > > SAN-snapshot
>> > > > > > > > > > > > feature
>> > > > > > > > > > > > > > in 4.6 which is mentioned.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > From my discussions with users and operators
>> around
>> > > > > > snapshots
>> > > > > > > > > > > > > > I'd make
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > following observations:
>> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
>> long-term
>> > > and
>> > > > > > short
>> > > > > > > > > > > > > > term)
>> > > > > > > > > > > > with
>> > > > > > > > > > > > > > the expectation that they can use them for
>> recovery
>> > > if
>> > > > > > > > required.
>> > > > > > > > > > > > > > b. operators fall back to snapshots if something
>> > has
>> > > > gone
>> > > > > > > wrong
>> > > > > > > > > > > > > > with primary storage.
>> > > > > > > > > > > > > > c. users sometimes want to be able to export
>> > > snapshots
>> > > > as
>> > > > > > > well
>> > > > > > > > > > > > > > as
>> > > > > > > > > > > > create
>> > > > > > > > > > > > > > new VMs from their snapshots
>> > > > > > > > > > > > > > d. snapshots are a currently a massive pain for
>> > > > > operators,
>> > > > > > I
>> > > > > > > > > > > > > > know at
>> > > > > > > > > > > > > least
>> > > > > > > > > > > > > > one public cloud who have snapshots which take 2
>> > days
>> > > > to
>> > > > > > > > > complete.
>> > > > > > > > > > > > > > e. snapshots (as they are) can't be used for
>> > multiple
>> > > > LVM
>> > > > > > > > disks.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I think the process Mike has used in the
>> SolidFire
>> > > > plugin
>> > > > > > > (only
>> > > > > > > > > > > > > > moving
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > disk image to secondary storage when you
>> absolutely
>> > > > have
>> > > > > > to)
>> > > > > > > is
>> > > > > > > > > > > > > > a very
>> > > > > > > > > > > > > good
>> > > > > > > > > > > > > > and pragmatic solution. I wonder what problems
>> an
>> > > > > operator
>> > > > > > > > might
>> > > > > > > > > > > > > experience
>> > > > > > > > > > > > > > if they have an issue with a given primary
>> storage
>> > > pool
>> > > > > in
>> > > > > > a
>> > > > > > > > > > cluster.
>> > > > > > > > > > > > (I
>> > > > > > > > > > > > > > know that that is REALLY unlikely in the
>> SolidFire
>> > > case
>> > > > > > Mike
>> > > > > > > :)
>> > > > > > > > > > > > > > ) And
>> > > > > > > > > > > > if
>> > > > > > > > > > > > > > the transfer from primary to secondary is slow,
>> the
>> > > > time
>> > > > > to
>> > > > > > > > > > > > > > being able
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > create a template or export the volume will be
>> > slow.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > So for me the issue is around making sure that
>> the
>> > > end
>> > > > > > users
>> > > > > > > > > > > > expectations
>> > > > > > > > > > > > > > are met (while improving the speed/efficiency of
>> > the
>> > > > back
>> > > > > > > end)
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
>> Paul
>> > > > Angus
>> > > > > > VP
>> > > > > > > > > > > > > > Technology , ShapeBlue
>> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
>> > > > > > > > > > > > > >
>> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
>> > > > > > > |
>> > > > > > > > m:
>> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
>> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
>> > > > > <javascript:;> | t:
>> > > > > > @cloudyangus*
>> > > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
>> > > > <javascript:;>
>> > > > > > %20%7C%20t:%20@cloudyangus> | w:
>> > > > > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
>> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
>> 4HS
>> > UK
>> > > > > Shape
>> > > > > > > > Blue
>> > > > > > > > > > > > > > Ltd is a company incorporated in England &
>> Wales.
>> > > > > ShapeBlue
>> > > > > > > > > > > > > > Services India LLP is a company incorporated in
>> > India
>> > > > and
>> > > > > > is
>> > > > > > > > > > > > > > operated
>> > > > > > > > > > > > > under
>> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
>> > > > > Consultoria
>> > > > > > > Ltda
>> > > > > > > > > > > > > > is a company incorporated in Brasil and is
>> operated
>> > > > under
>> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty
>> Ltd
>> > is
>> > > a
>> > > > > > > company
>> > > > > > > > > > > > > > registered by The Republic
>> > > > > > > > > > > > of
>> > > > > > > > > > > > > > South Africa and is traded under license from
>> Shape
>> > > > Blue
>> > > > > > Ltd.
>> > > > > > > > > > > > > > ShapeBlue
>> > > > > > > > > > > > > is
>> > > > > > > > > > > > > > a registered trademark.
>> > > > > > > > > > > > > > This email and any attachments to it may be
>> > > > confidential
>> > > > > > and
>> > > > > > > > are
>> > > > > > > > > > > > intended
>> > > > > > > > > > > > > > solely for the use of the individual to whom it
>> is
>> > > > > > addressed.
>> > > > > > > > > > > > > > Any views
>> > > > > > > > > > > > > or
>> > > > > > > > > > > > > > opinions expressed are solely those of the
>> author
>> > and
>> > > > do
>> > > > > > not
>> > > > > > > > > > > > necessarily
>> > > > > > > > > > > > > > represent those of Shape Blue Ltd or related
>> > > companies.
>> > > > > If
>> > > > > > > you
>> > > > > > > > > > > > > > are not
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > intended recipient of this email, you must
>> neither
>> > > take
>> > > > > any
>> > > > > > > > > > > > > > action
>> > > > > > > > > > > > based
>> > > > > > > > > > > > > > upon its contents, nor copy or show it to
>> anyone.
>> > > > Please
>> > > > > > > > contact
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > sender
>> > > > > > > > > > > > > > if you believe you have received this email in
>> > error.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > -----Original Message-----
>> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
>> pdion@cloudops.com
>> > > > > <javascript:;>
>> > > > > > <javascript:;>]
>> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
>> > > > > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
>> > > > > <javascript:;>
>> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
>> > Snapshots
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi Mike,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > The idea of introducing a new API:
>> StorageSnapshot
>> > > for
>> > > > > > > managed
>> > > > > > > > > > > > > > storage
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > because the VolumeSnapshot default, or expected,
>> > > > behavior
>> > > > > > is
>> > > > > > > to
>> > > > > > > > > > > > > > archive snapshots into the Secondary Storage.
>> So a
>> > > > > > > > > > > > > > StorageSnapshot API would be
>> > > > > > > > > > > > > for
>> > > > > > > > > > > > > > snapshot that remain on the managed storage
>> > > appliance.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Quickly looking at the API doc and I don't see a
>> > > strong
>> > > > > > > > > > > > > > requirement for volume snapshots to be moved
>> into
>> > > > > secondary
>> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is not
>> > useful,
>> > > > but
>> > > > > > > both
>> > > > > > > > > > > > > > use cases are required. A snapshot that remain
>> on
>> > the
>> > > > > > managed
>> > > > > > > > > > > > > > storage, and another type of
>> > > > > > > > > > > > snapshot
>> > > > > > > > > > > > > > that end up into the secondary storage. Since
>> > you've
>> > > > > done a
>> > > > > > > lot
>> > > > > > > > > > > > > > of
>> > > > > > > > > > > > work,
>> > > > > > > > > > > > > > might easier to just add a parameter to the
>> current
>> > > > > > snapshot
>> > > > > > > > API
>> > > > > > > > > > > > > > that
>> > > > > > > > > > > > > would
>> > > > > > > > > > > > > > trigger an extraction of the storage snapshot
>> into
>> > > the
>> > > > > > > > secondary
>> > > > > > > > > > > > storage?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > PL
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
>> > > > > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
>> > > > > <javascript:;>
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > I think that all sounds reasonable then -
>> thanks!
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
>> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
>> <javascript:;>>
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >> You are correct Mike in terms of the
>> > requirements.
>> > > > One
>> > > > > > of
>> > > > > > > > our
>> > > > > > > > > > > > earlier
>> > > > > > > > > > > > > > >> iterations on this was to have an argument to
>> > the
>> > > > > create
>> > > > > > > > > > > > > > >> snapshot
>> > > > > > > > > > > > API
>> > > > > > > > > > > > > > >> which decides whether to backup the volume to
>> > sec
>> > > > > > storage
>> > > > > > > > but
>> > > > > > > > > > > > > > >> we realized it would make management of
>> > snapshots
>> > > > > quite
>> > > > > > > > messy
>> > > > > > > > > > > > > > >> so we proposed a new api instead.
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
>> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com <javascript:;>
>> > > > > <javascript:;>>
>> > > > > > > > > > > > > > >> wrote:
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >>> Hi,
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> Just to make sure I understand all the
>> > > requirements
>> > > > > > here:
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> 1) This relates only to managed storage (1:1
>> > > > mapping
>> > > > > > > > between
>> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN volume).
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> 2) We want to take the current (introduced
>> in
>> > > 4.6)
>> > > > > > > > > > > > > > >>> functionality, which creates a snapshot on
>> the
>> > > SAN,
>> > > > > and
>> > > > > > > > > > > > > > >>> extend it via a config option (or
>> > > > > > > > > > > > > > >>> something) to not only take the SAN
>> snapshot,
>> > but
>> > > > to
>> > > > > > copy
>> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to NFS.
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken. It's
>> the
>> > > > backup
>> > > > > to
>> > > > > > > NFS
>> > > > > > > > > > > > > > >>> that is optional.
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> 4) Templates can be created from the
>> snapshot
>> > > > that's
>> > > > > on
>> > > > > > > the
>> > > > > > > > > > > > > > >>> SAN (already works).
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created from
>> the
>> > > > > snapshot
>> > > > > > > > > > > > > > >>> that's on
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > >>> SAN (already works as long as the new
>> > CloudStack
>> > > > > volume
>> > > > > > > > ends
>> > > > > > > > > > > > > > >>> up on the same primary storage).
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> Would we have a need for a storage snapshot
>> API
>> > > > then
>> > > > > or
>> > > > > > > > > > > > > > >>> would that just be the standard volume
>> snapshot
>> > > > > without
>> > > > > > > the
>> > > > > > > > > > > > > > >>> backup to
>> > > > > > > > > > > NFS?
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> Thanks!
>> > > > > > > > > > > > > > >>> Mike
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
>> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
>> > > > > <javascript:;>>
>> > > > > > > > > > > > > > >>> wrote:
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>>> Is it possible to have both functionalities
>> > > > > (snapshot
>> > > > > > on
>> > > > > > > > > > > > > > >>>> SAN & Sec
>> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we would
>> > like
>> > > > to
>> > > > > > have
>> > > > > > > > > both.
>> > > > > > > > > > > > > > >>>> For example, some of our customers want to
>> > > > implement
>> > > > > > > their
>> > > > > > > > > > > > > > >>>> own backup strategies and do encryption to
>> > their
>> > > > > > backups
>> > > > > > > > > > > > > > >>>> which is a perfect use case for Storage
>> > Snapshot
>> > > > > while
>> > > > > > > our
>> > > > > > > > > > > > > > >>>> other customers will still keep using the
>> > > standard
>> > > > > > > volume
>> > > > > > > > > > > snapshot.
>> > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > >>>> To keep things backward compatible, we can
>> > add a
>> > > > > > setting
>> > > > > > > > > > > > > > >>>> which
>> > > > > > > > > > > > says
>> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
>> because,
>> > > after
>> > > > > > all,
>> > > > > > > > you
>> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when doing
>> a
>> > > > Volume
>> > > > > > > > > Snapshot.
>> > > > > > > > > > > > > > >>>> You could stop the process there and not do
>> > the
>> > > > > > upload.
>> > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > >>>> What do you think about this approach?
>> > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > >>>> Thanks,
>> > > > > > > > > > > > > > >>>> -Syed
>> > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
>> > Tutkowski <
>> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com
>> <javascript:;>
>> > > > > <javascript:;>> wrote:
>> > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > >>>>> So, this is just me thinking out load
>> here,
>> > but
>> > > > if
>> > > > > a
>> > > > > > > > given
>> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need to
>> > > provide
>> > > > > > both
>> > > > > > > > the
>> > > > > > > > > > > > ability
>> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it to
>> NFS
>> > (if
>> > > > > just
>> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we
>> might
>> > be
>> > > > able
>> > > > > > to
>> > > > > > > > get
>> > > > > > > > > > > > > > >>>>> away with no new
>> > > > > > > > > > > > API
>> > > > > > > > > > > > > > >>>>> calls and simply implement a new custom
>> > > snapshot
>> > > > > > > strategy
>> > > > > > > > > > > > > > >>>>> and
>> > > > > > > > > > > > data
>> > > > > > > > > > > > > > >>>>> motion strategy to handle the case where
>> the
>> > > > > > CloudStack
>> > > > > > > > > > > > > > >>>>> cloud
>> > > > > > > > > > > > does
>> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
>> exported-to-NFS
>> > > > > backup.
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>> In other words, the "default" behavior
>> would
>> > be
>> > > > to
>> > > > > > use
>> > > > > > > > the
>> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion strategy
>> > that
>> > > > we
>> > > > > > > > already
>> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
>> > snapshot).
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however, wants
>> to
>> > > take
>> > > > a
>> > > > > > SAN
>> > > > > > > > > > > > > > >>>>> snapshot and have the data exported to
>> NFS,
>> > > then
>> > > > we
>> > > > > > > could
>> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config file to
>> > make
>> > > > use
>> > > > > > of
>> > > > > > > a
>> > > > > > > > > > > > > > >>>>> new snapshot strategy and data motion
>> > strategy
>> > > > that
>> > > > > > > > > > > > > > >>>>> performs both of these
>> > > > > > > > > > > > > > activities.
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>> This way, the old behavior is still the
>> > default
>> > > > for
>> > > > > > > > users,
>> > > > > > > > > > > > > > >>>>> but CloudStack admins can change this
>> > behavior
>> > > > via
>> > > > > > > > > > > configuration.
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>> Thoughts?
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike
>> > > Tutkowski <
>> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
>> <javascript:;>
>> > > > > <javascript:;>> wrote:
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>>> Right...I think we will need to come up
>> > with a
>> > > > > > viable
>> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way for
>> them
>> > > to
>> > > > > move
>> > > > > > > > from
>> > > > > > > > > > > > > > >>>>>> the old way to the new way (and some
>> obvious
>> > > way
>> > > > > > that
>> > > > > > > > > > > > > > >>>>>> they will know they need
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > do this).
>> > > > > > > > > > > > > > >>>>>>
>> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed
>> > Mushtaq
>> > > <
>> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>
>> > > > > <javascript:;>> wrote:
>> > > > > > > > > > > > > > >>>>>>
>> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the upgrade
>> path
>> > > > > however,
>> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and are on a
>> > > > managed
>> > > > > > > > storage
>> > > > > > > > > > > > > > >>>>>>> would no longer have the same
>> functionality
>> > > > with
>> > > > > > > Volume
>> > > > > > > > > > > Snapshots.
>> > > > > > > > > > > > > > >>>>>>>
>> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed
>> > Mushtaq
>> > > <
>> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com <javascript:;>
>> > > > > <javascript:;>> wrote:
>> > > > > > > > > > > > > > >>>>>>>
>> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly, currently
>> > > > taking a
>> > > > > > > > Volume
>> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a managed
>> storage
>> > > > keeps
>> > > > > > it
>> > > > > > > on
>> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part of this
>> > > feature,
>> > > > we
>> > > > > > can
>> > > > > > > > > > > > > > >>>>>>>> make sure
>> > > > > > > > > > > > that
>> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed storage are
>> > > > uploaded
>> > > > > > to
>> > > > > > > > the
>> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would make the
>> > > Volume
>> > > > > > > Snapshot
>> > > > > > > > > > > > > > >>>>>>>> feature behave the same regardless of
>> the
>> > > > > storage
>> > > > > > > > > > > > > > >>>>>>>> (managed or
>> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing the
>> > > efficient
>> > > > > > > backend
>> > > > > > > > > > > > > > >>>>>>>> storage
>> > > > > > > > > > > > > > capabilities, we can use the new storage
>> snapshots
>> > > API.
>> > > > > > > > > > > > > > >>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike
>> > > > Tutkowski <
>> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
>> > <javascript:;>
>> > > > > <javascript:;>>
>> > > > > > wrote:
>> > > > > > > > > > > > > > >>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
>> > > > > > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need to have a
>> > plan
>> > > > to
>> > > > > > deal
>> > > > > > > > > > > > > > >>>>>>>>> with the fact that we already have a
>> > > feature
>> > > > > (in
>> > > > > > > 4.6
>> > > > > > > > > > > > > > >>>>>>>>> and
>> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to use the
>> > existing
>> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a
>> volume
>> > > > > snapshot
>> > > > > > > (for
>> > > > > > > > > > > > > > >>>>>>>>> managed
>> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a backend SAN
>> > > > (using a
>> > > > > > > > custom
>> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom data
>> > motion
>> > > > > > > strategy).
>> > > > > > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then how
>> should
>> > > the
>> > > > > > > original
>> > > > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6 and
>> later)
>> > > be
>> > > > > > > changed?
>> > > > > > > > > > > > > > >>>>>>>>> If it
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support customers
>> who
>> > > are
>> > > > > > > already
>> > > > > > > > > > > > > > >>>>>>>>> using
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API to take
>> > > > snapshots
>> > > > > > on a
>> > > > > > > > > > > > > > >>>>>>>>> backend
>> > > > > > > > > > > > > SAN?
>> > > > > > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>> Thanks,
>> > > > > > > > > > > > > > >>>>>>>>> Mike
>> > > > > > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will
>> > > > Stevens <
>> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com <javascript:;>
>> > > > > <javascript:;>> wrote:
>> > > > > > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create a Template
>> > > from a
>> > > > > > > > > > > > StorageSnapshot?
>> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template be stored
>> in
>> > the
>> > > > > > > secondary
>> > > > > > > > > > > > > > >>>>>>>>>> storage like normal templates or will
>> > that
>> > > > be
>> > > > > > > > handled
>> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
>> > > > > > > > > > > > > > vendor side?
>> > > > > > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
>> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
>> > > > > > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions
>> Experts
>> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec
>> *|*
>> > > H3J
>> > > > > 1S6
>> > > > > > w
>> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
>> > > > > > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed
>> > > > Mushtaq <
>> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
>> <javascript:;>
>> > > > > <javascript:;>> wrote:
>> > > > > > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
>> > > > > > > > > > > > > > >>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will
>> > > > Stevens
>> > > > > <
>> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com
>> <javascript:;>
>> > > > > <javascript:;>> wrote:
>> > > > > > > > > > > > > > >>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design
>> Spec in
>> > > the
>> > > > > > Jira
>> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not clear in
>> the
>> > > > > > 'mention'
>> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows as a page
>> > 'you
>> > > do
>> > > > > not
>> > > > > > > > have
>> > > > > > > > > > > permission to'.
>> > > > > > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
>> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
>> > > > > > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions
>> Experts
>> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec
>> > *|*
>> > > > H3J
>> > > > > > 1S6
>> > > > > > > w
>> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
>> > > > > > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM,
>> Syed
>> > > Ahmed
>> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
>> <javascript:;>
>> > > > > <javascript:;>
>> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
>> > > > > > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > >
>> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
>> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
>> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
>> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of
>> APIs
>> > to
>> > > > do
>> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage
>> backends
>> > > > like
>> > > > > > > > > SolidFire.
>> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed
>> storage
>> > > stay
>> > > > > on
>> > > > > > > the
>> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary to what
>> > > > CloudStack
>> > > > > > > calls
>> > > > > > > > > > > snpshots.
>> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage
>> and
>> > > > keeping
>> > > > > > it
>> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to have both
>> > ways
>> > > > of
>> > > > > > > doing
>> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4
>> new
>> > > APIs
>> > > > to
>> > > > > > > > create
>> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of this
>> > > feature? I
>> > > > > > would
>> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am
>> > > working
>> > > > on
>> > > > > > > > making
>> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
>> > > > > > > > > > > > spec
>> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a
>> > high
>> > > > > level
>> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before starting to
>> > work
>> > > on
>> > > > > it.
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
>> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>> --
>> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
>> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer,
>> SolidFire
>> > > Inc.*
>> > > > > > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
>> > > > <javascript:;>
>> > > > > <javascript:;>
>> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
>> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world uses the
>> > cloud
>> > > > > > > > > > > > > > >>>>>>>>> <
>> > > > > > > http://solidfire.com/solution/overview/?video=play
>> > > > > > > > >*™
>> > > > > > > > > > > > > > >>>>>>>>> *
>> > > > > > > > > > > > > > >>>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>>
>> > > > > > > > > > > > > > >>>>>>>
>> > > > > > > > > > > > > > >>>>>>
>> > > > > > > > > > > > > > >>>>>>
>> > > > > > > > > > > > > > >>>>>> --
>> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
>> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire
>> > Inc.*
>> > > > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
>> > > <javascript:;>
>> > > > > <javascript:;>
>> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
>> > > > > > > > > > > > > > >>>>>> Advancing the way the world uses the
>> cloud
>> > > > > > > > > > > > > > >>>>>> <
>> > > > > http://solidfire.com/solution/overview/?video=play
>> > > > > > > >*™*
>> > > > > > > > > > > > > > >>>>>>
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>> --
>> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
>> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire
>> Inc.*
>> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
>> > <javascript:;>
>> > > > > <javascript:;>
>> > > > > > > > > > > > > > >>>>> o: 303.746.7302
>> > > > > > > > > > > > > > >>>>> Advancing the way the world uses the cloud
>> > > > > > > > > > > > > > >>>>> <
>> > > > > http://solidfire.com/solution/overview/?video=play
>> > > > > > > >*™*
>> > > > > > > > > > > > > > >>>>>
>> > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>> --
>> > > > > > > > > > > > > > >>> *Mike Tutkowski*
>> > > > > > > > > > > > > > >>> *Senior CloudStack Developer, SolidFire
>> Inc.*
>> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
>> <javascript:;>
>> > > > > <javascript:;>
>> > > > > > > > > > > > > > >>> o: 303.746.7302
>> > > > > > > > > > > > > > >>> Advancing the way the world uses the cloud
>> > > > > > > > > > > > > > >>> <
>> > > > http://solidfire.com/solution/overview/?video=play
>> > > > > > >*™*
>> > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > *Mike Tutkowski*
>> > > > > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
>> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com
>> <javascript:;>
>> > > > > <javascript:;>
>> > > > > > > > > > > > > > > o: 303.746.7302
>> > > > > > > > > > > > > > > Advancing the way the world uses the cloud
>> > > > > > > > > > > > > > > <
>> > > http://solidfire.com/solution/overview/?video=play
>> > > > > >*™*
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Find out more about ShapeBlue and our range of
>> > > > CloudStack
>> > > > > > > > > > > > > > related
>> > > > > > > > > > > > > services:
>> > > > > > > > > > > > > > IaaS Cloud Design & Build
>> > > > > > > > > > > > > > <
>> > http://shapeblue.com/iaas-cloud-design-and-build//>
>> > > |
>> > > > > > > > CSForge –
>> > > > > > > > > > > > > > rapid IaaS deployment framework <
>> > > > > > > http://shapeblue.com/csforge/
>> > > > > > > > >
>> > > > > > > > > > > > > > CloudStack Consulting
>> > > > > > > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/>
>> |
>> > > > > > > > > > > > > CloudStack
>> > > > > > > > > > > > > > Software Engineering
>> > > > > > > > > > > > > > <
>> > > http://shapeblue.com/cloudstack-software-engineering/
>> > > > >
>> > > > > > > > > > > > > > CloudStack Infrastructure Support
>> > > > > > > > > > > > > > <
>> > > > http://shapeblue.com/cloudstack-infrastructure-support/
>> > > > > >
>> > > > > > |
>> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
>> > > > > > > > > > > > > > <http://shapeblue.com/cloudstack-training/>
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > --
>> > > > > > > > > > > > *Mike Tutkowski*
>> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
>> > > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
>> > > > > <javascript:;>
>> > > > > > > > > > > > o: 303.746.7302
>> > > > > > > > > > > > Advancing the way the world uses the cloud
>> > > > > > > > > > > > <http://solidfire.com/solution/overview/?video=play
>> > >*™*
>> > > > > > > > > > > >
>> > > > > > > > > > > Find out more about ShapeBlue and our range of
>> CloudStack
>> > > > > related
>> > > > > > > > > > services:
>> > > > > > > > > > > IaaS Cloud Design & Build
>> > > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//>
>> |
>> > > > > CSForge –
>> > > > > > > > rapid
>> > > > > > > > > > > IaaS deployment framework <
>> http://shapeblue.com/csforge/
>> > >
>> > > > > > > CloudStack
>> > > > > > > > > > > Consulting <
>> http://shapeblue.com/cloudstack-consultancy/
>> > >
>> > > |
>> > > > > > > > CloudStack
>> > > > > > > > > > > Software Engineering
>> > > > > > > > > > > <
>> http://shapeblue.com/cloudstack-software-engineering/>
>> > > > > > > > > > > CloudStack Infrastructure Support
>> > > > > > > > > > > <
>> http://shapeblue.com/cloudstack-infrastructure-support/
>> > >
>> > > |
>> > > > > > > > CloudStack
>> > > > > > > > > > > Bootcamp Training Courses <
>> > > > > > > http://shapeblue.com/cloudstack-training/
>> > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > Find out more about ShapeBlue and our range of
>> CloudStack
>> > > > related
>> > > > > > > > > services:
>> > > > > > > > > > IaaS Cloud Design & Build
>> > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
>> > > > CSForge –
>> > > > > > > rapid
>> > > > > > > > > > IaaS deployment framework <
>> http://shapeblue.com/csforge/>
>> > > > > > > > > > CloudStack Consulting <
>> > > > > > http://shapeblue.com/cloudstack-consultancy/>
>> > > > > > > |
>> > > > > > > > > CloudStack
>> > > > > > > > > > Software Engineering
>> > > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
>> > > > > > > > > > CloudStack Infrastructure Support
>> > > > > > > > > > <
>> http://shapeblue.com/cloudstack-infrastructure-support/>
>> > |
>> > > > > > > CloudStack
>> > > > > > > > > > Bootcamp Training Courses <
>> > > > > > http://shapeblue.com/cloudstack-training/
>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > *Mike Tutkowski*
>> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
>> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
>> > > > > > > o: 303.746.7302
>> > > > > > > Advancing the way the world uses the cloud
>> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > *Mike Tutkowski*
>> > > > > *Senior CloudStack Developer, SolidFire Inc.*
>> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
>> > > > > o: 303.746.7302
>> > > > > Advancing the way the world uses the cloud
>> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Ian Rae
>> > > > CEO | PDG
>> > > > c: 514.944.4008
>> > > >
>> > > > CloudOps | Cloud Infrastructure and Networking Solutions
>> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > *Mike Tutkowski*
>> > *Senior CloudStack Developer, SolidFire Inc.*
>> > e: mike.tutkowski@solidfire.com
>> > o: 303.746.7302
>> > Advancing the way the world uses the cloud
>> > <http://solidfire.com/solution/overview/?video=play>*™*
>> >
>>
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

It's not ideal - true, but it does allow us to be backward compatible.

If you have other ideas, though, about how to maintain backward
compatibility, I'm definitely open to hear them.

Thanks!

On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <sy...@gmail.com>
wrote:

> Hi Mike,
>
> Adding a flag to createSnapshot was the first and the most obvious thing
> that came to our minds. The problem that I had with this was that:
>
> 1) I feel it is exposing something to the end user that is internal to the
> cloud.
>
> 2) We have to follow two different ways of restore/deletion in the same
> code path depending on where the Snapshot resides which I find kind of a
> bad design.
>
> But if exposing a archive flag to the end user is acceptable then we can
> definitely use this instead of adding the StorageSnapshot API
>
> Thanks,
> -Syed
>
>
> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com
> > wrote:
>
> > Hi Pierre-Luc,
> >
> > My recommendation would be this:
> >
> > Add an "archive" flag to the current volume-snapshot API. Its default
> would
> > be "false" because that would be backward compatible with how 4.6 has
> > volume snapshots implemented (i.e. they stay on the SAN in 4.6, 4.7, and
> > 4.8).
> >
> > If you set archive=true, then we would perform a background migration of
> > the snapshot from the SAN to the secondary storage (then delete the SAN
> > snapshot).
> >
> > That archive parameter would only be valid for managed storage.
> >
> > Sound reasonable?
> >
> > Also, a VM snapshot that includes disks provided by managed storage
> should
> > work.
> >
> > Talk to you later,
> > Mike
> >
> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <pd...@cloudops.com>
> > wrote:
> >
> > > Mike,
> > >
> > > In terms of API's, would you prefer introducing a parameter to the
> > existing
> > > VolumeSnapshot, example:   extract={true|false}  with a default value
> of
> > > true  which would extract snapshot into the secondary storage, which is
> > the
> > > current default behavior. Then for SAN snapshot that remain on the SAN
> we
> > > would just set "extract=false" ?  as oppose to create a new
> > >  StorageSnapshot API ?
> > >
> > >
> > > Paul,
> > >
> > > From what I'm seeing so far, we can't do a VM-snapshot when using
> managed
> > > storage for VM having more than one Volume. For the reason that
> snapshot
> > > are performed outside of the hypervisor awareness and asynchronously.
> If
> > > someone have a way to address that, it would make thinks much more
> > > attractive.
> > >
> > >
> > >
> > >
> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com> wrote:
> > >
> > > > I think a service provider backup scenario is more likely to take
> > > advantage
> > > > of SAN snapshot. There are a few reasons for this. Traditional
> backups
> > > > involve access to the file system, and there is an expectation that
> > this
> > > > can be done with reasonably short time frames without negatively
> > > impacting
> > > > VM performance, and that the backup orchestrator can apply various
> > logic
> > > > and or transformations to the data (compress, encrypt, deltas
> etc...).
> > > > While it is true that one could apply a backup process to a cloud
> > > snapshot,
> > > > this would be slow and inefficient requiring the data to be moved
> > several
> > > > times and there are some major bottlenecks with cloud snapshots.
> With a
> > > > cloud snapshot - there seems to be no reasonable expectation of being
> > > able
> > > > to do differential snapshots (I think this depends on the hypervisor)
> > and
> > > > if you do differential snapshots this will make file backups from
> those
> > > > snapshots even more complicated to orchestrate.
> > > >
> > > > Suspect there needs to be a different thread on how to better enable
> > > > backups, as a service. As per Paul's suggestion, but it is a related
> > > > workflow so relevant to this discussion.
> > > >
> > > > Ian
> > > >
> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com>
> > > > wrote:
> > > >
> > > > > To me it sounds like number two and number three are different uses
> > for
> > > > the
> > > > > same "thing"(which is totally fine).
> > > > >
> > > > > As for taking a fast SAN snapshot and exporting it asynchronously,
> do
> > > we
> > > > > see the SSVM as performing the export?
> > > > >
> > > > > To be backwards compatible with what we have in 4.6 and later for
> > > volume
> > > > > snapshots for managed storage, I think it might be easier if we
> pass
> > > in a
> > > > > flag that says whether or not to archive the SAN snapshot (which, I
> > > > think,
> > > > > is something that you suggested, as well, Pierre-Luc).
> > > > >
> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <pdion@cloudops.com
> > > > > <javascript:;>> wrote:
> > > > >
> > > > > > Hi Mike,
> > > > > >
> > > > > > The reason behind the creation of a SAN snapshot which is
> exported
> > > into
> > > > > > secondary storage, is because creating a copy of the .VHD
> directly
> > > > would
> > > > > > impact uptime of the VM as creating that copy take lots of time.
> > Has
> > > > > oppose
> > > > > > to a SAN snapshot that is close to instantaneous which can
> > afterward
> > > be
> > > > > > clone into Secondary Storage asynchronously.
> > > > > >
> > > > > > I would suspect an extracted VolumeSnapshot taken from a SAN
> > snapshot
> > > > > could
> > > > > > have is SAN snapshot deleted to avoid duplica and space
> consumption
> > > on
> > > > > the
> > > > > > Primary Storage side.
> > > > > >
> > > > > >
> > > > > > I see 3 definitions in our current discussion regarding the term
> > > > snapshot
> > > > > > (these are not official terminology but by own interpretation of
> > > them):
> > > > > >
> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a
> > > > snapshot):
> > > > > > it's a volume snapshot at the storage level, point in time of
> your
> > > > data.
> > > > > it
> > > > > > reside on the primary storage. Useful and efficient for software
> > side
> > > > > > incident.
> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud
> backup
> > > > aws-S3
> > > > > > style ): Point in time copy of the Virtual Disk that reside on a
> > > > > different
> > > > > > storage array then the original Volume. Facilitate data migration
> > > > between
> > > > > > clusters and, in case of primary storage incident, Volume
> snapshots
> > > are
> > > > > not
> > > > > > impacted and can be reuse.
> > > > > > 3. *Backup*: Archival of your Virtual-machines data that also
> > > validate
> > > > > data
> > > > > > integrity, provide a storage efficient archiving method and an
> > > > > independent
> > > > > > way to restore your data in case of an major infrastructure
> > disaster.
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > PL
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > > > > > wrote:
> > > > > >
> > > > > > > So, let's see if I currently follow the requirements:
> > > > > > >
> > > > > > > * Augment volume snapshots for managed storage to conditionally
> > > > export
> > > > > > data
> > > > > > > to NFS. The current process of taking a snapshot on the SAN is
> > > fine,
> > > > > but
> > > > > > > we'd like the option to export the data to NFS, as well.
> > > > > > >
> > > > > > > Questions:
> > > > > > >
> > > > > > > Once the data has been exported to NFS, do we keep the SAN
> > snapshot
> > > > or
> > > > > > > delete it?
> > > > > > >
> > > > > > > If we are deleting the SAN snapshot, then why don't we just
> copy
> > > the
> > > > > VHD
> > > > > > > from primary to secondary the way we do today for non-managed
> > (i.e.
> > > > > > > traditional) storage? Why create a SAN snapshot in that
> scenario?
> > > > > Perhaps
> > > > > > > to have the SSVM mount and perform the VHD copy to secondary
> > > storage
> > > > > > > instead of a XenServer host?
> > > > > > >
> > > > > > > Thanks for the clarification.
> > > > > > >
> > > > > > > By the way, to me a backup is when you copy data from one
> storage
> > > > > system
> > > > > > to
> > > > > > > another (regardless of features, if any, to restore the data in
> > the
> > > > > > > future). A snapshot is a point-in-time view of the data of a
> > volume
> > > > and
> > > > > > > it's stored on the same storage system as the volume.
> > > > > > >
> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > > pdion@cloudops.com
> > > > > <javascript:;>
> > > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > That's fun to see that discussion happening. I 100% agree
> with
> > > > Paul's
> > > > > > > > points of view. VolumeSnapshot are not a backup, but I do
> > > consider
> > > > > them
> > > > > > > as
> > > > > > > > a safety vest against Primary Storage failure, because
> failure
> > > > append
> > > > > > > :-( .
> > > > > > > >
> > > > > > > > The current proposal around snapshots that reside on the
> > primary
> > > > > > storage
> > > > > > > or
> > > > > > > > snapshots that end in the Secondary Storage  is not to
> address
> > > any
> > > > > kind
> > > > > > > of
> > > > > > > > backups requirement because a snapshot is not a backup, event
> > an
> > > > > > > extracted
> > > > > > > > VM snapshot.
> > > > > > > >
> > > > > > > > The main idea, and again this is for managed storage;
> > > > > > > >
> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> capability
> > > for
> > > > > > fast
> > > > > > > > response time that support rollback to previous timestamp,
> > create
> > > > new
> > > > > > > > volume and maybe create template.
> > > > > > > >     not required to be a new API if the work is already
> done, I
> > > > think
> > > > > > > this
> > > > > > > > is a different behaviors than the user expectation of a
> > > > > > volume-snapshot.
> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior
> that
> > > > create
> > > > > > an
> > > > > > > > extraction of a volume into SecondaryStorage which can be
> reuse
> > > to
> > > > > > > create a
> > > > > > > > new volume into another Primary Storage. This type of
> snapshot
> > > is a
> > > > > > slow
> > > > > > > > job since yes it would have to copy the full volume size on
> the
> > > > > > Secondary
> > > > > > > > storage.
> > > > > > > >
> > > > > > > >
> > > > > > > > PL
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > > <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I think I share you view on the 'Ideal world'. Backup (via
> > > Volume
> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is
> > > amplified
> > > > > > > > especially
> > > > > > > > > when you have a object storage as your secondary storage
> > > because
> > > > it
> > > > > > > > > requires two copies (one to an NFS staging area and from
> > there
> > > to
> > > > > > > object
> > > > > > > > > storage). And not to mention that all these copies are
> > > consuming
> > > > > > > > hypervisor
> > > > > > > > > resources. Xenserver's Dom0 is also a huge bottleneck as
> all
> > > the
> > > > > > > Network
> > > > > > > > > and I/O flow through it. So our intention of proposing the
> > > > "Storage
> > > > > > > > > Snapshots" is to give a better way of achiving snapshots
> > while
> > > > > still
> > > > > > > > > keeping the original definition of volume snpashots (ie
> > upload
> > > to
> > > > > sec
> > > > > > > > > storage).
> > > > > > > > >
> > > > > > > > > But as Erik pointed out volume snapshots are not backups.
> > They
> > > > > don't
> > > > > > > work
> > > > > > > > > form multi-disk LVM volume groups and dynamic disks. I am
> all
> > > in
> > > > > for
> > > > > > a
> > > > > > > > > better backup solution which handles these use cases and
> > > utilizes
> > > > > the
> > > > > > > > > storage's advanced features.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > In the beginning... there were CloudStack snapshots and
> > they
> > > > were
> > > > > > > > > actually
> > > > > > > > > > volume snapshots not hypervisor point-in-time snapshots.
> > > > > > > > > > Then VM snapshots were created (which are point-in-time
> > > > > hypervisor
> > > > > > > > > > snapshots) and we started referring to the original
> > snapshots
> > > > as
> > > > > > > volume
> > > > > > > > > > snapshots.
> > > > > > > > > >
> > > > > > > > > > CloudStack does not offer 'backups', but many people use
> > > volume
> > > > > > > > snapshots
> > > > > > > > > > as backups. However you can't in-place restore volume
> > > snapshots
> > > > > and
> > > > > > > if
> > > > > > > > > you
> > > > > > > > > > have a VM with multiple volumes, the volume snapshots
> must
> > be
> > > > > done
> > > > > > in
> > > > > > > > > > series, meaning that the state across all of the volumes
> is
> > > > > > unlikely
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > consistent.
> > > > > > > > > >
> > > > > > > > > > 'Actual Backups' would enable all of the restore options
> > > which
> > > > > > users
> > > > > > > > > might
> > > > > > > > > > expect as well options as to where they might be stored.
> In
> > > my
> > > > > > ideal
> > > > > > > > > world
> > > > > > > > > > they would also be able to leverage back-end hardware
> (such
> > > as
> > > > > > > > Solidfire,
> > > > > > > > > > NetApp etc :) ) and software such as Veeam, Commvault etc
> > to
> > > > > > > accelerate
> > > > > > > > > the
> > > > > > > > > > process.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > > > > > > Paul Angus
> > > > > > > > > > VP Technology ,  ShapeBlue
> > > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > > >
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> > > > > m:
> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> > <javascript:;>
> > > |
> > > > > t: @cloudyangus*
> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > > <ja...@cloudyangus>
> > > > > > |  w:
> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > > > > > > Shape Blue Ltd is a company incorporated in England &
> > Wales.
> > > > > > > ShapeBlue
> > > > > > > > > > Services India LLP is a company incorporated in India and
> > is
> > > > > > operated
> > > > > > > > > under
> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> Consultoria
> > > Ltda
> > > > > is
> > > > > > a
> > > > > > > > > > company incorporated in Brasil and is operated under
> > license
> > > > from
> > > > > > > Shape
> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by
> > The
> > > > > > > Republic
> > > > > > > > of
> > > > > > > > > > South Africa and is traded under license from Shape Blue
> > Ltd.
> > > > > > > ShapeBlue
> > > > > > > > > is
> > > > > > > > > > a registered trademark.
> > > > > > > > > > This email and any attachments to it may be confidential
> > and
> > > > are
> > > > > > > > intended
> > > > > > > > > > solely for the use of the individual to whom it is
> > addressed.
> > > > Any
> > > > > > > views
> > > > > > > > > or
> > > > > > > > > > opinions expressed are solely those of the author and do
> > not
> > > > > > > > necessarily
> > > > > > > > > > represent those of Shape Blue Ltd or related companies.
> If
> > > you
> > > > > are
> > > > > > > not
> > > > > > > > > the
> > > > > > > > > > intended recipient of this email, you must neither take
> any
> > > > > action
> > > > > > > > based
> > > > > > > > > > upon its contents, nor copy or show it to anyone. Please
> > > > contact
> > > > > > the
> > > > > > > > > sender
> > > > > > > > > > if you believe you have received this email in error.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > > > > <javascript:;> <javascript:;>]
> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > <javascript:;>
> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > > >
> > > > > > > > > > Paul,
> > > > > > > > > >
> > > > > > > > > > When you say actual backups, how would it be different
> from
> > > the
> > > > > > > Volume
> > > > > > > > > > Snapshots that exist currently. My understanding is that
> > > > Backups
> > > > > > end
> > > > > > > up
> > > > > > > > > in
> > > > > > > > > > Sec Storage whereas Snapshots are just a point-in-time
> > state
> > > of
> > > > > > your
> > > > > > > > > volume
> > > > > > > > > > which can be restored back correct?
> > > > > > > > > >
> > > > > > > > > > -Syed
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Syed,
> > > > > > > > > > >
> > > > > > > > > > > As I understand it, the SolidFire plugin will export
> the
> > > > > snapshot
> > > > > > > to
> > > > > > > > > > > secondary storage if the user requests a template from
> > the
> > > > > > snapshot
> > > > > > > > or
> > > > > > > > > > > wants to download the snapshot from the cloud. This is
> a
> > > > good,
> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire storage
> is
> > > > super
> > > > > > > > > > > reliable and snapshots on SolidFire arrays take up next
> > to
> > > no
> > > > > > > space.
> > > > > > > > > > > BUT I think that we are talking about a more general
> > > purpose
> > > > > API,
> > > > > > > and
> > > > > > > > > > > other storage systems may not be as awesome as Mike's.
> > > That's
> > > > > my
> > > > > > > > > > > concern. Also, the time to transfer for say 1TB to move
> > > from
> > > > > > > primary
> > > > > > > > > > > to sec storage and then create a VM template out of it
> > may
> > > be
> > > > > too
> > > > > > > > long
> > > > > > > > > > for users.
> > > > > > > > > > >
> > > > > > > > > > > @Mike I don’t think 'we' use the term volume snapshot
> for
> > > > > backup,
> > > > > > > > it's
> > > > > > > > > > > just that users want to do backups and a volume
> snapshot
> > is
> > > > the
> > > > > > > only
> > > > > > > > > > > type of snapshot that copies the disk elsewhere and can
> > be
> > > > > > > scheduled.
> > > > > > > > > > >
> > > > > > > > > > > I'm 'pondering' the implications of enabling actual
> > backups
> > > > > > > (through
> > > > > > > > > > > recognised backup providers) and the user requirements
> > > around
> > > > > > them
> > > > > > > > > > > (particularly restoration use cases) as a separate
> thread
> > > of
> > > > > > work.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul
> Angus
> > > VP
> > > > > > > > Technology
> > > > > > > > > > > , ShapeBlue
> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > > > >
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > |
> > > > > m:
> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > <javascript:;> |
> > > > > t: @cloudyangus*
> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> <javascript:;>
> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> > Shape
> > > > > Blue
> > > > > > > Ltd
> > > > > > > > > > > is a company incorporated in England & Wales. ShapeBlue
> > > > > Services
> > > > > > > > India
> > > > > > > > > > > LLP is a company incorporated in India and is operated
> > > under
> > > > > > > license
> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> > is
> > > a
> > > > > > > company
> > > > > > > > > > > incorporated in Brasil and is operated under license
> from
> > > > Shape
> > > > > > > Blue
> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by
> The
> > > > > Republic
> > > > > > > of
> > > > > > > > > > > South Africa and is traded under license from Shape
> Blue
> > > Ltd.
> > > > > > > > > > > ShapeBlue is a registered trademark.
> > > > > > > > > > > This email and any attachments to it may be
> confidential
> > > and
> > > > > are
> > > > > > > > > > > intended solely for the use of the individual to whom
> it
> > is
> > > > > > > > addressed.
> > > > > > > > > > > Any views or opinions expressed are solely those of the
> > > > author
> > > > > > and
> > > > > > > do
> > > > > > > > > > > not necessarily represent those of Shape Blue Ltd or
> > > related
> > > > > > > > > > > companies. If you are not the intended recipient of
> this
> > > > email,
> > > > > > you
> > > > > > > > > > > must neither take any action based upon its contents,
> nor
> > > > copy
> > > > > or
> > > > > > > > show
> > > > > > > > > > > it to anyone. Please contact the sender if you believe
> > you
> > > > have
> > > > > > > > > received
> > > > > > > > > > this email in error.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > > > > <javascript:;>
> > > > > > <javascript:;>]
> > > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > <javascript:;>
> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > > > >
> > > > > > > > > > > I think the terminology confusion comes from AWS where
> > they
> > > > do
> > > > > > EBS
> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
> followed
> > > > that.
> > > > > > And
> > > > > > > > as
> > > > > > > > > > > an end user who is oblivious to the internals of my
> > > provider,
> > > > > my
> > > > > > > > > > > expectation would be something similar to what AWS as
> > that
> > > is
> > > > > my
> > > > > > > > > > > biggest reference point.
> > > > > > > > > > >
> > > > > > > > > > > To your point Mike, I agree that a Primary Storage
> > failure
> > > on
> > > > > > > > > > > SolidFire is unlikely, there are other motivations for
> us
> > > to
> > > > > push
> > > > > > > > data
> > > > > > > > > > > to secondary storage. Primary storage (atleast for us)
> > > costs
> > > > > > > around 3
> > > > > > > > > > > times as much as secondary storage and snapshots on
> > primary
> > > > > > storage
> > > > > > > > > > > are rarely used (especially for some of our customers
> who
> > > do
> > > > > > daily
> > > > > > > > > > backups).
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>>
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Some of the weirdness is around terminology.
> > > > > > > > > > > >
> > > > > > > > > > > > For most systems I've worked on, a snapshot and a
> > backup
> > > > are
> > > > > > two
> > > > > > > > > > > > completely different things (but CloudStack has
> > > > traditionally
> > > > > > > used
> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> > > > > > > > > > > >
> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
> though,
> > > that
> > > > > if
> > > > > > > your
> > > > > > > > > > > > primary storage is running on SolidFire that it is
> > > unlikely
> > > > > > > you'll
> > > > > > > > > > > > encounter an issue where your primary storage goes
> > > offline
> > > > > (and
> > > > > > > > > > > > you'll even maintain your performance guarantees
> during
> > > > > failure
> > > > > > > > > > > > scenarios and upgrades, as well). That being the
> case,
> > it
> > > > is
> > > > > > less
> > > > > > > > > > > > useful to require a backup to Swift (but it's
> perfectly
> > > OK
> > > > if
> > > > > > > > that's
> > > > > > > > > > > > what we want to do
> > > > > > > > > > > here).
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> > <javascript:;>>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Paul,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I believe with the current implementation of
> > Snapshots
> > > on
> > > > > > > managed
> > > > > > > > > > > > > storage
> > > > > > > > > > > > > (SolidFire) the snapshots are never exported to the
> > > > > secondary
> > > > > > > > > > storage.
> > > > > > > > > > > > > While this solves the problem of having snapshots
> > > taking
> > > > > > > forever
> > > > > > > > > > > > > to get to sec storage, this leaves us with a
> > > > > > > > > > > > huge
> > > > > > > > > > > > > liability if our primary storage goes down. We see
> > > > > snapshots
> > > > > > as
> > > > > > > > > > > > > our recovery path because we store them in Swift
> > which
> > > is
> > > > > > > > reliable
> > > > > > > > > > > > > and resilient to failures.
> > > > > > > > > > > > >
> > > > > > > > > > > > > With Storage snpashots our goal is to have Volume
> > > > snapshots
> > > > > > > > always
> > > > > > > > > > > > > backed up to secondary storage and Storage
> Snapshots
> > > stay
> > > > > on
> > > > > > > the
> > > > > > > > > > > > > primary
> > > > > > > > > > > > storage.
> > > > > > > > > > > > > A provider could potentially mix both these and
> solve
> > > the
> > > > > > > problem
> > > > > > > > > > > > > that you mentioned where you want to meet user's
> > > > > expectation
> > > > > > > of a
> > > > > > > > > > > > > snapshot (ie backup to sec storage) while having an
> > > > ability
> > > > > > to
> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the device)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hope this clarifies things.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > -Syed
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > <javascript:;>>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > HI guys,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Could someone point me to the Jira bug of FS for
> > the
> > > > > > > > > > > > > > SAN-snapshot
> > > > > > > > > > > > feature
> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > From my discussions with users and operators
> around
> > > > > > snapshots
> > > > > > > > > > > > > > I'd make
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > following observations:
> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
> long-term
> > > and
> > > > > > short
> > > > > > > > > > > > > > term)
> > > > > > > > > > > > with
> > > > > > > > > > > > > > the expectation that they can use them for
> recovery
> > > if
> > > > > > > > required.
> > > > > > > > > > > > > > b. operators fall back to snapshots if something
> > has
> > > > gone
> > > > > > > wrong
> > > > > > > > > > > > > > with primary storage.
> > > > > > > > > > > > > > c. users sometimes want to be able to export
> > > snapshots
> > > > as
> > > > > > > well
> > > > > > > > > > > > > > as
> > > > > > > > > > > > create
> > > > > > > > > > > > > > new VMs from their snapshots
> > > > > > > > > > > > > > d. snapshots are a currently a massive pain for
> > > > > operators,
> > > > > > I
> > > > > > > > > > > > > > know at
> > > > > > > > > > > > > least
> > > > > > > > > > > > > > one public cloud who have snapshots which take 2
> > days
> > > > to
> > > > > > > > > complete.
> > > > > > > > > > > > > > e. snapshots (as they are) can't be used for
> > multiple
> > > > LVM
> > > > > > > > disks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think the process Mike has used in the
> SolidFire
> > > > plugin
> > > > > > > (only
> > > > > > > > > > > > > > moving
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > disk image to secondary storage when you
> absolutely
> > > > have
> > > > > > to)
> > > > > > > is
> > > > > > > > > > > > > > a very
> > > > > > > > > > > > > good
> > > > > > > > > > > > > > and pragmatic solution. I wonder what problems an
> > > > > operator
> > > > > > > > might
> > > > > > > > > > > > > experience
> > > > > > > > > > > > > > if they have an issue with a given primary
> storage
> > > pool
> > > > > in
> > > > > > a
> > > > > > > > > > cluster.
> > > > > > > > > > > > (I
> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> SolidFire
> > > case
> > > > > > Mike
> > > > > > > :)
> > > > > > > > > > > > > > ) And
> > > > > > > > > > > > if
> > > > > > > > > > > > > > the transfer from primary to secondary is slow,
> the
> > > > time
> > > > > to
> > > > > > > > > > > > > > being able
> > > > > > > > > > > > to
> > > > > > > > > > > > > > create a template or export the volume will be
> > slow.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So for me the issue is around making sure that
> the
> > > end
> > > > > > users
> > > > > > > > > > > > expectations
> > > > > > > > > > > > > > are met (while improving the speed/efficiency of
> > the
> > > > back
> > > > > > > end)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> Paul
> > > > Angus
> > > > > > VP
> > > > > > > > > > > > > > Technology , ShapeBlue
> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > > > > > > >
> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > > |
> > > > > > > > m:
> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > > > <javascript:;> | t:
> > > > > > @cloudyangus*
> > > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > <javascript:;>
> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N
> 4HS
> > UK
> > > > > Shape
> > > > > > > > Blue
> > > > > > > > > > > > > > Ltd is a company incorporated in England & Wales.
> > > > > ShapeBlue
> > > > > > > > > > > > > > Services India LLP is a company incorporated in
> > India
> > > > and
> > > > > > is
> > > > > > > > > > > > > > operated
> > > > > > > > > > > > > under
> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > > > Consultoria
> > > > > > > Ltda
> > > > > > > > > > > > > > is a company incorporated in Brasil and is
> operated
> > > > under
> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd
> > is
> > > a
> > > > > > > company
> > > > > > > > > > > > > > registered by The Republic
> > > > > > > > > > > > of
> > > > > > > > > > > > > > South Africa and is traded under license from
> Shape
> > > > Blue
> > > > > > Ltd.
> > > > > > > > > > > > > > ShapeBlue
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > a registered trademark.
> > > > > > > > > > > > > > This email and any attachments to it may be
> > > > confidential
> > > > > > and
> > > > > > > > are
> > > > > > > > > > > > intended
> > > > > > > > > > > > > > solely for the use of the individual to whom it
> is
> > > > > > addressed.
> > > > > > > > > > > > > > Any views
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > opinions expressed are solely those of the author
> > and
> > > > do
> > > > > > not
> > > > > > > > > > > > necessarily
> > > > > > > > > > > > > > represent those of Shape Blue Ltd or related
> > > companies.
> > > > > If
> > > > > > > you
> > > > > > > > > > > > > > are not
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > intended recipient of this email, you must
> neither
> > > take
> > > > > any
> > > > > > > > > > > > > > action
> > > > > > > > > > > > based
> > > > > > > > > > > > > > upon its contents, nor copy or show it to anyone.
> > > > Please
> > > > > > > > contact
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > sender
> > > > > > > > > > > > > > if you believe you have received this email in
> > error.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com
> > > > > <javascript:;>
> > > > > > <javascript:;>]
> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > > <javascript:;>
> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > Snapshots
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Mike,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The idea of introducing a new API:
> StorageSnapshot
> > > for
> > > > > > > managed
> > > > > > > > > > > > > > storage
> > > > > > > > > > > > is
> > > > > > > > > > > > > > because the VolumeSnapshot default, or expected,
> > > > behavior
> > > > > > is
> > > > > > > to
> > > > > > > > > > > > > > archive snapshots into the Secondary Storage. So
> a
> > > > > > > > > > > > > > StorageSnapshot API would be
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > snapshot that remain on the managed storage
> > > appliance.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Quickly looking at the API doc and I don't see a
> > > strong
> > > > > > > > > > > > > > requirement for volume snapshots to be moved into
> > > > > secondary
> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is not
> > useful,
> > > > but
> > > > > > > both
> > > > > > > > > > > > > > use cases are required. A snapshot that remain on
> > the
> > > > > > managed
> > > > > > > > > > > > > > storage, and another type of
> > > > > > > > > > > > snapshot
> > > > > > > > > > > > > > that end up into the secondary storage. Since
> > you've
> > > > > done a
> > > > > > > lot
> > > > > > > > > > > > > > of
> > > > > > > > > > > > work,
> > > > > > > > > > > > > > might easier to just add a parameter to the
> current
> > > > > > snapshot
> > > > > > > > API
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > trigger an extraction of the storage snapshot
> into
> > > the
> > > > > > > > secondary
> > > > > > > > > > > > storage?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > PL
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > > <javascript:;>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think that all sounds reasonable then -
> thanks!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;>
> <javascript:;>>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> > requirements.
> > > > One
> > > > > > of
> > > > > > > > our
> > > > > > > > > > > > earlier
> > > > > > > > > > > > > > >> iterations on this was to have an argument to
> > the
> > > > > create
> > > > > > > > > > > > > > >> snapshot
> > > > > > > > > > > > API
> > > > > > > > > > > > > > >> which decides whether to backup the volume to
> > sec
> > > > > > storage
> > > > > > > > but
> > > > > > > > > > > > > > >> we realized it would make management of
> > snapshots
> > > > > quite
> > > > > > > > messy
> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com <javascript:;>
> > > > > <javascript:;>>
> > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>> Hi,
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Just to make sure I understand all the
> > > requirements
> > > > > > here:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> 1) This relates only to managed storage (1:1
> > > > mapping
> > > > > > > > between
> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> 2) We want to take the current (introduced in
> > > 4.6)
> > > > > > > > > > > > > > >>> functionality, which creates a snapshot on
> the
> > > SAN,
> > > > > and
> > > > > > > > > > > > > > >>> extend it via a config option (or
> > > > > > > > > > > > > > >>> something) to not only take the SAN snapshot,
> > but
> > > > to
> > > > > > copy
> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken. It's the
> > > > backup
> > > > > to
> > > > > > > NFS
> > > > > > > > > > > > > > >>> that is optional.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> 4) Templates can be created from the snapshot
> > > > that's
> > > > > on
> > > > > > > the
> > > > > > > > > > > > > > >>> SAN (already works).
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created from the
> > > > > snapshot
> > > > > > > > > > > > > > >>> that's on
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >>> SAN (already works as long as the new
> > CloudStack
> > > > > volume
> > > > > > > > ends
> > > > > > > > > > > > > > >>> up on the same primary storage).
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Would we have a need for a storage snapshot
> API
> > > > then
> > > > > or
> > > > > > > > > > > > > > >>> would that just be the standard volume
> snapshot
> > > > > without
> > > > > > > the
> > > > > > > > > > > > > > >>> backup to
> > > > > > > > > > > NFS?
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Thanks!
> > > > > > > > > > > > > > >>> Mike
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
> > > > > <javascript:;>>
> > > > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>> Is it possible to have both functionalities
> > > > > (snapshot
> > > > > > on
> > > > > > > > > > > > > > >>>> SAN & Sec
> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we would
> > like
> > > > to
> > > > > > have
> > > > > > > > > both.
> > > > > > > > > > > > > > >>>> For example, some of our customers want to
> > > > implement
> > > > > > > their
> > > > > > > > > > > > > > >>>> own backup strategies and do encryption to
> > their
> > > > > > backups
> > > > > > > > > > > > > > >>>> which is a perfect use case for Storage
> > Snapshot
> > > > > while
> > > > > > > our
> > > > > > > > > > > > > > >>>> other customers will still keep using the
> > > standard
> > > > > > > volume
> > > > > > > > > > > snapshot.
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> To keep things backward compatible, we can
> > add a
> > > > > > setting
> > > > > > > > > > > > > > >>>> which
> > > > > > > > > > > > says
> > > > > > > > > > > > > > >>>> to not upload on secondary storage, because,
> > > after
> > > > > > all,
> > > > > > > > you
> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when doing a
> > > > Volume
> > > > > > > > > Snapshot.
> > > > > > > > > > > > > > >>>> You could stop the process there and not do
> > the
> > > > > > upload.
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> What do you think about this approach?
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> Thanks,
> > > > > > > > > > > > > > >>>> -Syed
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> > Tutkowski <
> > > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com <javascript:;>
> > > > > <javascript:;>> wrote:
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>> So, this is just me thinking out load here,
> > but
> > > > if
> > > > > a
> > > > > > > > given
> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need to
> > > provide
> > > > > > both
> > > > > > > > the
> > > > > > > > > > > > ability
> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it to NFS
> > (if
> > > > > just
> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we might
> > be
> > > > able
> > > > > > to
> > > > > > > > get
> > > > > > > > > > > > > > >>>>> away with no new
> > > > > > > > > > > > API
> > > > > > > > > > > > > > >>>>> calls and simply implement a new custom
> > > snapshot
> > > > > > > strategy
> > > > > > > > > > > > > > >>>>> and
> > > > > > > > > > > > data
> > > > > > > > > > > > > > >>>>> motion strategy to handle the case where
> the
> > > > > > CloudStack
> > > > > > > > > > > > > > >>>>> cloud
> > > > > > > > > > > > does
> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> exported-to-NFS
> > > > > backup.
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> In other words, the "default" behavior
> would
> > be
> > > > to
> > > > > > use
> > > > > > > > the
> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion strategy
> > that
> > > > we
> > > > > > > > already
> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> > snapshot).
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however, wants to
> > > take
> > > > a
> > > > > > SAN
> > > > > > > > > > > > > > >>>>> snapshot and have the data exported to NFS,
> > > then
> > > > we
> > > > > > > could
> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config file to
> > make
> > > > use
> > > > > > of
> > > > > > > a
> > > > > > > > > > > > > > >>>>> new snapshot strategy and data motion
> > strategy
> > > > that
> > > > > > > > > > > > > > >>>>> performs both of these
> > > > > > > > > > > > > > activities.
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> This way, the old behavior is still the
> > default
> > > > for
> > > > > > > > users,
> > > > > > > > > > > > > > >>>>> but CloudStack admins can change this
> > behavior
> > > > via
> > > > > > > > > > > configuration.
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> Thoughts?
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike
> > > Tutkowski <
> > > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com
> <javascript:;>
> > > > > <javascript:;>> wrote:
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>>> Right...I think we will need to come up
> > with a
> > > > > > viable
> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way for
> them
> > > to
> > > > > move
> > > > > > > > from
> > > > > > > > > > > > > > >>>>>> the old way to the new way (and some
> obvious
> > > way
> > > > > > that
> > > > > > > > > > > > > > >>>>>> they will know they need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > do this).
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed
> > Mushtaq
> > > <
> > > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > > > > <javascript:;>> wrote:
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the upgrade
> path
> > > > > however,
> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and are on a
> > > > managed
> > > > > > > > storage
> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> functionality
> > > > with
> > > > > > > Volume
> > > > > > > > > > > Snapshots.
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed
> > Mushtaq
> > > <
> > > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > > > > <javascript:;>> wrote:
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>>> So if I understand correctly, currently
> > > > taking a
> > > > > > > > Volume
> > > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a managed
> storage
> > > > keeps
> > > > > > it
> > > > > > > on
> > > > > > > > > > > > > > >>>>>>>> the storage array. As a part of this
> > > feature,
> > > > we
> > > > > > can
> > > > > > > > > > > > > > >>>>>>>> make sure
> > > > > > > > > > > > that
> > > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed storage are
> > > > uploaded
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > > > >>>>>>>> secondary storage. This would make the
> > > Volume
> > > > > > > Snapshot
> > > > > > > > > > > > > > >>>>>>>> feature behave the same regardless of
> the
> > > > > storage
> > > > > > > > > > > > > > >>>>>>>> (managed or
> > > > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing the
> > > efficient
> > > > > > > backend
> > > > > > > > > > > > > > >>>>>>>> storage
> > > > > > > > > > > > > > capabilities, we can use the new storage
> snapshots
> > > API.
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike
> > > > Tutkowski <
> > > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> > <javascript:;>
> > > > > <javascript:;>>
> > > > > > wrote:
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need to have a
> > plan
> > > > to
> > > > > > deal
> > > > > > > > > > > > > > >>>>>>>>> with the fact that we already have a
> > > feature
> > > > > (in
> > > > > > > 4.6
> > > > > > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > > > > > >>>>>>>>> later) that allows you to use the
> > existing
> > > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a volume
> > > > > snapshot
> > > > > > > (for
> > > > > > > > > > > > > > >>>>>>>>> managed
> > > > > > > > > > > > > > >>>>>>>>> storage) that resides on a backend SAN
> > > > (using a
> > > > > > > > custom
> > > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom data
> > motion
> > > > > > > strategy).
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then how
> should
> > > the
> > > > > > > original
> > > > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6 and
> later)
> > > be
> > > > > > > changed?
> > > > > > > > > > > > > > >>>>>>>>> If it
> > > > > > > > > > > > is
> > > > > > > > > > > > > > >>>>>>>>> changed, how do we support customers
> who
> > > are
> > > > > > > already
> > > > > > > > > > > > > > >>>>>>>>> using
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API to take
> > > > snapshots
> > > > > > on a
> > > > > > > > > > > > > > >>>>>>>>> backend
> > > > > > > > > > > > > SAN?
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > > > > > > > >>>>>>>>> Mike
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will
> > > > Stevens <
> > > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com <javascript:;>
> > > > > <javascript:;>> wrote:
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>> Will you be able to create a Template
> > > from a
> > > > > > > > > > > > StorageSnapshot?
> > > > > > > > > > > > > > >>>>>>>>>> If yes, will the template be stored in
> > the
> > > > > > > secondary
> > > > > > > > > > > > > > >>>>>>>>>> storage like normal templates or will
> > that
> > > > be
> > > > > > > > handled
> > > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > > > > > > > > vendor side?
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec
> *|*
> > > H3J
> > > > > 1S6
> > > > > > w
> > > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed
> > > > Mushtaq <
> > > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com
> <javascript:;>
> > > > > <javascript:;>> wrote:
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will
> > > > Stevens
> > > > > <
> > > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com <javascript:;>
> > > > > <javascript:;>> wrote:
> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec
> in
> > > the
> > > > > > Jira
> > > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not clear in
> the
> > > > > > 'mention'
> > > > > > > > > > > > > > >>>>>>>>>>>> section because it shows as a page
> > 'you
> > > do
> > > > > not
> > > > > > > > have
> > > > > > > > > > > permission to'.
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions
> Experts
> > > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec
> > *|*
> > > > H3J
> > > > > > 1S6
> > > > > > > w
> > > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed
> > > Ahmed
> > > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com <javascript:;>
> > > > > <javascript:;>
> > > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of
> APIs
> > to
> > > > do
> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage
> backends
> > > > like
> > > > > > > > > SolidFire.
> > > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed
> storage
> > > stay
> > > > > on
> > > > > > > the
> > > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary to what
> > > > CloudStack
> > > > > > > calls
> > > > > > > > > > > snpshots.
> > > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage and
> > > > keeping
> > > > > > it
> > > > > > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > > > > > > > > > and
> > > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to have both
> > ways
> > > > of
> > > > > > > doing
> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new
> > > APIs
> > > > to
> > > > > > > > create
> > > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of this
> > > feature? I
> > > > > > would
> > > > > > > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am
> > > working
> > > > on
> > > > > > > > making
> > > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > > > > > > > spec
> > > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a
> > high
> > > > > level
> > > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before starting to
> > work
> > > on
> > > > > it.
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>> --
> > > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire
> > > Inc.*
> > > > > > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > > <javascript:;>
> > > > > <javascript:;>
> > > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > > > > > > > > >>>>>>>>> Advancing the way the world uses the
> > cloud
> > > > > > > > > > > > > > >>>>>>>>> <
> > > > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > > >*™
> > > > > > > > > > > > > > >>>>>>>>> *
> > > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>> --
> > > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire
> > Inc.*
> > > > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > <javascript:;>
> > > > > <javascript:;>
> > > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > > > > > > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > > > > >>>>>> <
> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > >*™*
> > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> --
> > > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire
> Inc.*
> > > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > <javascript:;>
> > > > > <javascript:;>
> > > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > > > > > > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > > > > >>>>> <
> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > >*™*
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> --
> > > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > > > > > > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com
> <javascript:;>
> > > > > <javascript:;>
> > > > > > > > > > > > > > >>> o: 303.746.7302
> > > > > > > > > > > > > > >>> Advancing the way the world uses the cloud
> > > > > > > > > > > > > > >>> <
> > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >*™*
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > <javascript:;>
> > > > > > > > > > > > > > > o: 303.746.7302
> > > > > > > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > > > > > > <
> > > http://solidfire.com/solution/overview/?video=play
> > > > > >*™*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > Find out more about ShapeBlue and our range of
> > > > CloudStack
> > > > > > > > > > > > > > related
> > > > > > > > > > > > > services:
> > > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > > > > > > <
> > http://shapeblue.com/iaas-cloud-design-and-build//>
> > > |
> > > > > > > > CSForge –
> > > > > > > > > > > > > > rapid IaaS deployment framework <
> > > > > > > http://shapeblue.com/csforge/
> > > > > > > > >
> > > > > > > > > > > > > > CloudStack Consulting
> > > > > > > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > > > > > > > > > CloudStack
> > > > > > > > > > > > > > Software Engineering
> > > > > > > > > > > > > > <
> > > http://shapeblue.com/cloudstack-software-engineering/
> > > > >
> > > > > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > > > > > > <
> > > > http://shapeblue.com/cloudstack-infrastructure-support/
> > > > > >
> > > > > > |
> > > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > <javascript:;>
> > > > > > > > > > > > o: 303.746.7302
> > > > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > > > <http://solidfire.com/solution/overview/?video=play
> > >*™*
> > > > > > > > > > > >
> > > > > > > > > > > Find out more about ShapeBlue and our range of
> CloudStack
> > > > > related
> > > > > > > > > > services:
> > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > > > CSForge –
> > > > > > > > rapid
> > > > > > > > > > > IaaS deployment framework <
> http://shapeblue.com/csforge/
> > >
> > > > > > > CloudStack
> > > > > > > > > > > Consulting <
> http://shapeblue.com/cloudstack-consultancy/
> > >
> > > |
> > > > > > > > CloudStack
> > > > > > > > > > > Software Engineering
> > > > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/
> >
> > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > > > <
> http://shapeblue.com/cloudstack-infrastructure-support/
> > >
> > > |
> > > > > > > > CloudStack
> > > > > > > > > > > Bootcamp Training Courses <
> > > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > related
> > > > > > > > > services:
> > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > > CSForge –
> > > > > > > rapid
> > > > > > > > > > IaaS deployment framework <http://shapeblue.com/csforge/
> >
> > > > > > > > > > CloudStack Consulting <
> > > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > > > > |
> > > > > > > > > CloudStack
> > > > > > > > > > Software Engineering
> > > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/
> >
> > |
> > > > > > > CloudStack
> > > > > > > > > > Bootcamp Training Courses <
> > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Mike Tutkowski*
> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > > > > > o: 303.746.7302
> > > > > > > Advancing the way the world uses the cloud
> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > >
> > > >
> > > > --
> > > > Ian Rae
> > > > CEO | PDG
> > > > c: 514.944.4008
> > > >
> > > > CloudOps | Cloud Infrastructure and Networking Solutions
> > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
> > > >
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

Hi Mike,

Adding a flag to createSnapshot was the first and the most obvious thing
that came to our minds. The problem that I had with this was that:

1) I feel it is exposing something to the end user that is internal to the
cloud.

2) We have to follow two different ways of restore/deletion in the same
code path depending on where the Snapshot resides which I find kind of a
bad design.

But if exposing a archive flag to the end user is acceptable then we can
definitely use this instead of adding the StorageSnapshot API

Thanks,
-Syed


On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> Hi Pierre-Luc,
>
> My recommendation would be this:
>
> Add an "archive" flag to the current volume-snapshot API. Its default would
> be "false" because that would be backward compatible with how 4.6 has
> volume snapshots implemented (i.e. they stay on the SAN in 4.6, 4.7, and
> 4.8).
>
> If you set archive=true, then we would perform a background migration of
> the snapshot from the SAN to the secondary storage (then delete the SAN
> snapshot).
>
> That archive parameter would only be valid for managed storage.
>
> Sound reasonable?
>
> Also, a VM snapshot that includes disks provided by managed storage should
> work.
>
> Talk to you later,
> Mike
>
> On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <pd...@cloudops.com>
> wrote:
>
> > Mike,
> >
> > In terms of API's, would you prefer introducing a parameter to the
> existing
> > VolumeSnapshot, example:   extract={true|false}  with a default value of
> > true  which would extract snapshot into the secondary storage, which is
> the
> > current default behavior. Then for SAN snapshot that remain on the SAN we
> > would just set "extract=false" ?  as oppose to create a new
> >  StorageSnapshot API ?
> >
> >
> > Paul,
> >
> > From what I'm seeing so far, we can't do a VM-snapshot when using managed
> > storage for VM having more than one Volume. For the reason that snapshot
> > are performed outside of the hypervisor awareness and asynchronously. If
> > someone have a way to address that, it would make thinks much more
> > attractive.
> >
> >
> >
> >
> > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com> wrote:
> >
> > > I think a service provider backup scenario is more likely to take
> > advantage
> > > of SAN snapshot. There are a few reasons for this. Traditional backups
> > > involve access to the file system, and there is an expectation that
> this
> > > can be done with reasonably short time frames without negatively
> > impacting
> > > VM performance, and that the backup orchestrator can apply various
> logic
> > > and or transformations to the data (compress, encrypt, deltas etc...).
> > > While it is true that one could apply a backup process to a cloud
> > snapshot,
> > > this would be slow and inefficient requiring the data to be moved
> several
> > > times and there are some major bottlenecks with cloud snapshots. With a
> > > cloud snapshot - there seems to be no reasonable expectation of being
> > able
> > > to do differential snapshots (I think this depends on the hypervisor)
> and
> > > if you do differential snapshots this will make file backups from those
> > > snapshots even more complicated to orchestrate.
> > >
> > > Suspect there needs to be a different thread on how to better enable
> > > backups, as a service. As per Paul's suggestion, but it is a related
> > > workflow so relevant to this discussion.
> > >
> > > Ian
> > >
> > > On Monday, February 8, 2016, Mike Tutkowski <
> > mike.tutkowski@solidfire.com>
> > > wrote:
> > >
> > > > To me it sounds like number two and number three are different uses
> for
> > > the
> > > > same "thing"(which is totally fine).
> > > >
> > > > As for taking a fast SAN snapshot and exporting it asynchronously, do
> > we
> > > > see the SSVM as performing the export?
> > > >
> > > > To be backwards compatible with what we have in 4.6 and later for
> > volume
> > > > snapshots for managed storage, I think it might be easier if we pass
> > in a
> > > > flag that says whether or not to archive the SAN snapshot (which, I
> > > think,
> > > > is something that you suggested, as well, Pierre-Luc).
> > > >
> > > > On Monday, February 8, 2016, Pierre-Luc Dion <pdion@cloudops.com
> > > > <javascript:;>> wrote:
> > > >
> > > > > Hi Mike,
> > > > >
> > > > > The reason behind the creation of a SAN snapshot which is exported
> > into
> > > > > secondary storage, is because creating a copy of the .VHD directly
> > > would
> > > > > impact uptime of the VM as creating that copy take lots of time.
> Has
> > > > oppose
> > > > > to a SAN snapshot that is close to instantaneous which can
> afterward
> > be
> > > > > clone into Secondary Storage asynchronously.
> > > > >
> > > > > I would suspect an extracted VolumeSnapshot taken from a SAN
> snapshot
> > > > could
> > > > > have is SAN snapshot deleted to avoid duplica and space consumption
> > on
> > > > the
> > > > > Primary Storage side.
> > > > >
> > > > >
> > > > > I see 3 definitions in our current discussion regarding the term
> > > snapshot
> > > > > (these are not official terminology but by own interpretation of
> > them):
> > > > >
> > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a
> > > snapshot):
> > > > > it's a volume snapshot at the storage level, point in time of your
> > > data.
> > > > it
> > > > > reside on the primary storage. Useful and efficient for software
> side
> > > > > incident.
> > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud backup
> > > aws-S3
> > > > > style ): Point in time copy of the Virtual Disk that reside on a
> > > > different
> > > > > storage array then the original Volume. Facilitate data migration
> > > between
> > > > > clusters and, in case of primary storage incident, Volume snapshots
> > are
> > > > not
> > > > > impacted and can be reuse.
> > > > > 3. *Backup*: Archival of your Virtual-machines data that also
> > validate
> > > > data
> > > > > integrity, provide a storage efficient archiving method and an
> > > > independent
> > > > > way to restore your data in case of an major infrastructure
> disaster.
> > > > >
> > > > >
> > > > > Regards,
> > > > >
> > > > > PL
> > > > >
> > > > >
> > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > > > > wrote:
> > > > >
> > > > > > So, let's see if I currently follow the requirements:
> > > > > >
> > > > > > * Augment volume snapshots for managed storage to conditionally
> > > export
> > > > > data
> > > > > > to NFS. The current process of taking a snapshot on the SAN is
> > fine,
> > > > but
> > > > > > we'd like the option to export the data to NFS, as well.
> > > > > >
> > > > > > Questions:
> > > > > >
> > > > > > Once the data has been exported to NFS, do we keep the SAN
> snapshot
> > > or
> > > > > > delete it?
> > > > > >
> > > > > > If we are deleting the SAN snapshot, then why don't we just copy
> > the
> > > > VHD
> > > > > > from primary to secondary the way we do today for non-managed
> (i.e.
> > > > > > traditional) storage? Why create a SAN snapshot in that scenario?
> > > > Perhaps
> > > > > > to have the SSVM mount and perform the VHD copy to secondary
> > storage
> > > > > > instead of a XenServer host?
> > > > > >
> > > > > > Thanks for the clarification.
> > > > > >
> > > > > > By the way, to me a backup is when you copy data from one storage
> > > > system
> > > > > to
> > > > > > another (regardless of features, if any, to restore the data in
> the
> > > > > > future). A snapshot is a point-in-time view of the data of a
> volume
> > > and
> > > > > > it's stored on the same storage system as the volume.
> > > > > >
> > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > pdion@cloudops.com
> > > > <javascript:;>
> > > > > <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > That's fun to see that discussion happening. I 100% agree with
> > > Paul's
> > > > > > > points of view. VolumeSnapshot are not a backup, but I do
> > consider
> > > > them
> > > > > > as
> > > > > > > a safety vest against Primary Storage failure, because failure
> > > append
> > > > > > :-( .
> > > > > > >
> > > > > > > The current proposal around snapshots that reside on the
> primary
> > > > > storage
> > > > > > or
> > > > > > > snapshots that end in the Secondary Storage  is not to address
> > any
> > > > kind
> > > > > > of
> > > > > > > backups requirement because a snapshot is not a backup, event
> an
> > > > > > extracted
> > > > > > > VM snapshot.
> > > > > > >
> > > > > > > The main idea, and again this is for managed storage;
> > > > > > >
> > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot capability
> > for
> > > > > fast
> > > > > > > response time that support rollback to previous timestamp,
> create
> > > new
> > > > > > > volume and maybe create template.
> > > > > > >     not required to be a new API if the work is already done, I
> > > think
> > > > > > this
> > > > > > > is a different behaviors than the user expectation of a
> > > > > volume-snapshot.
> > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior that
> > > create
> > > > > an
> > > > > > > extraction of a volume into SecondaryStorage which can be reuse
> > to
> > > > > > create a
> > > > > > > new volume into another Primary Storage. This type of snapshot
> > is a
> > > > > slow
> > > > > > > job since yes it would have to copy the full volume size on the
> > > > > Secondary
> > > > > > > storage.
> > > > > > >
> > > > > > >
> > > > > > > PL
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > > syed1.mushtaq@gmail.com <javascript:;>
> > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I think I share you view on the 'Ideal world'. Backup (via
> > Volume
> > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is
> > amplified
> > > > > > > especially
> > > > > > > > when you have a object storage as your secondary storage
> > because
> > > it
> > > > > > > > requires two copies (one to an NFS staging area and from
> there
> > to
> > > > > > object
> > > > > > > > storage). And not to mention that all these copies are
> > consuming
> > > > > > > hypervisor
> > > > > > > > resources. Xenserver's Dom0 is also a huge bottleneck as all
> > the
> > > > > > Network
> > > > > > > > and I/O flow through it. So our intention of proposing the
> > > "Storage
> > > > > > > > Snapshots" is to give a better way of achiving snapshots
> while
> > > > still
> > > > > > > > keeping the original definition of volume snpashots (ie
> upload
> > to
> > > > sec
> > > > > > > > storage).
> > > > > > > >
> > > > > > > > But as Erik pointed out volume snapshots are not backups.
> They
> > > > don't
> > > > > > work
> > > > > > > > form multi-disk LVM volume groups and dynamic disks. I am all
> > in
> > > > for
> > > > > a
> > > > > > > > better backup solution which handles these use cases and
> > utilizes
> > > > the
> > > > > > > > storage's advanced features.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > In the beginning... there were CloudStack snapshots and
> they
> > > were
> > > > > > > > actually
> > > > > > > > > volume snapshots not hypervisor point-in-time snapshots.
> > > > > > > > > Then VM snapshots were created (which are point-in-time
> > > > hypervisor
> > > > > > > > > snapshots) and we started referring to the original
> snapshots
> > > as
> > > > > > volume
> > > > > > > > > snapshots.
> > > > > > > > >
> > > > > > > > > CloudStack does not offer 'backups', but many people use
> > volume
> > > > > > > snapshots
> > > > > > > > > as backups. However you can't in-place restore volume
> > snapshots
> > > > and
> > > > > > if
> > > > > > > > you
> > > > > > > > > have a VM with multiple volumes, the volume snapshots must
> be
> > > > done
> > > > > in
> > > > > > > > > series, meaning that the state across all of the volumes is
> > > > > unlikely
> > > > > > to
> > > > > > > > be
> > > > > > > > > consistent.
> > > > > > > > >
> > > > > > > > > 'Actual Backups' would enable all of the restore options
> > which
> > > > > users
> > > > > > > > might
> > > > > > > > > expect as well options as to where they might be stored. In
> > my
> > > > > ideal
> > > > > > > > world
> > > > > > > > > they would also be able to leverage back-end hardware (such
> > as
> > > > > > > Solidfire,
> > > > > > > > > NetApp etc :) ) and software such as Veeam, Commvault etc
> to
> > > > > > accelerate
> > > > > > > > the
> > > > > > > > > process.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > > > > > Paul Angus
> > > > > > > > > VP Technology ,  ShapeBlue
> > > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > >
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> > > > m:
> > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;>
> <javascript:;>
> > |
> > > > t: @cloudyangus*
> > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > > <ja...@cloudyangus>
> > > > > |  w:
> > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > > > > > Shape Blue Ltd is a company incorporated in England &
> Wales.
> > > > > > ShapeBlue
> > > > > > > > > Services India LLP is a company incorporated in India and
> is
> > > > > operated
> > > > > > > > under
> > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria
> > Ltda
> > > > is
> > > > > a
> > > > > > > > > company incorporated in Brasil and is operated under
> license
> > > from
> > > > > > Shape
> > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by
> The
> > > > > > Republic
> > > > > > > of
> > > > > > > > > South Africa and is traded under license from Shape Blue
> Ltd.
> > > > > > ShapeBlue
> > > > > > > > is
> > > > > > > > > a registered trademark.
> > > > > > > > > This email and any attachments to it may be confidential
> and
> > > are
> > > > > > > intended
> > > > > > > > > solely for the use of the individual to whom it is
> addressed.
> > > Any
> > > > > > views
> > > > > > > > or
> > > > > > > > > opinions expressed are solely those of the author and do
> not
> > > > > > > necessarily
> > > > > > > > > represent those of Shape Blue Ltd or related companies. If
> > you
> > > > are
> > > > > > not
> > > > > > > > the
> > > > > > > > > intended recipient of this email, you must neither take any
> > > > action
> > > > > > > based
> > > > > > > > > upon its contents, nor copy or show it to anyone. Please
> > > contact
> > > > > the
> > > > > > > > sender
> > > > > > > > > if you believe you have received this email in error.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > > > <javascript:;> <javascript:;>]
> > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> <javascript:;>
> > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > >
> > > > > > > > > Paul,
> > > > > > > > >
> > > > > > > > > When you say actual backups, how would it be different from
> > the
> > > > > > Volume
> > > > > > > > > Snapshots that exist currently. My understanding is that
> > > Backups
> > > > > end
> > > > > > up
> > > > > > > > in
> > > > > > > > > Sec Storage whereas Snapshots are just a point-in-time
> state
> > of
> > > > > your
> > > > > > > > volume
> > > > > > > > > which can be restored back correct?
> > > > > > > > >
> > > > > > > > > -Syed
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Syed,
> > > > > > > > > >
> > > > > > > > > > As I understand it, the SolidFire plugin will export the
> > > > snapshot
> > > > > > to
> > > > > > > > > > secondary storage if the user requests a template from
> the
> > > > > snapshot
> > > > > > > or
> > > > > > > > > > wants to download the snapshot from the cloud. This is a
> > > good,
> > > > > > > > > > pragmatic approach and yes Mike the SolidFire storage is
> > > super
> > > > > > > > > > reliable and snapshots on SolidFire arrays take up next
> to
> > no
> > > > > > space.
> > > > > > > > > > BUT I think that we are talking about a more general
> > purpose
> > > > API,
> > > > > > and
> > > > > > > > > > other storage systems may not be as awesome as Mike's.
> > That's
> > > > my
> > > > > > > > > > concern. Also, the time to transfer for say 1TB to move
> > from
> > > > > > primary
> > > > > > > > > > to sec storage and then create a VM template out of it
> may
> > be
> > > > too
> > > > > > > long
> > > > > > > > > for users.
> > > > > > > > > >
> > > > > > > > > > @Mike I don’t think 'we' use the term volume snapshot for
> > > > backup,
> > > > > > > it's
> > > > > > > > > > just that users want to do backups and a volume snapshot
> is
> > > the
> > > > > > only
> > > > > > > > > > type of snapshot that copies the disk elsewhere and can
> be
> > > > > > scheduled.
> > > > > > > > > >
> > > > > > > > > > I'm 'pondering' the implications of enabling actual
> backups
> > > > > > (through
> > > > > > > > > > recognised backup providers) and the user requirements
> > around
> > > > > them
> > > > > > > > > > (particularly restoration use cases) as a separate thread
> > of
> > > > > work.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus
> > VP
> > > > > > > Technology
> > > > > > > > > > , ShapeBlue
> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > > >
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > |
> > > > m:
> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > <javascript:;> |
> > > > t: @cloudyangus*
> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;> <javascript:;>
> > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> Shape
> > > > Blue
> > > > > > Ltd
> > > > > > > > > > is a company incorporated in England & Wales. ShapeBlue
> > > > Services
> > > > > > > India
> > > > > > > > > > LLP is a company incorporated in India and is operated
> > under
> > > > > > license
> > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> is
> > a
> > > > > > company
> > > > > > > > > > incorporated in Brasil and is operated under license from
> > > Shape
> > > > > > Blue
> > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > > > Republic
> > > > > > of
> > > > > > > > > > South Africa and is traded under license from Shape Blue
> > Ltd.
> > > > > > > > > > ShapeBlue is a registered trademark.
> > > > > > > > > > This email and any attachments to it may be confidential
> > and
> > > > are
> > > > > > > > > > intended solely for the use of the individual to whom it
> is
> > > > > > > addressed.
> > > > > > > > > > Any views or opinions expressed are solely those of the
> > > author
> > > > > and
> > > > > > do
> > > > > > > > > > not necessarily represent those of Shape Blue Ltd or
> > related
> > > > > > > > > > companies. If you are not the intended recipient of this
> > > email,
> > > > > you
> > > > > > > > > > must neither take any action based upon its contents, nor
> > > copy
> > > > or
> > > > > > > show
> > > > > > > > > > it to anyone. Please contact the sender if you believe
> you
> > > have
> > > > > > > > received
> > > > > > > > > this email in error.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > > > <javascript:;>
> > > > > <javascript:;>]
> > > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > <javascript:;>
> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > > >
> > > > > > > > > > I think the terminology confusion comes from AWS where
> they
> > > do
> > > > > EBS
> > > > > > > > > > snapshots backed up to S3 and CloudStack sort of followed
> > > that.
> > > > > And
> > > > > > > as
> > > > > > > > > > an end user who is oblivious to the internals of my
> > provider,
> > > > my
> > > > > > > > > > expectation would be something similar to what AWS as
> that
> > is
> > > > my
> > > > > > > > > > biggest reference point.
> > > > > > > > > >
> > > > > > > > > > To your point Mike, I agree that a Primary Storage
> failure
> > on
> > > > > > > > > > SolidFire is unlikely, there are other motivations for us
> > to
> > > > push
> > > > > > > data
> > > > > > > > > > to secondary storage. Primary storage (atleast for us)
> > costs
> > > > > > around 3
> > > > > > > > > > times as much as secondary storage and snapshots on
> primary
> > > > > storage
> > > > > > > > > > are rarely used (especially for some of our customers who
> > do
> > > > > daily
> > > > > > > > > backups).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>>
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Some of the weirdness is around terminology.
> > > > > > > > > > >
> > > > > > > > > > > For most systems I've worked on, a snapshot and a
> backup
> > > are
> > > > > two
> > > > > > > > > > > completely different things (but CloudStack has
> > > traditionally
> > > > > > used
> > > > > > > > > > > the term "volume snapshot" to mean backup).
> > > > > > > > > > >
> > > > > > > > > > > I will put in a SolidFire "plug" here and say, though,
> > that
> > > > if
> > > > > > your
> > > > > > > > > > > primary storage is running on SolidFire that it is
> > unlikely
> > > > > > you'll
> > > > > > > > > > > encounter an issue where your primary storage goes
> > offline
> > > > (and
> > > > > > > > > > > you'll even maintain your performance guarantees during
> > > > failure
> > > > > > > > > > > scenarios and upgrades, as well). That being the case,
> it
> > > is
> > > > > less
> > > > > > > > > > > useful to require a backup to Swift (but it's perfectly
> > OK
> > > if
> > > > > > > that's
> > > > > > > > > > > what we want to do
> > > > > > > > > > here).
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;>
> <javascript:;>>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Paul,
> > > > > > > > > > > >
> > > > > > > > > > > > I believe with the current implementation of
> Snapshots
> > on
> > > > > > managed
> > > > > > > > > > > > storage
> > > > > > > > > > > > (SolidFire) the snapshots are never exported to the
> > > > secondary
> > > > > > > > > storage.
> > > > > > > > > > > > While this solves the problem of having snapshots
> > taking
> > > > > > forever
> > > > > > > > > > > > to get to sec storage, this leaves us with a
> > > > > > > > > > > huge
> > > > > > > > > > > > liability if our primary storage goes down. We see
> > > > snapshots
> > > > > as
> > > > > > > > > > > > our recovery path because we store them in Swift
> which
> > is
> > > > > > > reliable
> > > > > > > > > > > > and resilient to failures.
> > > > > > > > > > > >
> > > > > > > > > > > > With Storage snpashots our goal is to have Volume
> > > snapshots
> > > > > > > always
> > > > > > > > > > > > backed up to secondary storage and Storage Snapshots
> > stay
> > > > on
> > > > > > the
> > > > > > > > > > > > primary
> > > > > > > > > > > storage.
> > > > > > > > > > > > A provider could potentially mix both these and solve
> > the
> > > > > > problem
> > > > > > > > > > > > that you mentioned where you want to meet user's
> > > > expectation
> > > > > > of a
> > > > > > > > > > > > snapshot (ie backup to sec storage) while having an
> > > ability
> > > > > to
> > > > > > > > > > > > utilize faster sanpshots (i.e. on the device)
> > > > > > > > > > > >
> > > > > > > > > > > > Hope this clarifies things.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > -Syed
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > <javascript:;>>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > HI guys,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Could someone point me to the Jira bug of FS for
> the
> > > > > > > > > > > > > SAN-snapshot
> > > > > > > > > > > feature
> > > > > > > > > > > > > in 4.6 which is mentioned.
> > > > > > > > > > > > >
> > > > > > > > > > > > > From my discussions with users and operators around
> > > > > snapshots
> > > > > > > > > > > > > I'd make
> > > > > > > > > > > > the
> > > > > > > > > > > > > following observations:
> > > > > > > > > > > > > a. 'users' use snapshots as backups (both long-term
> > and
> > > > > short
> > > > > > > > > > > > > term)
> > > > > > > > > > > with
> > > > > > > > > > > > > the expectation that they can use them for recovery
> > if
> > > > > > > required.
> > > > > > > > > > > > > b. operators fall back to snapshots if something
> has
> > > gone
> > > > > > wrong
> > > > > > > > > > > > > with primary storage.
> > > > > > > > > > > > > c. users sometimes want to be able to export
> > snapshots
> > > as
> > > > > > well
> > > > > > > > > > > > > as
> > > > > > > > > > > create
> > > > > > > > > > > > > new VMs from their snapshots
> > > > > > > > > > > > > d. snapshots are a currently a massive pain for
> > > > operators,
> > > > > I
> > > > > > > > > > > > > know at
> > > > > > > > > > > > least
> > > > > > > > > > > > > one public cloud who have snapshots which take 2
> days
> > > to
> > > > > > > > complete.
> > > > > > > > > > > > > e. snapshots (as they are) can't be used for
> multiple
> > > LVM
> > > > > > > disks.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think the process Mike has used in the SolidFire
> > > plugin
> > > > > > (only
> > > > > > > > > > > > > moving
> > > > > > > > > > > > the
> > > > > > > > > > > > > disk image to secondary storage when you absolutely
> > > have
> > > > > to)
> > > > > > is
> > > > > > > > > > > > > a very
> > > > > > > > > > > > good
> > > > > > > > > > > > > and pragmatic solution. I wonder what problems an
> > > > operator
> > > > > > > might
> > > > > > > > > > > > experience
> > > > > > > > > > > > > if they have an issue with a given primary storage
> > pool
> > > > in
> > > > > a
> > > > > > > > > cluster.
> > > > > > > > > > > (I
> > > > > > > > > > > > > know that that is REALLY unlikely in the SolidFire
> > case
> > > > > Mike
> > > > > > :)
> > > > > > > > > > > > > ) And
> > > > > > > > > > > if
> > > > > > > > > > > > > the transfer from primary to secondary is slow, the
> > > time
> > > > to
> > > > > > > > > > > > > being able
> > > > > > > > > > > to
> > > > > > > > > > > > > create a template or export the volume will be
> slow.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So for me the issue is around making sure that the
> > end
> > > > > users
> > > > > > > > > > > expectations
> > > > > > > > > > > > > are met (while improving the speed/efficiency of
> the
> > > back
> > > > > > end)
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul
> > > Angus
> > > > > VP
> > > > > > > > > > > > > Technology , ShapeBlue
> > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > > > > > >
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > > |
> > > > > > > m:
> > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > > <javascript:;> | t:
> > > > > @cloudyangus*
> > > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > <javascript:;>
> > > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS
> UK
> > > > Shape
> > > > > > > Blue
> > > > > > > > > > > > > Ltd is a company incorporated in England & Wales.
> > > > ShapeBlue
> > > > > > > > > > > > > Services India LLP is a company incorporated in
> India
> > > and
> > > > > is
> > > > > > > > > > > > > operated
> > > > > > > > > > > > under
> > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > > Consultoria
> > > > > > Ltda
> > > > > > > > > > > > > is a company incorporated in Brasil and is operated
> > > under
> > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd
> is
> > a
> > > > > > company
> > > > > > > > > > > > > registered by The Republic
> > > > > > > > > > > of
> > > > > > > > > > > > > South Africa and is traded under license from Shape
> > > Blue
> > > > > Ltd.
> > > > > > > > > > > > > ShapeBlue
> > > > > > > > > > > > is
> > > > > > > > > > > > > a registered trademark.
> > > > > > > > > > > > > This email and any attachments to it may be
> > > confidential
> > > > > and
> > > > > > > are
> > > > > > > > > > > intended
> > > > > > > > > > > > > solely for the use of the individual to whom it is
> > > > > addressed.
> > > > > > > > > > > > > Any views
> > > > > > > > > > > > or
> > > > > > > > > > > > > opinions expressed are solely those of the author
> and
> > > do
> > > > > not
> > > > > > > > > > > necessarily
> > > > > > > > > > > > > represent those of Shape Blue Ltd or related
> > companies.
> > > > If
> > > > > > you
> > > > > > > > > > > > > are not
> > > > > > > > > > > > the
> > > > > > > > > > > > > intended recipient of this email, you must neither
> > take
> > > > any
> > > > > > > > > > > > > action
> > > > > > > > > > > based
> > > > > > > > > > > > > upon its contents, nor copy or show it to anyone.
> > > Please
> > > > > > > contact
> > > > > > > > > > > > > the
> > > > > > > > > > > > sender
> > > > > > > > > > > > > if you believe you have received this email in
> error.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com
> > > > <javascript:;>
> > > > > <javascript:;>]
> > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > <javascript:;>
> > > > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> Snapshots
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Mike,
> > > > > > > > > > > > >
> > > > > > > > > > > > > The idea of introducing a new API: StorageSnapshot
> > for
> > > > > > managed
> > > > > > > > > > > > > storage
> > > > > > > > > > > is
> > > > > > > > > > > > > because the VolumeSnapshot default, or expected,
> > > behavior
> > > > > is
> > > > > > to
> > > > > > > > > > > > > archive snapshots into the Secondary Storage. So a
> > > > > > > > > > > > > StorageSnapshot API would be
> > > > > > > > > > > > for
> > > > > > > > > > > > > snapshot that remain on the managed storage
> > appliance.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Quickly looking at the API doc and I don't see a
> > strong
> > > > > > > > > > > > > requirement for volume snapshots to be moved into
> > > > secondary
> > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is not
> useful,
> > > but
> > > > > > both
> > > > > > > > > > > > > use cases are required. A snapshot that remain on
> the
> > > > > managed
> > > > > > > > > > > > > storage, and another type of
> > > > > > > > > > > snapshot
> > > > > > > > > > > > > that end up into the secondary storage. Since
> you've
> > > > done a
> > > > > > lot
> > > > > > > > > > > > > of
> > > > > > > > > > > work,
> > > > > > > > > > > > > might easier to just add a parameter to the current
> > > > > snapshot
> > > > > > > API
> > > > > > > > > > > > > that
> > > > > > > > > > > > would
> > > > > > > > > > > > > trigger an extraction of the storage snapshot into
> > the
> > > > > > > secondary
> > > > > > > > > > > storage?
> > > > > > > > > > > > >
> > > > > > > > > > > > > PL
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I think that all sounds reasonable then - thanks!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;> <javascript:;>>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> You are correct Mike in terms of the
> requirements.
> > > One
> > > > > of
> > > > > > > our
> > > > > > > > > > > earlier
> > > > > > > > > > > > > >> iterations on this was to have an argument to
> the
> > > > create
> > > > > > > > > > > > > >> snapshot
> > > > > > > > > > > API
> > > > > > > > > > > > > >> which decides whether to backup the volume to
> sec
> > > > > storage
> > > > > > > but
> > > > > > > > > > > > > >> we realized it would make management of
> snapshots
> > > > quite
> > > > > > > messy
> > > > > > > > > > > > > >> so we proposed a new api instead.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > > > > > > > > >> <mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>>
> > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>> Hi,
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Just to make sure I understand all the
> > requirements
> > > > > here:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> 1) This relates only to managed storage (1:1
> > > mapping
> > > > > > > between
> > > > > > > > > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> 2) We want to take the current (introduced in
> > 4.6)
> > > > > > > > > > > > > >>> functionality, which creates a snapshot on the
> > SAN,
> > > > and
> > > > > > > > > > > > > >>> extend it via a config option (or
> > > > > > > > > > > > > >>> something) to not only take the SAN snapshot,
> but
> > > to
> > > > > copy
> > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken. It's the
> > > backup
> > > > to
> > > > > > NFS
> > > > > > > > > > > > > >>> that is optional.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> 4) Templates can be created from the snapshot
> > > that's
> > > > on
> > > > > > the
> > > > > > > > > > > > > >>> SAN (already works).
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> 5) CloudStack volumes can be created from the
> > > > snapshot
> > > > > > > > > > > > > >>> that's on
> > > > > > > > > > > the
> > > > > > > > > > > > > >>> SAN (already works as long as the new
> CloudStack
> > > > volume
> > > > > > > ends
> > > > > > > > > > > > > >>> up on the same primary storage).
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Would we have a need for a storage snapshot API
> > > then
> > > > or
> > > > > > > > > > > > > >>> would that just be the standard volume snapshot
> > > > without
> > > > > > the
> > > > > > > > > > > > > >>> backup to
> > > > > > > > > > NFS?
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Thanks!
> > > > > > > > > > > > > >>> Mike
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
> > > > <javascript:;>>
> > > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> Is it possible to have both functionalities
> > > > (snapshot
> > > > > on
> > > > > > > > > > > > > >>>> SAN & Sec
> > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we would
> like
> > > to
> > > > > have
> > > > > > > > both.
> > > > > > > > > > > > > >>>> For example, some of our customers want to
> > > implement
> > > > > > their
> > > > > > > > > > > > > >>>> own backup strategies and do encryption to
> their
> > > > > backups
> > > > > > > > > > > > > >>>> which is a perfect use case for Storage
> Snapshot
> > > > while
> > > > > > our
> > > > > > > > > > > > > >>>> other customers will still keep using the
> > standard
> > > > > > volume
> > > > > > > > > > snapshot.
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> To keep things backward compatible, we can
> add a
> > > > > setting
> > > > > > > > > > > > > >>>> which
> > > > > > > > > > > says
> > > > > > > > > > > > > >>>> to not upload on secondary storage, because,
> > after
> > > > > all,
> > > > > > > you
> > > > > > > > > > > > > >>>> would take a SAN snapshot first when doing a
> > > Volume
> > > > > > > > Snapshot.
> > > > > > > > > > > > > >>>> You could stop the process there and not do
> the
> > > > > upload.
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> What do you think about this approach?
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> Thanks,
> > > > > > > > > > > > > >>>> -Syed
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> Tutkowski <
> > > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>> wrote:
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> So, this is just me thinking out load here,
> but
> > > if
> > > > a
> > > > > > > given
> > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need to
> > provide
> > > > > both
> > > > > > > the
> > > > > > > > > > > ability
> > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it to NFS
> (if
> > > > just
> > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we might
> be
> > > able
> > > > > to
> > > > > > > get
> > > > > > > > > > > > > >>>>> away with no new
> > > > > > > > > > > API
> > > > > > > > > > > > > >>>>> calls and simply implement a new custom
> > snapshot
> > > > > > strategy
> > > > > > > > > > > > > >>>>> and
> > > > > > > > > > > data
> > > > > > > > > > > > > >>>>> motion strategy to handle the case where the
> > > > > CloudStack
> > > > > > > > > > > > > >>>>> cloud
> > > > > > > > > > > does
> > > > > > > > > > > > > >>>>> want both a SAN snapshot and exported-to-NFS
> > > > backup.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> In other words, the "default" behavior would
> be
> > > to
> > > > > use
> > > > > > > the
> > > > > > > > > > > > > >>>>> snapshot strategy and data motion strategy
> that
> > > we
> > > > > > > already
> > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> snapshot).
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> If your CloudStack cloud, however, wants to
> > take
> > > a
> > > > > SAN
> > > > > > > > > > > > > >>>>> snapshot and have the data exported to NFS,
> > then
> > > we
> > > > > > could
> > > > > > > > > > > > > >>>>> have you manipulate a Swing config file to
> make
> > > use
> > > > > of
> > > > > > a
> > > > > > > > > > > > > >>>>> new snapshot strategy and data motion
> strategy
> > > that
> > > > > > > > > > > > > >>>>> performs both of these
> > > > > > > > > > > > > activities.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> This way, the old behavior is still the
> default
> > > for
> > > > > > > users,
> > > > > > > > > > > > > >>>>> but CloudStack admins can change this
> behavior
> > > via
> > > > > > > > > > configuration.
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> Thoughts?
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike
> > Tutkowski <
> > > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>> wrote:
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>> Right...I think we will need to come up
> with a
> > > > > viable
> > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way for them
> > to
> > > > move
> > > > > > > from
> > > > > > > > > > > > > >>>>>> the old way to the new way (and some obvious
> > way
> > > > > that
> > > > > > > > > > > > > >>>>>> they will know they need
> > > > > > > > > > > to
> > > > > > > > > > > > > do this).
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed
> Mushtaq
> > <
> > > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > > > <javascript:;>> wrote:
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>>> I'm not really sure about the upgrade path
> > > > however,
> > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and are on a
> > > managed
> > > > > > > storage
> > > > > > > > > > > > > >>>>>>> would no longer have the same functionality
> > > with
> > > > > > Volume
> > > > > > > > > > Snapshots.
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed
> Mushtaq
> > <
> > > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > > > <javascript:;>> wrote:
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>>> So if I understand correctly, currently
> > > taking a
> > > > > > > Volume
> > > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a managed storage
> > > keeps
> > > > > it
> > > > > > on
> > > > > > > > > > > > > >>>>>>>> the storage array. As a part of this
> > feature,
> > > we
> > > > > can
> > > > > > > > > > > > > >>>>>>>> make sure
> > > > > > > > > > > that
> > > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed storage are
> > > uploaded
> > > > > to
> > > > > > > the
> > > > > > > > > > > > > >>>>>>>> secondary storage. This would make the
> > Volume
> > > > > > Snapshot
> > > > > > > > > > > > > >>>>>>>> feature behave the same regardless of the
> > > > storage
> > > > > > > > > > > > > >>>>>>>> (managed or
> > > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing the
> > efficient
> > > > > > backend
> > > > > > > > > > > > > >>>>>>>> storage
> > > > > > > > > > > > > capabilities, we can use the new storage snapshots
> > API.
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike
> > > Tutkowski <
> > > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com
> <javascript:;>
> > > > <javascript:;>>
> > > > > wrote:
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need to have a
> plan
> > > to
> > > > > deal
> > > > > > > > > > > > > >>>>>>>>> with the fact that we already have a
> > feature
> > > > (in
> > > > > > 4.6
> > > > > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > > > > >>>>>>>>> later) that allows you to use the
> existing
> > > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a volume
> > > > snapshot
> > > > > > (for
> > > > > > > > > > > > > >>>>>>>>> managed
> > > > > > > > > > > > > >>>>>>>>> storage) that resides on a backend SAN
> > > (using a
> > > > > > > custom
> > > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom data
> motion
> > > > > > strategy).
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then how should
> > the
> > > > > > original
> > > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6 and later)
> > be
> > > > > > changed?
> > > > > > > > > > > > > >>>>>>>>> If it
> > > > > > > > > > > is
> > > > > > > > > > > > > >>>>>>>>> changed, how do we support customers who
> > are
> > > > > > already
> > > > > > > > > > > > > >>>>>>>>> using
> > > > > > > > > > > the
> > > > > > > > > > > > > >>>>>>>>> original volume-snapshot API to take
> > > snapshots
> > > > > on a
> > > > > > > > > > > > > >>>>>>>>> backend
> > > > > > > > > > > > SAN?
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > > > > > > >>>>>>>>> Mike
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will
> > > Stevens <
> > > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com <javascript:;>
> > > > <javascript:;>> wrote:
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>> Will you be able to create a Template
> > from a
> > > > > > > > > > > StorageSnapshot?
> > > > > > > > > > > > > >>>>>>>>>> If yes, will the template be stored in
> the
> > > > > > secondary
> > > > > > > > > > > > > >>>>>>>>>> storage like normal templates or will
> that
> > > be
> > > > > > > handled
> > > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > > > > > > > vendor side?
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|*
> > H3J
> > > > 1S6
> > > > > w
> > > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed
> > > Mushtaq <
> > > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > > > <javascript:;>> wrote:
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will
> > > Stevens
> > > > <
> > > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com <javascript:;>
> > > > <javascript:;>> wrote:
> > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in
> > the
> > > > > Jira
> > > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not clear in the
> > > > > 'mention'
> > > > > > > > > > > > > >>>>>>>>>>>> section because it shows as a page
> 'you
> > do
> > > > not
> > > > > > > have
> > > > > > > > > > permission to'.
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec
> *|*
> > > H3J
> > > > > 1S6
> > > > > > w
> > > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed
> > Ahmed
> > > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com <javascript:;>
> > > > <javascript:;>
> > > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs
> to
> > > do
> > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage backends
> > > like
> > > > > > > > SolidFire.
> > > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed storage
> > stay
> > > > on
> > > > > > the
> > > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary to what
> > > CloudStack
> > > > > > calls
> > > > > > > > > > snpshots.
> > > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage and
> > > keeping
> > > > > it
> > > > > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > > > > > > > > and
> > > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to have both
> ways
> > > of
> > > > > > doing
> > > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new
> > APIs
> > > to
> > > > > > > create
> > > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of this
> > feature? I
> > > > > would
> > > > > > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am
> > working
> > > on
> > > > > > > making
> > > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > > > > > > spec
> > > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a
> high
> > > > level
> > > > > > > > > > > > > >>>>>>>>>>>>> feedback first before starting to
> work
> > on
> > > > it.
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>> --
> > > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire
> > Inc.*
> > > > > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > <javascript:;>
> > > > <javascript:;>
> > > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > > > > > > > >>>>>>>>> Advancing the way the world uses the
> cloud
> > > > > > > > > > > > > >>>>>>>>> <
> > > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > > >*™
> > > > > > > > > > > > > >>>>>>>>> *
> > > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>> --
> > > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire
> Inc.*
> > > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > <javascript:;>
> > > > <javascript:;>
> > > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > > > > > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > > > >>>>>> <
> > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >*™*
> > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> --
> > > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> <javascript:;>
> > > > <javascript:;>
> > > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > > > > > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > > > >>>>> <
> > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >*™*
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> --
> > > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > > > > > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>
> > > > > > > > > > > > > >>> o: 303.746.7302
> > > > > > > > > > > > > >>> Advancing the way the world uses the cloud
> > > > > > > > > > > > > >>> <
> > > http://solidfire.com/solution/overview/?video=play
> > > > > >*™*
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>
> > > > > > > > > > > > > > o: 303.746.7302
> > > > > > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > > > > > <
> > http://solidfire.com/solution/overview/?video=play
> > > > >*™*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > Find out more about ShapeBlue and our range of
> > > CloudStack
> > > > > > > > > > > > > related
> > > > > > > > > > > > services:
> > > > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > > > > > <
> http://shapeblue.com/iaas-cloud-design-and-build//>
> > |
> > > > > > > CSForge –
> > > > > > > > > > > > > rapid IaaS deployment framework <
> > > > > > http://shapeblue.com/csforge/
> > > > > > > >
> > > > > > > > > > > > > CloudStack Consulting
> > > > > > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > > > > > > > > CloudStack
> > > > > > > > > > > > > Software Engineering
> > > > > > > > > > > > > <
> > http://shapeblue.com/cloudstack-software-engineering/
> > > >
> > > > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > > > > > <
> > > http://shapeblue.com/cloudstack-infrastructure-support/
> > > > >
> > > > > |
> > > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > <javascript:;>
> > > > > > > > > > > o: 303.746.7302
> > > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > > <http://solidfire.com/solution/overview/?video=play
> >*™*
> > > > > > > > > > >
> > > > > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > related
> > > > > > > > > services:
> > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > > CSForge –
> > > > > > > rapid
> > > > > > > > > > IaaS deployment framework <http://shapeblue.com/csforge/
> >
> > > > > > CloudStack
> > > > > > > > > > Consulting <http://shapeblue.com/cloudstack-consultancy/
> >
> > |
> > > > > > > CloudStack
> > > > > > > > > > Software Engineering
> > > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/
> >
> > |
> > > > > > > CloudStack
> > > > > > > > > > Bootcamp Training Courses <
> > > > > > http://shapeblue.com/cloudstack-training/
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > related
> > > > > > > > services:
> > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > CSForge –
> > > > > > rapid
> > > > > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > > > > > CloudStack Consulting <
> > > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > > > |
> > > > > > > > CloudStack
> > > > > > > > > Software Engineering
> > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/>
> |
> > > > > > CloudStack
> > > > > > > > > Bootcamp Training Courses <
> > > > > http://shapeblue.com/cloudstack-training/
> > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > e: mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > > > > o: 303.746.7302
> > > > > > Advancing the way the world uses the cloud
> > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > >
> > >
> > > --
> > > Ian Rae
> > > CEO | PDG
> > > c: 514.944.4008
> > >
> > > CloudOps | Cloud Infrastructure and Networking Solutions
> > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Hi Pierre-Luc,

My recommendation would be this:

Add an "archive" flag to the current volume-snapshot API. Its default would
be "false" because that would be backward compatible with how 4.6 has
volume snapshots implemented (i.e. they stay on the SAN in 4.6, 4.7, and
4.8).

If you set archive=true, then we would perform a background migration of
the snapshot from the SAN to the secondary storage (then delete the SAN
snapshot).

That archive parameter would only be valid for managed storage.

Sound reasonable?

Also, a VM snapshot that includes disks provided by managed storage should
work.

Talk to you later,
Mike

On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <pd...@cloudops.com> wrote:

> Mike,
>
> In terms of API's, would you prefer introducing a parameter to the existing
> VolumeSnapshot, example:   extract={true|false}  with a default value of
> true  which would extract snapshot into the secondary storage, which is the
> current default behavior. Then for SAN snapshot that remain on the SAN we
> would just set "extract=false" ?  as oppose to create a new
>  StorageSnapshot API ?
>
>
> Paul,
>
> From what I'm seeing so far, we can't do a VM-snapshot when using managed
> storage for VM having more than one Volume. For the reason that snapshot
> are performed outside of the hypervisor awareness and asynchronously. If
> someone have a way to address that, it would make thinks much more
> attractive.
>
>
>
>
> On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com> wrote:
>
> > I think a service provider backup scenario is more likely to take
> advantage
> > of SAN snapshot. There are a few reasons for this. Traditional backups
> > involve access to the file system, and there is an expectation that this
> > can be done with reasonably short time frames without negatively
> impacting
> > VM performance, and that the backup orchestrator can apply various logic
> > and or transformations to the data (compress, encrypt, deltas etc...).
> > While it is true that one could apply a backup process to a cloud
> snapshot,
> > this would be slow and inefficient requiring the data to be moved several
> > times and there are some major bottlenecks with cloud snapshots. With a
> > cloud snapshot - there seems to be no reasonable expectation of being
> able
> > to do differential snapshots (I think this depends on the hypervisor) and
> > if you do differential snapshots this will make file backups from those
> > snapshots even more complicated to orchestrate.
> >
> > Suspect there needs to be a different thread on how to better enable
> > backups, as a service. As per Paul's suggestion, but it is a related
> > workflow so relevant to this discussion.
> >
> > Ian
> >
> > On Monday, February 8, 2016, Mike Tutkowski <
> mike.tutkowski@solidfire.com>
> > wrote:
> >
> > > To me it sounds like number two and number three are different uses for
> > the
> > > same "thing"(which is totally fine).
> > >
> > > As for taking a fast SAN snapshot and exporting it asynchronously, do
> we
> > > see the SSVM as performing the export?
> > >
> > > To be backwards compatible with what we have in 4.6 and later for
> volume
> > > snapshots for managed storage, I think it might be easier if we pass
> in a
> > > flag that says whether or not to archive the SAN snapshot (which, I
> > think,
> > > is something that you suggested, as well, Pierre-Luc).
> > >
> > > On Monday, February 8, 2016, Pierre-Luc Dion <pdion@cloudops.com
> > > <javascript:;>> wrote:
> > >
> > > > Hi Mike,
> > > >
> > > > The reason behind the creation of a SAN snapshot which is exported
> into
> > > > secondary storage, is because creating a copy of the .VHD directly
> > would
> > > > impact uptime of the VM as creating that copy take lots of time. Has
> > > oppose
> > > > to a SAN snapshot that is close to instantaneous which can afterward
> be
> > > > clone into Secondary Storage asynchronously.
> > > >
> > > > I would suspect an extracted VolumeSnapshot taken from a SAN snapshot
> > > could
> > > > have is SAN snapshot deleted to avoid duplica and space consumption
> on
> > > the
> > > > Primary Storage side.
> > > >
> > > >
> > > > I see 3 definitions in our current discussion regarding the term
> > snapshot
> > > > (these are not official terminology but by own interpretation of
> them):
> > > >
> > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a
> > snapshot):
> > > > it's a volume snapshot at the storage level, point in time of your
> > data.
> > > it
> > > > reside on the primary storage. Useful and efficient for software side
> > > > incident.
> > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud backup
> > aws-S3
> > > > style ): Point in time copy of the Virtual Disk that reside on a
> > > different
> > > > storage array then the original Volume. Facilitate data migration
> > between
> > > > clusters and, in case of primary storage incident, Volume snapshots
> are
> > > not
> > > > impacted and can be reuse.
> > > > 3. *Backup*: Archival of your Virtual-machines data that also
> validate
> > > data
> > > > integrity, provide a storage efficient archiving method and an
> > > independent
> > > > way to restore your data in case of an major infrastructure disaster.
> > > >
> > > >
> > > > Regards,
> > > >
> > > > PL
> > > >
> > > >
> > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > > > wrote:
> > > >
> > > > > So, let's see if I currently follow the requirements:
> > > > >
> > > > > * Augment volume snapshots for managed storage to conditionally
> > export
> > > > data
> > > > > to NFS. The current process of taking a snapshot on the SAN is
> fine,
> > > but
> > > > > we'd like the option to export the data to NFS, as well.
> > > > >
> > > > > Questions:
> > > > >
> > > > > Once the data has been exported to NFS, do we keep the SAN snapshot
> > or
> > > > > delete it?
> > > > >
> > > > > If we are deleting the SAN snapshot, then why don't we just copy
> the
> > > VHD
> > > > > from primary to secondary the way we do today for non-managed (i.e.
> > > > > traditional) storage? Why create a SAN snapshot in that scenario?
> > > Perhaps
> > > > > to have the SSVM mount and perform the VHD copy to secondary
> storage
> > > > > instead of a XenServer host?
> > > > >
> > > > > Thanks for the clarification.
> > > > >
> > > > > By the way, to me a backup is when you copy data from one storage
> > > system
> > > > to
> > > > > another (regardless of features, if any, to restore the data in the
> > > > > future). A snapshot is a point-in-time view of the data of a volume
> > and
> > > > > it's stored on the same storage system as the volume.
> > > > >
> > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> pdion@cloudops.com
> > > <javascript:;>
> > > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > That's fun to see that discussion happening. I 100% agree with
> > Paul's
> > > > > > points of view. VolumeSnapshot are not a backup, but I do
> consider
> > > them
> > > > > as
> > > > > > a safety vest against Primary Storage failure, because failure
> > append
> > > > > :-( .
> > > > > >
> > > > > > The current proposal around snapshots that reside on the primary
> > > > storage
> > > > > or
> > > > > > snapshots that end in the Secondary Storage  is not to address
> any
> > > kind
> > > > > of
> > > > > > backups requirement because a snapshot is not a backup, event an
> > > > > extracted
> > > > > > VM snapshot.
> > > > > >
> > > > > > The main idea, and again this is for managed storage;
> > > > > >
> > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot capability
> for
> > > > fast
> > > > > > response time that support rollback to previous timestamp, create
> > new
> > > > > > volume and maybe create template.
> > > > > >     not required to be a new API if the work is already done, I
> > think
> > > > > this
> > > > > > is a different behaviors than the user expectation of a
> > > > volume-snapshot.
> > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior that
> > create
> > > > an
> > > > > > extraction of a volume into SecondaryStorage which can be reuse
> to
> > > > > create a
> > > > > > new volume into another Primary Storage. This type of snapshot
> is a
> > > > slow
> > > > > > job since yes it would have to copy the full volume size on the
> > > > Secondary
> > > > > > storage.
> > > > > >
> > > > > >
> > > > > > PL
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > > syed1.mushtaq@gmail.com <javascript:;>
> > > > <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > I think I share you view on the 'Ideal world'. Backup (via
> Volume
> > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is
> amplified
> > > > > > especially
> > > > > > > when you have a object storage as your secondary storage
> because
> > it
> > > > > > > requires two copies (one to an NFS staging area and from there
> to
> > > > > object
> > > > > > > storage). And not to mention that all these copies are
> consuming
> > > > > > hypervisor
> > > > > > > resources. Xenserver's Dom0 is also a huge bottleneck as all
> the
> > > > > Network
> > > > > > > and I/O flow through it. So our intention of proposing the
> > "Storage
> > > > > > > Snapshots" is to give a better way of achiving snapshots while
> > > still
> > > > > > > keeping the original definition of volume snpashots (ie upload
> to
> > > sec
> > > > > > > storage).
> > > > > > >
> > > > > > > But as Erik pointed out volume snapshots are not backups. They
> > > don't
> > > > > work
> > > > > > > form multi-disk LVM volume groups and dynamic disks. I am all
> in
> > > for
> > > > a
> > > > > > > better backup solution which handles these use cases and
> utilizes
> > > the
> > > > > > > storage's advanced features.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > In the beginning... there were CloudStack snapshots and they
> > were
> > > > > > > actually
> > > > > > > > volume snapshots not hypervisor point-in-time snapshots.
> > > > > > > > Then VM snapshots were created (which are point-in-time
> > > hypervisor
> > > > > > > > snapshots) and we started referring to the original snapshots
> > as
> > > > > volume
> > > > > > > > snapshots.
> > > > > > > >
> > > > > > > > CloudStack does not offer 'backups', but many people use
> volume
> > > > > > snapshots
> > > > > > > > as backups. However you can't in-place restore volume
> snapshots
> > > and
> > > > > if
> > > > > > > you
> > > > > > > > have a VM with multiple volumes, the volume snapshots must be
> > > done
> > > > in
> > > > > > > > series, meaning that the state across all of the volumes is
> > > > unlikely
> > > > > to
> > > > > > > be
> > > > > > > > consistent.
> > > > > > > >
> > > > > > > > 'Actual Backups' would enable all of the restore options
> which
> > > > users
> > > > > > > might
> > > > > > > > expect as well options as to where they might be stored. In
> my
> > > > ideal
> > > > > > > world
> > > > > > > > they would also be able to leverage back-end hardware (such
> as
> > > > > > Solidfire,
> > > > > > > > NetApp etc :) ) and software such as Veeam, Commvault etc to
> > > > > accelerate
> > > > > > > the
> > > > > > > > process.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > > > > Paul Angus
> > > > > > > > VP Technology ,  ShapeBlue
> > > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> > > m:
> > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > e:  *paul.angus@shapeblue.com <javascript:;> <javascript:;>
> |
> > > t: @cloudyangus*
> > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > > <ja...@cloudyangus>
> > > > |  w:
> > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > > > > Shape Blue Ltd is a company incorporated in England & Wales.
> > > > > ShapeBlue
> > > > > > > > Services India LLP is a company incorporated in India and is
> > > > operated
> > > > > > > under
> > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria
> Ltda
> > > is
> > > > a
> > > > > > > > company incorporated in Brasil and is operated under license
> > from
> > > > > Shape
> > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > > > > Republic
> > > > > > of
> > > > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > ShapeBlue
> > > > > > > is
> > > > > > > > a registered trademark.
> > > > > > > > This email and any attachments to it may be confidential and
> > are
> > > > > > intended
> > > > > > > > solely for the use of the individual to whom it is addressed.
> > Any
> > > > > views
> > > > > > > or
> > > > > > > > opinions expressed are solely those of the author and do not
> > > > > > necessarily
> > > > > > > > represent those of Shape Blue Ltd or related companies. If
> you
> > > are
> > > > > not
> > > > > > > the
> > > > > > > > intended recipient of this email, you must neither take any
> > > action
> > > > > > based
> > > > > > > > upon its contents, nor copy or show it to anyone. Please
> > contact
> > > > the
> > > > > > > sender
> > > > > > > > if you believe you have received this email in error.
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > > <javascript:;> <javascript:;>]
> > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > > > To: dev@cloudstack.apache.org <javascript:;> <javascript:;>
> > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > >
> > > > > > > > Paul,
> > > > > > > >
> > > > > > > > When you say actual backups, how would it be different from
> the
> > > > > Volume
> > > > > > > > Snapshots that exist currently. My understanding is that
> > Backups
> > > > end
> > > > > up
> > > > > > > in
> > > > > > > > Sec Storage whereas Snapshots are just a point-in-time state
> of
> > > > your
> > > > > > > volume
> > > > > > > > which can be restored back correct?
> > > > > > > >
> > > > > > > > -Syed
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Syed,
> > > > > > > > >
> > > > > > > > > As I understand it, the SolidFire plugin will export the
> > > snapshot
> > > > > to
> > > > > > > > > secondary storage if the user requests a template from the
> > > > snapshot
> > > > > > or
> > > > > > > > > wants to download the snapshot from the cloud. This is a
> > good,
> > > > > > > > > pragmatic approach and yes Mike the SolidFire storage is
> > super
> > > > > > > > > reliable and snapshots on SolidFire arrays take up next to
> no
> > > > > space.
> > > > > > > > > BUT I think that we are talking about a more general
> purpose
> > > API,
> > > > > and
> > > > > > > > > other storage systems may not be as awesome as Mike's.
> That's
> > > my
> > > > > > > > > concern. Also, the time to transfer for say 1TB to move
> from
> > > > > primary
> > > > > > > > > to sec storage and then create a VM template out of it may
> be
> > > too
> > > > > > long
> > > > > > > > for users.
> > > > > > > > >
> > > > > > > > > @Mike I don’t think 'we' use the term volume snapshot for
> > > backup,
> > > > > > it's
> > > > > > > > > just that users want to do backups and a volume snapshot is
> > the
> > > > > only
> > > > > > > > > type of snapshot that copies the disk elsewhere and can be
> > > > > scheduled.
> > > > > > > > >
> > > > > > > > > I'm 'pondering' the implications of enabling actual backups
> > > > > (through
> > > > > > > > > recognised backup providers) and the user requirements
> around
> > > > them
> > > > > > > > > (particularly restoration use cases) as a separate thread
> of
> > > > work.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus
> VP
> > > > > > Technology
> > > > > > > > > , ShapeBlue
> > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > |
> > > m:
> > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> <javascript:;> |
> > > t: @cloudyangus*
> > > > > > > > > <paul.angus@shapeblue.com <javascript:;> <javascript:;>
> > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape
> > > Blue
> > > > > Ltd
> > > > > > > > > is a company incorporated in England & Wales. ShapeBlue
> > > Services
> > > > > > India
> > > > > > > > > LLP is a company incorporated in India and is operated
> under
> > > > > license
> > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
> a
> > > > > company
> > > > > > > > > incorporated in Brasil and is operated under license from
> > Shape
> > > > > Blue
> > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > > Republic
> > > > > of
> > > > > > > > > South Africa and is traded under license from Shape Blue
> Ltd.
> > > > > > > > > ShapeBlue is a registered trademark.
> > > > > > > > > This email and any attachments to it may be confidential
> and
> > > are
> > > > > > > > > intended solely for the use of the individual to whom it is
> > > > > > addressed.
> > > > > > > > > Any views or opinions expressed are solely those of the
> > author
> > > > and
> > > > > do
> > > > > > > > > not necessarily represent those of Shape Blue Ltd or
> related
> > > > > > > > > companies. If you are not the intended recipient of this
> > email,
> > > > you
> > > > > > > > > must neither take any action based upon its contents, nor
> > copy
> > > or
> > > > > > show
> > > > > > > > > it to anyone. Please contact the sender if you believe you
> > have
> > > > > > > received
> > > > > > > > this email in error.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > > <javascript:;>
> > > > <javascript:;>]
> > > > > > > > > Sent: 05 February 2016 15:31
> > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> <javascript:;>
> > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > >
> > > > > > > > > I think the terminology confusion comes from AWS where they
> > do
> > > > EBS
> > > > > > > > > snapshots backed up to S3 and CloudStack sort of followed
> > that.
> > > > And
> > > > > > as
> > > > > > > > > an end user who is oblivious to the internals of my
> provider,
> > > my
> > > > > > > > > expectation would be something similar to what AWS as that
> is
> > > my
> > > > > > > > > biggest reference point.
> > > > > > > > >
> > > > > > > > > To your point Mike, I agree that a Primary Storage failure
> on
> > > > > > > > > SolidFire is unlikely, there are other motivations for us
> to
> > > push
> > > > > > data
> > > > > > > > > to secondary storage. Primary storage (atleast for us)
> costs
> > > > > around 3
> > > > > > > > > times as much as secondary storage and snapshots on primary
> > > > storage
> > > > > > > > > are rarely used (especially for some of our customers who
> do
> > > > daily
> > > > > > > > backups).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>>
> > > wrote:
> > > > > > > > >
> > > > > > > > > > Some of the weirdness is around terminology.
> > > > > > > > > >
> > > > > > > > > > For most systems I've worked on, a snapshot and a backup
> > are
> > > > two
> > > > > > > > > > completely different things (but CloudStack has
> > traditionally
> > > > > used
> > > > > > > > > > the term "volume snapshot" to mean backup).
> > > > > > > > > >
> > > > > > > > > > I will put in a SolidFire "plug" here and say, though,
> that
> > > if
> > > > > your
> > > > > > > > > > primary storage is running on SolidFire that it is
> unlikely
> > > > > you'll
> > > > > > > > > > encounter an issue where your primary storage goes
> offline
> > > (and
> > > > > > > > > > you'll even maintain your performance guarantees during
> > > failure
> > > > > > > > > > scenarios and upgrades, as well). That being the case, it
> > is
> > > > less
> > > > > > > > > > useful to require a backup to Swift (but it's perfectly
> OK
> > if
> > > > > > that's
> > > > > > > > > > what we want to do
> > > > > > > > > here).
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;> <javascript:;>>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Paul,
> > > > > > > > > > >
> > > > > > > > > > > I believe with the current implementation of Snapshots
> on
> > > > > managed
> > > > > > > > > > > storage
> > > > > > > > > > > (SolidFire) the snapshots are never exported to the
> > > secondary
> > > > > > > > storage.
> > > > > > > > > > > While this solves the problem of having snapshots
> taking
> > > > > forever
> > > > > > > > > > > to get to sec storage, this leaves us with a
> > > > > > > > > > huge
> > > > > > > > > > > liability if our primary storage goes down. We see
> > > snapshots
> > > > as
> > > > > > > > > > > our recovery path because we store them in Swift which
> is
> > > > > > reliable
> > > > > > > > > > > and resilient to failures.
> > > > > > > > > > >
> > > > > > > > > > > With Storage snpashots our goal is to have Volume
> > snapshots
> > > > > > always
> > > > > > > > > > > backed up to secondary storage and Storage Snapshots
> stay
> > > on
> > > > > the
> > > > > > > > > > > primary
> > > > > > > > > > storage.
> > > > > > > > > > > A provider could potentially mix both these and solve
> the
> > > > > problem
> > > > > > > > > > > that you mentioned where you want to meet user's
> > > expectation
> > > > > of a
> > > > > > > > > > > snapshot (ie backup to sec storage) while having an
> > ability
> > > > to
> > > > > > > > > > > utilize faster sanpshots (i.e. on the device)
> > > > > > > > > > >
> > > > > > > > > > > Hope this clarifies things.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > -Syed
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> <javascript:;>>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > HI guys,
> > > > > > > > > > > >
> > > > > > > > > > > > Could someone point me to the Jira bug of FS for the
> > > > > > > > > > > > SAN-snapshot
> > > > > > > > > > feature
> > > > > > > > > > > > in 4.6 which is mentioned.
> > > > > > > > > > > >
> > > > > > > > > > > > From my discussions with users and operators around
> > > > snapshots
> > > > > > > > > > > > I'd make
> > > > > > > > > > > the
> > > > > > > > > > > > following observations:
> > > > > > > > > > > > a. 'users' use snapshots as backups (both long-term
> and
> > > > short
> > > > > > > > > > > > term)
> > > > > > > > > > with
> > > > > > > > > > > > the expectation that they can use them for recovery
> if
> > > > > > required.
> > > > > > > > > > > > b. operators fall back to snapshots if something has
> > gone
> > > > > wrong
> > > > > > > > > > > > with primary storage.
> > > > > > > > > > > > c. users sometimes want to be able to export
> snapshots
> > as
> > > > > well
> > > > > > > > > > > > as
> > > > > > > > > > create
> > > > > > > > > > > > new VMs from their snapshots
> > > > > > > > > > > > d. snapshots are a currently a massive pain for
> > > operators,
> > > > I
> > > > > > > > > > > > know at
> > > > > > > > > > > least
> > > > > > > > > > > > one public cloud who have snapshots which take 2 days
> > to
> > > > > > > complete.
> > > > > > > > > > > > e. snapshots (as they are) can't be used for multiple
> > LVM
> > > > > > disks.
> > > > > > > > > > > >
> > > > > > > > > > > > I think the process Mike has used in the SolidFire
> > plugin
> > > > > (only
> > > > > > > > > > > > moving
> > > > > > > > > > > the
> > > > > > > > > > > > disk image to secondary storage when you absolutely
> > have
> > > > to)
> > > > > is
> > > > > > > > > > > > a very
> > > > > > > > > > > good
> > > > > > > > > > > > and pragmatic solution. I wonder what problems an
> > > operator
> > > > > > might
> > > > > > > > > > > experience
> > > > > > > > > > > > if they have an issue with a given primary storage
> pool
> > > in
> > > > a
> > > > > > > > cluster.
> > > > > > > > > > (I
> > > > > > > > > > > > know that that is REALLY unlikely in the SolidFire
> case
> > > > Mike
> > > > > :)
> > > > > > > > > > > > ) And
> > > > > > > > > > if
> > > > > > > > > > > > the transfer from primary to secondary is slow, the
> > time
> > > to
> > > > > > > > > > > > being able
> > > > > > > > > > to
> > > > > > > > > > > > create a template or export the volume will be slow.
> > > > > > > > > > > >
> > > > > > > > > > > > So for me the issue is around making sure that the
> end
> > > > users
> > > > > > > > > > expectations
> > > > > > > > > > > > are met (while improving the speed/efficiency of the
> > back
> > > > > end)
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul
> > Angus
> > > > VP
> > > > > > > > > > > > Technology , ShapeBlue
> > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > > > > >
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > > |
> > > > > > m:
> > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > > <javascript:;> | t:
> > > > @cloudyangus*
> > > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > <javascript:;>
> > > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > Shape
> > > > > > Blue
> > > > > > > > > > > > Ltd is a company incorporated in England & Wales.
> > > ShapeBlue
> > > > > > > > > > > > Services India LLP is a company incorporated in India
> > and
> > > > is
> > > > > > > > > > > > operated
> > > > > > > > > > > under
> > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > Consultoria
> > > > > Ltda
> > > > > > > > > > > > is a company incorporated in Brasil and is operated
> > under
> > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a
> > > > > company
> > > > > > > > > > > > registered by The Republic
> > > > > > > > > > of
> > > > > > > > > > > > South Africa and is traded under license from Shape
> > Blue
> > > > Ltd.
> > > > > > > > > > > > ShapeBlue
> > > > > > > > > > > is
> > > > > > > > > > > > a registered trademark.
> > > > > > > > > > > > This email and any attachments to it may be
> > confidential
> > > > and
> > > > > > are
> > > > > > > > > > intended
> > > > > > > > > > > > solely for the use of the individual to whom it is
> > > > addressed.
> > > > > > > > > > > > Any views
> > > > > > > > > > > or
> > > > > > > > > > > > opinions expressed are solely those of the author and
> > do
> > > > not
> > > > > > > > > > necessarily
> > > > > > > > > > > > represent those of Shape Blue Ltd or related
> companies.
> > > If
> > > > > you
> > > > > > > > > > > > are not
> > > > > > > > > > > the
> > > > > > > > > > > > intended recipient of this email, you must neither
> take
> > > any
> > > > > > > > > > > > action
> > > > > > > > > > based
> > > > > > > > > > > > upon its contents, nor copy or show it to anyone.
> > Please
> > > > > > contact
> > > > > > > > > > > > the
> > > > > > > > > > > sender
> > > > > > > > > > > > if you believe you have received this email in error.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com
> > > <javascript:;>
> > > > <javascript:;>]
> > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > <javascript:;>
> > > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Mike,
> > > > > > > > > > > >
> > > > > > > > > > > > The idea of introducing a new API: StorageSnapshot
> for
> > > > > managed
> > > > > > > > > > > > storage
> > > > > > > > > > is
> > > > > > > > > > > > because the VolumeSnapshot default, or expected,
> > behavior
> > > > is
> > > > > to
> > > > > > > > > > > > archive snapshots into the Secondary Storage. So a
> > > > > > > > > > > > StorageSnapshot API would be
> > > > > > > > > > > for
> > > > > > > > > > > > snapshot that remain on the managed storage
> appliance.
> > > > > > > > > > > >
> > > > > > > > > > > > Quickly looking at the API doc and I don't see a
> strong
> > > > > > > > > > > > requirement for volume snapshots to be moved into
> > > secondary
> > > > > > > > > > > > storage. So, maybe StorageSnapshot API is not useful,
> > but
> > > > > both
> > > > > > > > > > > > use cases are required. A snapshot that remain on the
> > > > managed
> > > > > > > > > > > > storage, and another type of
> > > > > > > > > > snapshot
> > > > > > > > > > > > that end up into the secondary storage. Since you've
> > > done a
> > > > > lot
> > > > > > > > > > > > of
> > > > > > > > > > work,
> > > > > > > > > > > > might easier to just add a parameter to the current
> > > > snapshot
> > > > > > API
> > > > > > > > > > > > that
> > > > > > > > > > > would
> > > > > > > > > > > > trigger an extraction of the storage snapshot into
> the
> > > > > > secondary
> > > > > > > > > > storage?
> > > > > > > > > > > >
> > > > > > > > > > > > PL
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I think that all sounds reasonable then - thanks!
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > > > > > > > syed1.mushtaq@gmail.com <javascript:;> <javascript:;>>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> You are correct Mike in terms of the requirements.
> > One
> > > > of
> > > > > > our
> > > > > > > > > > earlier
> > > > > > > > > > > > >> iterations on this was to have an argument to the
> > > create
> > > > > > > > > > > > >> snapshot
> > > > > > > > > > API
> > > > > > > > > > > > >> which decides whether to backup the volume to sec
> > > > storage
> > > > > > but
> > > > > > > > > > > > >> we realized it would make management of snapshots
> > > quite
> > > > > > messy
> > > > > > > > > > > > >> so we proposed a new api instead.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > > > > > > > >> <mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>> Hi,
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Just to make sure I understand all the
> requirements
> > > > here:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> 1) This relates only to managed storage (1:1
> > mapping
> > > > > > between
> > > > > > > > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> 2) We want to take the current (introduced in
> 4.6)
> > > > > > > > > > > > >>> functionality, which creates a snapshot on the
> SAN,
> > > and
> > > > > > > > > > > > >>> extend it via a config option (or
> > > > > > > > > > > > >>> something) to not only take the SAN snapshot, but
> > to
> > > > copy
> > > > > > > > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> 3) The SAN snapshot is always taken. It's the
> > backup
> > > to
> > > > > NFS
> > > > > > > > > > > > >>> that is optional.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> 4) Templates can be created from the snapshot
> > that's
> > > on
> > > > > the
> > > > > > > > > > > > >>> SAN (already works).
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> 5) CloudStack volumes can be created from the
> > > snapshot
> > > > > > > > > > > > >>> that's on
> > > > > > > > > > the
> > > > > > > > > > > > >>> SAN (already works as long as the new CloudStack
> > > volume
> > > > > > ends
> > > > > > > > > > > > >>> up on the same primary storage).
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Would we have a need for a storage snapshot API
> > then
> > > or
> > > > > > > > > > > > >>> would that just be the standard volume snapshot
> > > without
> > > > > the
> > > > > > > > > > > > >>> backup to
> > > > > > > > > NFS?
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Thanks!
> > > > > > > > > > > > >>> Mike
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
> > > <javascript:;>>
> > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>> Is it possible to have both functionalities
> > > (snapshot
> > > > on
> > > > > > > > > > > > >>>> SAN & Sec
> > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we would like
> > to
> > > > have
> > > > > > > both.
> > > > > > > > > > > > >>>> For example, some of our customers want to
> > implement
> > > > > their
> > > > > > > > > > > > >>>> own backup strategies and do encryption to their
> > > > backups
> > > > > > > > > > > > >>>> which is a perfect use case for Storage Snapshot
> > > while
> > > > > our
> > > > > > > > > > > > >>>> other customers will still keep using the
> standard
> > > > > volume
> > > > > > > > > snapshot.
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> To keep things backward compatible, we can add a
> > > > setting
> > > > > > > > > > > > >>>> which
> > > > > > > > > > says
> > > > > > > > > > > > >>>> to not upload on secondary storage, because,
> after
> > > > all,
> > > > > > you
> > > > > > > > > > > > >>>> would take a SAN snapshot first when doing a
> > Volume
> > > > > > > Snapshot.
> > > > > > > > > > > > >>>> You could stop the process there and not do the
> > > > upload.
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> What do you think about this approach?
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> Thanks,
> > > > > > > > > > > > >>>> -Syed
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > > > > > > > > >>>> mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>> wrote:
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>> So, this is just me thinking out load here, but
> > if
> > > a
> > > > > > given
> > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need to
> provide
> > > > both
> > > > > > the
> > > > > > > > > > ability
> > > > > > > > > > > > >>>>> to take a SAN snapshot and export it to NFS (if
> > > just
> > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we might be
> > able
> > > > to
> > > > > > get
> > > > > > > > > > > > >>>>> away with no new
> > > > > > > > > > API
> > > > > > > > > > > > >>>>> calls and simply implement a new custom
> snapshot
> > > > > strategy
> > > > > > > > > > > > >>>>> and
> > > > > > > > > > data
> > > > > > > > > > > > >>>>> motion strategy to handle the case where the
> > > > CloudStack
> > > > > > > > > > > > >>>>> cloud
> > > > > > > > > > does
> > > > > > > > > > > > >>>>> want both a SAN snapshot and exported-to-NFS
> > > backup.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> In other words, the "default" behavior would be
> > to
> > > > use
> > > > > > the
> > > > > > > > > > > > >>>>> snapshot strategy and data motion strategy that
> > we
> > > > > > already
> > > > > > > > > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> If your CloudStack cloud, however, wants to
> take
> > a
> > > > SAN
> > > > > > > > > > > > >>>>> snapshot and have the data exported to NFS,
> then
> > we
> > > > > could
> > > > > > > > > > > > >>>>> have you manipulate a Swing config file to make
> > use
> > > > of
> > > > > a
> > > > > > > > > > > > >>>>> new snapshot strategy and data motion strategy
> > that
> > > > > > > > > > > > >>>>> performs both of these
> > > > > > > > > > > > activities.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> This way, the old behavior is still the default
> > for
> > > > > > users,
> > > > > > > > > > > > >>>>> but CloudStack admins can change this behavior
> > via
> > > > > > > > > configuration.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> Thoughts?
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike
> Tutkowski <
> > > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>> wrote:
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>>> Right...I think we will need to come up with a
> > > > viable
> > > > > > > > > > > > >>>>>> upgrade path or some reasonable way for them
> to
> > > move
> > > > > > from
> > > > > > > > > > > > >>>>>> the old way to the new way (and some obvious
> way
> > > > that
> > > > > > > > > > > > >>>>>> they will know they need
> > > > > > > > > > to
> > > > > > > > > > > > do this).
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq
> <
> > > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > > <javascript:;>> wrote:
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>> I'm not really sure about the upgrade path
> > > however,
> > > > > > > > > > > > >>>>>>> customers who are using 4.6 and are on a
> > managed
> > > > > > storage
> > > > > > > > > > > > >>>>>>> would no longer have the same functionality
> > with
> > > > > Volume
> > > > > > > > > Snapshots.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq
> <
> > > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > > <javascript:;>> wrote:
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>>> So if I understand correctly, currently
> > taking a
> > > > > > Volume
> > > > > > > > > > > > >>>>>>>> Snapshots of a volume on a managed storage
> > keeps
> > > > it
> > > > > on
> > > > > > > > > > > > >>>>>>>> the storage array. As a part of this
> feature,
> > we
> > > > can
> > > > > > > > > > > > >>>>>>>> make sure
> > > > > > > > > > that
> > > > > > > > > > > > >>>>>>>> Volume Snapshots on managed storage are
> > uploaded
> > > > to
> > > > > > the
> > > > > > > > > > > > >>>>>>>> secondary storage. This would make the
> Volume
> > > > > Snapshot
> > > > > > > > > > > > >>>>>>>> feature behave the same regardless of the
> > > storage
> > > > > > > > > > > > >>>>>>>> (managed or
> > > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing the
> efficient
> > > > > backend
> > > > > > > > > > > > >>>>>>>> storage
> > > > > > > > > > > > capabilities, we can use the new storage snapshots
> API.
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike
> > Tutkowski <
> > > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>>
> > > > wrote:
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Whatever we do here, we need to have a plan
> > to
> > > > deal
> > > > > > > > > > > > >>>>>>>>> with the fact that we already have a
> feature
> > > (in
> > > > > 4.6
> > > > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a volume
> > > snapshot
> > > > > (for
> > > > > > > > > > > > >>>>>>>>> managed
> > > > > > > > > > > > >>>>>>>>> storage) that resides on a backend SAN
> > (using a
> > > > > > custom
> > > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom data motion
> > > > > strategy).
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> If these new APIs go in, then how should
> the
> > > > > original
> > > > > > > > > > > > >>>>>>>>> implementation (present in 4.6 and later)
> be
> > > > > changed?
> > > > > > > > > > > > >>>>>>>>> If it
> > > > > > > > > > is
> > > > > > > > > > > > >>>>>>>>> changed, how do we support customers who
> are
> > > > > already
> > > > > > > > > > > > >>>>>>>>> using
> > > > > > > > > > the
> > > > > > > > > > > > >>>>>>>>> original volume-snapshot API to take
> > snapshots
> > > > on a
> > > > > > > > > > > > >>>>>>>>> backend
> > > > > > > > > > > SAN?
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > > > > > >>>>>>>>> Mike
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will
> > Stevens <
> > > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com <javascript:;>
> > > <javascript:;>> wrote:
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> Will you be able to create a Template
> from a
> > > > > > > > > > StorageSnapshot?
> > > > > > > > > > > > >>>>>>>>>> If yes, will the template be stored in the
> > > > > secondary
> > > > > > > > > > > > >>>>>>>>>> storage like normal templates or will that
> > be
> > > > > > handled
> > > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > > > > > > vendor side?
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|*
> H3J
> > > 1S6
> > > > w
> > > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed
> > Mushtaq <
> > > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > > <javascript:;>> wrote:
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will
> > Stevens
> > > <
> > > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com <javascript:;>
> > > <javascript:;>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in
> the
> > > > Jira
> > > > > > > > > > > > >>>>>>>>>>>> ticket because it was not clear in the
> > > > 'mention'
> > > > > > > > > > > > >>>>>>>>>>>> section because it shows as a page 'you
> do
> > > not
> > > > > > have
> > > > > > > > > permission to'.
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|*
> > H3J
> > > > 1S6
> > > > > w
> > > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed
> Ahmed
> > > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com <javascript:;>
> > > <javascript:;>
> > > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to
> > do
> > > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage backends
> > like
> > > > > > > SolidFire.
> > > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed storage
> stay
> > > on
> > > > > the
> > > > > > > > > > > > >>>>>>>>>>>>> device which is contrary to what
> > CloudStack
> > > > > calls
> > > > > > > > > snpshots.
> > > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage and
> > keeping
> > > > it
> > > > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > > > > > > > and
> > > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to have both ways
> > of
> > > > > doing
> > > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new
> APIs
> > to
> > > > > > create
> > > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of this
> feature? I
> > > > would
> > > > > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am
> working
> > on
> > > > > > making
> > > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > > > > > spec
> > > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a high
> > > level
> > > > > > > > > > > > >>>>>>>>>>>>> feedback first before starting to work
> on
> > > it.
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> --
> > > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire
> Inc.*
> > > > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > <javascript:;>
> > > <javascript:;>
> > > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > > > > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > > >>>>>>>>> <
> > > > > http://solidfire.com/solution/overview/?video=play
> > > > > > >*™
> > > > > > > > > > > > >>>>>>>>> *
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> --
> > > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> <javascript:;>
> > > <javascript:;>
> > > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > > > > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > > >>>>>> <
> > > http://solidfire.com/solution/overview/?video=play
> > > > > >*™*
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> --
> > > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>
> > > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > > > > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > > >>>>> <
> > > http://solidfire.com/solution/overview/?video=play
> > > > > >*™*
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> --
> > > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > > > > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>
> > > > > > > > > > > > >>> o: 303.746.7302
> > > > > > > > > > > > >>> Advancing the way the world uses the cloud
> > > > > > > > > > > > >>> <
> > http://solidfire.com/solution/overview/?video=play
> > > > >*™*
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>
> > > > > > > > > > > > > o: 303.746.7302
> > > > > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > > > > <
> http://solidfire.com/solution/overview/?video=play
> > > >*™*
> > > > > > > > > > > > >
> > > > > > > > > > > > Find out more about ShapeBlue and our range of
> > CloudStack
> > > > > > > > > > > > related
> > > > > > > > > > > services:
> > > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//>
> |
> > > > > > CSForge –
> > > > > > > > > > > > rapid IaaS deployment framework <
> > > > > http://shapeblue.com/csforge/
> > > > > > >
> > > > > > > > > > > > CloudStack Consulting
> > > > > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > > > > > > > CloudStack
> > > > > > > > > > > > Software Engineering
> > > > > > > > > > > > <
> http://shapeblue.com/cloudstack-software-engineering/
> > >
> > > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > > > > <
> > http://shapeblue.com/cloudstack-infrastructure-support/
> > > >
> > > > |
> > > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > <javascript:;>
> > > > > > > > > > o: 303.746.7302
> > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > > > >
> > > > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > related
> > > > > > > > services:
> > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > CSForge –
> > > > > > rapid
> > > > > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > CloudStack
> > > > > > > > > Consulting <http://shapeblue.com/cloudstack-consultancy/>
> |
> > > > > > CloudStack
> > > > > > > > > Software Engineering
> > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/>
> |
> > > > > > CloudStack
> > > > > > > > > Bootcamp Training Courses <
> > > > > http://shapeblue.com/cloudstack-training/
> > > > > > >
> > > > > > > > >
> > > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > related
> > > > > > > services:
> > > > > > > > IaaS Cloud Design & Build
> > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > CSForge –
> > > > > rapid
> > > > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > > > > CloudStack Consulting <
> > > > http://shapeblue.com/cloudstack-consultancy/>
> > > > > |
> > > > > > > CloudStack
> > > > > > > > Software Engineering
> > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > CloudStack Infrastructure Support
> > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > > CloudStack
> > > > > > > > Bootcamp Training Courses <
> > > > http://shapeblue.com/cloudstack-training/
> > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> >
> >
> > --
> > Ian Rae
> > CEO | PDG
> > c: 514.944.4008
> >
> > CloudOps | Cloud Infrastructure and Networking Solutions
> > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Pierre-Luc Dion <pd...@cloudops.com>]

Mike,

In terms of API's, would you prefer introducing a parameter to the existing
VolumeSnapshot, example:   extract={true|false}  with a default value of
true  which would extract snapshot into the secondary storage, which is the
current default behavior. Then for SAN snapshot that remain on the SAN we
would just set "extract=false" ?  as oppose to create a new
 StorageSnapshot API ?


Paul,

>From what I'm seeing so far, we can't do a VM-snapshot when using managed
storage for VM having more than one Volume. For the reason that snapshot
are performed outside of the hypervisor awareness and asynchronously. If
someone have a way to address that, it would make thinks much more
attractive.




On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <ir...@cloudops.com> wrote:

> I think a service provider backup scenario is more likely to take advantage
> of SAN snapshot. There are a few reasons for this. Traditional backups
> involve access to the file system, and there is an expectation that this
> can be done with reasonably short time frames without negatively impacting
> VM performance, and that the backup orchestrator can apply various logic
> and or transformations to the data (compress, encrypt, deltas etc...).
> While it is true that one could apply a backup process to a cloud snapshot,
> this would be slow and inefficient requiring the data to be moved several
> times and there are some major bottlenecks with cloud snapshots. With a
> cloud snapshot - there seems to be no reasonable expectation of being able
> to do differential snapshots (I think this depends on the hypervisor) and
> if you do differential snapshots this will make file backups from those
> snapshots even more complicated to orchestrate.
>
> Suspect there needs to be a different thread on how to better enable
> backups, as a service. As per Paul's suggestion, but it is a related
> workflow so relevant to this discussion.
>
> Ian
>
> On Monday, February 8, 2016, Mike Tutkowski <mi...@solidfire.com>
> wrote:
>
> > To me it sounds like number two and number three are different uses for
> the
> > same "thing"(which is totally fine).
> >
> > As for taking a fast SAN snapshot and exporting it asynchronously, do we
> > see the SSVM as performing the export?
> >
> > To be backwards compatible with what we have in 4.6 and later for volume
> > snapshots for managed storage, I think it might be easier if we pass in a
> > flag that says whether or not to archive the SAN snapshot (which, I
> think,
> > is something that you suggested, as well, Pierre-Luc).
> >
> > On Monday, February 8, 2016, Pierre-Luc Dion <pdion@cloudops.com
> > <javascript:;>> wrote:
> >
> > > Hi Mike,
> > >
> > > The reason behind the creation of a SAN snapshot which is exported into
> > > secondary storage, is because creating a copy of the .VHD directly
> would
> > > impact uptime of the VM as creating that copy take lots of time. Has
> > oppose
> > > to a SAN snapshot that is close to instantaneous which can afterward be
> > > clone into Secondary Storage asynchronously.
> > >
> > > I would suspect an extracted VolumeSnapshot taken from a SAN snapshot
> > could
> > > have is SAN snapshot deleted to avoid duplica and space consumption on
> > the
> > > Primary Storage side.
> > >
> > >
> > > I see 3 definitions in our current discussion regarding the term
> snapshot
> > > (these are not official terminology but by own interpretation of them):
> > >
> > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a
> snapshot):
> > > it's a volume snapshot at the storage level, point in time of your
> data.
> > it
> > > reside on the primary storage. Useful and efficient for software side
> > > incident.
> > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud backup
> aws-S3
> > > style ): Point in time copy of the Virtual Disk that reside on a
> > different
> > > storage array then the original Volume. Facilitate data migration
> between
> > > clusters and, in case of primary storage incident, Volume snapshots are
> > not
> > > impacted and can be reuse.
> > > 3. *Backup*: Archival of your Virtual-machines data that also validate
> > data
> > > integrity, provide a storage efficient archiving method and an
> > independent
> > > way to restore your data in case of an major infrastructure disaster.
> > >
> > >
> > > Regards,
> > >
> > > PL
> > >
> > >
> > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > > wrote:
> > >
> > > > So, let's see if I currently follow the requirements:
> > > >
> > > > * Augment volume snapshots for managed storage to conditionally
> export
> > > data
> > > > to NFS. The current process of taking a snapshot on the SAN is fine,
> > but
> > > > we'd like the option to export the data to NFS, as well.
> > > >
> > > > Questions:
> > > >
> > > > Once the data has been exported to NFS, do we keep the SAN snapshot
> or
> > > > delete it?
> > > >
> > > > If we are deleting the SAN snapshot, then why don't we just copy the
> > VHD
> > > > from primary to secondary the way we do today for non-managed (i.e.
> > > > traditional) storage? Why create a SAN snapshot in that scenario?
> > Perhaps
> > > > to have the SSVM mount and perform the VHD copy to secondary storage
> > > > instead of a XenServer host?
> > > >
> > > > Thanks for the clarification.
> > > >
> > > > By the way, to me a backup is when you copy data from one storage
> > system
> > > to
> > > > another (regardless of features, if any, to restore the data in the
> > > > future). A snapshot is a point-in-time view of the data of a volume
> and
> > > > it's stored on the same storage system as the volume.
> > > >
> > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <pdion@cloudops.com
> > <javascript:;>
> > > <javascript:;>>
> > > > wrote:
> > > >
> > > > > That's fun to see that discussion happening. I 100% agree with
> Paul's
> > > > > points of view. VolumeSnapshot are not a backup, but I do consider
> > them
> > > > as
> > > > > a safety vest against Primary Storage failure, because failure
> append
> > > > :-( .
> > > > >
> > > > > The current proposal around snapshots that reside on the primary
> > > storage
> > > > or
> > > > > snapshots that end in the Secondary Storage  is not to address any
> > kind
> > > > of
> > > > > backups requirement because a snapshot is not a backup, event an
> > > > extracted
> > > > > VM snapshot.
> > > > >
> > > > > The main idea, and again this is for managed storage;
> > > > >
> > > > > 1. StorageSnapshotAPI: Provide storage side snapshot capability for
> > > fast
> > > > > response time that support rollback to previous timestamp, create
> new
> > > > > volume and maybe create template.
> > > > >     not required to be a new API if the work is already done, I
> think
> > > > this
> > > > > is a different behaviors than the user expectation of a
> > > volume-snapshot.
> > > > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior that
> create
> > > an
> > > > > extraction of a volume into SecondaryStorage which can be reuse to
> > > > create a
> > > > > new volume into another Primary Storage. This type of snapshot is a
> > > slow
> > > > > job since yes it would have to copy the full volume size on the
> > > Secondary
> > > > > storage.
> > > > >
> > > > >
> > > > > PL
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > syed1.mushtaq@gmail.com <javascript:;>
> > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > I think I share you view on the 'Ideal world'. Backup (via Volume
> > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is amplified
> > > > > especially
> > > > > > when you have a object storage as your secondary storage because
> it
> > > > > > requires two copies (one to an NFS staging area and from there to
> > > > object
> > > > > > storage). And not to mention that all these copies are consuming
> > > > > hypervisor
> > > > > > resources. Xenserver's Dom0 is also a huge bottleneck as all the
> > > > Network
> > > > > > and I/O flow through it. So our intention of proposing the
> "Storage
> > > > > > Snapshots" is to give a better way of achiving snapshots while
> > still
> > > > > > keeping the original definition of volume snpashots (ie upload to
> > sec
> > > > > > storage).
> > > > > >
> > > > > > But as Erik pointed out volume snapshots are not backups. They
> > don't
> > > > work
> > > > > > form multi-disk LVM volume groups and dynamic disks. I am all in
> > for
> > > a
> > > > > > better backup solution which handles these use cases and utilizes
> > the
> > > > > > storage's advanced features.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > In the beginning... there were CloudStack snapshots and they
> were
> > > > > > actually
> > > > > > > volume snapshots not hypervisor point-in-time snapshots.
> > > > > > > Then VM snapshots were created (which are point-in-time
> > hypervisor
> > > > > > > snapshots) and we started referring to the original snapshots
> as
> > > > volume
> > > > > > > snapshots.
> > > > > > >
> > > > > > > CloudStack does not offer 'backups', but many people use volume
> > > > > snapshots
> > > > > > > as backups. However you can't in-place restore volume snapshots
> > and
> > > > if
> > > > > > you
> > > > > > > have a VM with multiple volumes, the volume snapshots must be
> > done
> > > in
> > > > > > > series, meaning that the state across all of the volumes is
> > > unlikely
> > > > to
> > > > > > be
> > > > > > > consistent.
> > > > > > >
> > > > > > > 'Actual Backups' would enable all of the restore options which
> > > users
> > > > > > might
> > > > > > > expect as well options as to where they might be stored. In my
> > > ideal
> > > > > > world
> > > > > > > they would also be able to leverage back-end hardware (such as
> > > > > Solidfire,
> > > > > > > NetApp etc :) ) and software such as Veeam, Commvault etc to
> > > > accelerate
> > > > > > the
> > > > > > > process.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > > > Paul Angus
> > > > > > > VP Technology ,  ShapeBlue
> > > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> > m:
> > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > e:  *paul.angus@shapeblue.com <javascript:;> <javascript:;> |
> > t: @cloudyangus*
> > > > > > > <paul.angus@shapeblue.com <javascript:;>
> > <ja...@cloudyangus>
> > > |  w:
> > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > > > Shape Blue Ltd is a company incorporated in England & Wales.
> > > > ShapeBlue
> > > > > > > Services India LLP is a company incorporated in India and is
> > > operated
> > > > > > under
> > > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> > is
> > > a
> > > > > > > company incorporated in Brasil and is operated under license
> from
> > > > Shape
> > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > > > Republic
> > > > > of
> > > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > ShapeBlue
> > > > > > is
> > > > > > > a registered trademark.
> > > > > > > This email and any attachments to it may be confidential and
> are
> > > > > intended
> > > > > > > solely for the use of the individual to whom it is addressed.
> Any
> > > > views
> > > > > > or
> > > > > > > opinions expressed are solely those of the author and do not
> > > > > necessarily
> > > > > > > represent those of Shape Blue Ltd or related companies. If you
> > are
> > > > not
> > > > > > the
> > > > > > > intended recipient of this email, you must neither take any
> > action
> > > > > based
> > > > > > > upon its contents, nor copy or show it to anyone. Please
> contact
> > > the
> > > > > > sender
> > > > > > > if you believe you have received this email in error.
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > <javascript:;> <javascript:;>]
> > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > > To: dev@cloudstack.apache.org <javascript:;> <javascript:;>
> > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > >
> > > > > > > Paul,
> > > > > > >
> > > > > > > When you say actual backups, how would it be different from the
> > > > Volume
> > > > > > > Snapshots that exist currently. My understanding is that
> Backups
> > > end
> > > > up
> > > > > > in
> > > > > > > Sec Storage whereas Snapshots are just a point-in-time state of
> > > your
> > > > > > volume
> > > > > > > which can be restored back correct?
> > > > > > >
> > > > > > > -Syed
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Syed,
> > > > > > > >
> > > > > > > > As I understand it, the SolidFire plugin will export the
> > snapshot
> > > > to
> > > > > > > > secondary storage if the user requests a template from the
> > > snapshot
> > > > > or
> > > > > > > > wants to download the snapshot from the cloud. This is a
> good,
> > > > > > > > pragmatic approach and yes Mike the SolidFire storage is
> super
> > > > > > > > reliable and snapshots on SolidFire arrays take up next to no
> > > > space.
> > > > > > > > BUT I think that we are talking about a more general purpose
> > API,
> > > > and
> > > > > > > > other storage systems may not be as awesome as Mike's. That's
> > my
> > > > > > > > concern. Also, the time to transfer for say 1TB to move from
> > > > primary
> > > > > > > > to sec storage and then create a VM template out of it may be
> > too
> > > > > long
> > > > > > > for users.
> > > > > > > >
> > > > > > > > @Mike I don’t think 'we' use the term volume snapshot for
> > backup,
> > > > > it's
> > > > > > > > just that users want to do backups and a volume snapshot is
> the
> > > > only
> > > > > > > > type of snapshot that copies the disk elsewhere and can be
> > > > scheduled.
> > > > > > > >
> > > > > > > > I'm 'pondering' the implications of enabling actual backups
> > > > (through
> > > > > > > > recognised backup providers) and the user requirements around
> > > them
> > > > > > > > (particularly restoration use cases) as a separate thread of
> > > work.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > > Technology
> > > > > > > > , ShapeBlue
> > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> |
> > m:
> > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > e: *paul.angus@shapeblue.com <javascript:;> <javascript:;> |
> > t: @cloudyangus*
> > > > > > > > <paul.angus@shapeblue.com <javascript:;> <javascript:;>
> > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape
> > Blue
> > > > Ltd
> > > > > > > > is a company incorporated in England & Wales. ShapeBlue
> > Services
> > > > > India
> > > > > > > > LLP is a company incorporated in India and is operated under
> > > > license
> > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > > > company
> > > > > > > > incorporated in Brasil and is operated under license from
> Shape
> > > > Blue
> > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > Republic
> > > > of
> > > > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > > > > ShapeBlue is a registered trademark.
> > > > > > > > This email and any attachments to it may be confidential and
> > are
> > > > > > > > intended solely for the use of the individual to whom it is
> > > > > addressed.
> > > > > > > > Any views or opinions expressed are solely those of the
> author
> > > and
> > > > do
> > > > > > > > not necessarily represent those of Shape Blue Ltd or related
> > > > > > > > companies. If you are not the intended recipient of this
> email,
> > > you
> > > > > > > > must neither take any action based upon its contents, nor
> copy
> > or
> > > > > show
> > > > > > > > it to anyone. Please contact the sender if you believe you
> have
> > > > > > received
> > > > > > > this email in error.
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > <javascript:;>
> > > <javascript:;>]
> > > > > > > > Sent: 05 February 2016 15:31
> > > > > > > > To: dev@cloudstack.apache.org <javascript:;> <javascript:;>
> > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > >
> > > > > > > > I think the terminology confusion comes from AWS where they
> do
> > > EBS
> > > > > > > > snapshots backed up to S3 and CloudStack sort of followed
> that.
> > > And
> > > > > as
> > > > > > > > an end user who is oblivious to the internals of my provider,
> > my
> > > > > > > > expectation would be something similar to what AWS as that is
> > my
> > > > > > > > biggest reference point.
> > > > > > > >
> > > > > > > > To your point Mike, I agree that a Primary Storage failure on
> > > > > > > > SolidFire is unlikely, there are other motivations for us to
> > push
> > > > > data
> > > > > > > > to secondary storage. Primary storage (atleast for us) costs
> > > > around 3
> > > > > > > > times as much as secondary storage and snapshots on primary
> > > storage
> > > > > > > > are rarely used (especially for some of our customers who do
> > > daily
> > > > > > > backups).
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > > > > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>>
> > wrote:
> > > > > > > >
> > > > > > > > > Some of the weirdness is around terminology.
> > > > > > > > >
> > > > > > > > > For most systems I've worked on, a snapshot and a backup
> are
> > > two
> > > > > > > > > completely different things (but CloudStack has
> traditionally
> > > > used
> > > > > > > > > the term "volume snapshot" to mean backup).
> > > > > > > > >
> > > > > > > > > I will put in a SolidFire "plug" here and say, though, that
> > if
> > > > your
> > > > > > > > > primary storage is running on SolidFire that it is unlikely
> > > > you'll
> > > > > > > > > encounter an issue where your primary storage goes offline
> > (and
> > > > > > > > > you'll even maintain your performance guarantees during
> > failure
> > > > > > > > > scenarios and upgrades, as well). That being the case, it
> is
> > > less
> > > > > > > > > useful to require a backup to Swift (but it's perfectly OK
> if
> > > > > that's
> > > > > > > > > what we want to do
> > > > > > > > here).
> > > > > > > > >
> > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > > > > > > <syed1.mushtaq@gmail.com <javascript:;> <javascript:;>>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Paul,
> > > > > > > > > >
> > > > > > > > > > I believe with the current implementation of Snapshots on
> > > > managed
> > > > > > > > > > storage
> > > > > > > > > > (SolidFire) the snapshots are never exported to the
> > secondary
> > > > > > > storage.
> > > > > > > > > > While this solves the problem of having snapshots taking
> > > > forever
> > > > > > > > > > to get to sec storage, this leaves us with a
> > > > > > > > > huge
> > > > > > > > > > liability if our primary storage goes down. We see
> > snapshots
> > > as
> > > > > > > > > > our recovery path because we store them in Swift which is
> > > > > reliable
> > > > > > > > > > and resilient to failures.
> > > > > > > > > >
> > > > > > > > > > With Storage snpashots our goal is to have Volume
> snapshots
> > > > > always
> > > > > > > > > > backed up to secondary storage and Storage Snapshots stay
> > on
> > > > the
> > > > > > > > > > primary
> > > > > > > > > storage.
> > > > > > > > > > A provider could potentially mix both these and solve the
> > > > problem
> > > > > > > > > > that you mentioned where you want to meet user's
> > expectation
> > > > of a
> > > > > > > > > > snapshot (ie backup to sec storage) while having an
> ability
> > > to
> > > > > > > > > > utilize faster sanpshots (i.e. on the device)
> > > > > > > > > >
> > > > > > > > > > Hope this clarifies things.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > -Syed
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > HI guys,
> > > > > > > > > > >
> > > > > > > > > > > Could someone point me to the Jira bug of FS for the
> > > > > > > > > > > SAN-snapshot
> > > > > > > > > feature
> > > > > > > > > > > in 4.6 which is mentioned.
> > > > > > > > > > >
> > > > > > > > > > > From my discussions with users and operators around
> > > snapshots
> > > > > > > > > > > I'd make
> > > > > > > > > > the
> > > > > > > > > > > following observations:
> > > > > > > > > > > a. 'users' use snapshots as backups (both long-term and
> > > short
> > > > > > > > > > > term)
> > > > > > > > > with
> > > > > > > > > > > the expectation that they can use them for recovery if
> > > > > required.
> > > > > > > > > > > b. operators fall back to snapshots if something has
> gone
> > > > wrong
> > > > > > > > > > > with primary storage.
> > > > > > > > > > > c. users sometimes want to be able to export snapshots
> as
> > > > well
> > > > > > > > > > > as
> > > > > > > > > create
> > > > > > > > > > > new VMs from their snapshots
> > > > > > > > > > > d. snapshots are a currently a massive pain for
> > operators,
> > > I
> > > > > > > > > > > know at
> > > > > > > > > > least
> > > > > > > > > > > one public cloud who have snapshots which take 2 days
> to
> > > > > > complete.
> > > > > > > > > > > e. snapshots (as they are) can't be used for multiple
> LVM
> > > > > disks.
> > > > > > > > > > >
> > > > > > > > > > > I think the process Mike has used in the SolidFire
> plugin
> > > > (only
> > > > > > > > > > > moving
> > > > > > > > > > the
> > > > > > > > > > > disk image to secondary storage when you absolutely
> have
> > > to)
> > > > is
> > > > > > > > > > > a very
> > > > > > > > > > good
> > > > > > > > > > > and pragmatic solution. I wonder what problems an
> > operator
> > > > > might
> > > > > > > > > > experience
> > > > > > > > > > > if they have an issue with a given primary storage pool
> > in
> > > a
> > > > > > > cluster.
> > > > > > > > > (I
> > > > > > > > > > > know that that is REALLY unlikely in the SolidFire case
> > > Mike
> > > > :)
> > > > > > > > > > > ) And
> > > > > > > > > if
> > > > > > > > > > > the transfer from primary to secondary is slow, the
> time
> > to
> > > > > > > > > > > being able
> > > > > > > > > to
> > > > > > > > > > > create a template or export the volume will be slow.
> > > > > > > > > > >
> > > > > > > > > > > So for me the issue is around making sure that the end
> > > users
> > > > > > > > > expectations
> > > > > > > > > > > are met (while improving the speed/efficiency of the
> back
> > > > end)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul
> Angus
> > > VP
> > > > > > > > > > > Technology , ShapeBlue
> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > > > >
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > > |
> > > > > m:
> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> > <javascript:;> | t:
> > > @cloudyangus*
> > > > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> <javascript:;>
> > > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> > Shape
> > > > > Blue
> > > > > > > > > > > Ltd is a company incorporated in England & Wales.
> > ShapeBlue
> > > > > > > > > > > Services India LLP is a company incorporated in India
> and
> > > is
> > > > > > > > > > > operated
> > > > > > > > > > under
> > > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > Consultoria
> > > > Ltda
> > > > > > > > > > > is a company incorporated in Brasil and is operated
> under
> > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a
> > > > company
> > > > > > > > > > > registered by The Republic
> > > > > > > > > of
> > > > > > > > > > > South Africa and is traded under license from Shape
> Blue
> > > Ltd.
> > > > > > > > > > > ShapeBlue
> > > > > > > > > > is
> > > > > > > > > > > a registered trademark.
> > > > > > > > > > > This email and any attachments to it may be
> confidential
> > > and
> > > > > are
> > > > > > > > > intended
> > > > > > > > > > > solely for the use of the individual to whom it is
> > > addressed.
> > > > > > > > > > > Any views
> > > > > > > > > > or
> > > > > > > > > > > opinions expressed are solely those of the author and
> do
> > > not
> > > > > > > > > necessarily
> > > > > > > > > > > represent those of Shape Blue Ltd or related companies.
> > If
> > > > you
> > > > > > > > > > > are not
> > > > > > > > > > the
> > > > > > > > > > > intended recipient of this email, you must neither take
> > any
> > > > > > > > > > > action
> > > > > > > > > based
> > > > > > > > > > > upon its contents, nor copy or show it to anyone.
> Please
> > > > > contact
> > > > > > > > > > > the
> > > > > > > > > > sender
> > > > > > > > > > > if you believe you have received this email in error.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com
> > <javascript:;>
> > > <javascript:;>]
> > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > <javascript:;>
> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > > > >
> > > > > > > > > > > Hi Mike,
> > > > > > > > > > >
> > > > > > > > > > > The idea of introducing a new API: StorageSnapshot for
> > > > managed
> > > > > > > > > > > storage
> > > > > > > > > is
> > > > > > > > > > > because the VolumeSnapshot default, or expected,
> behavior
> > > is
> > > > to
> > > > > > > > > > > archive snapshots into the Secondary Storage. So a
> > > > > > > > > > > StorageSnapshot API would be
> > > > > > > > > > for
> > > > > > > > > > > snapshot that remain on the managed storage appliance.
> > > > > > > > > > >
> > > > > > > > > > > Quickly looking at the API doc and I don't see a strong
> > > > > > > > > > > requirement for volume snapshots to be moved into
> > secondary
> > > > > > > > > > > storage. So, maybe StorageSnapshot API is not useful,
> but
> > > > both
> > > > > > > > > > > use cases are required. A snapshot that remain on the
> > > managed
> > > > > > > > > > > storage, and another type of
> > > > > > > > > snapshot
> > > > > > > > > > > that end up into the secondary storage. Since you've
> > done a
> > > > lot
> > > > > > > > > > > of
> > > > > > > > > work,
> > > > > > > > > > > might easier to just add a parameter to the current
> > > snapshot
> > > > > API
> > > > > > > > > > > that
> > > > > > > > > > would
> > > > > > > > > > > trigger an extraction of the storage snapshot into the
> > > > > secondary
> > > > > > > > > storage?
> > > > > > > > > > >
> > > > > > > > > > > PL
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>
> > > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I think that all sounds reasonable then - thanks!
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > > > > > > syed1.mushtaq@gmail.com <javascript:;> <javascript:;>>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> You are correct Mike in terms of the requirements.
> One
> > > of
> > > > > our
> > > > > > > > > earlier
> > > > > > > > > > > >> iterations on this was to have an argument to the
> > create
> > > > > > > > > > > >> snapshot
> > > > > > > > > API
> > > > > > > > > > > >> which decides whether to backup the volume to sec
> > > storage
> > > > > but
> > > > > > > > > > > >> we realized it would make management of snapshots
> > quite
> > > > > messy
> > > > > > > > > > > >> so we proposed a new api instead.
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > > > > > > >> <mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >>> Hi,
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Just to make sure I understand all the requirements
> > > here:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> 1) This relates only to managed storage (1:1
> mapping
> > > > > between
> > > > > > > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > > > > > > > > >>> functionality, which creates a snapshot on the SAN,
> > and
> > > > > > > > > > > >>> extend it via a config option (or
> > > > > > > > > > > >>> something) to not only take the SAN snapshot, but
> to
> > > copy
> > > > > > > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> 3) The SAN snapshot is always taken. It's the
> backup
> > to
> > > > NFS
> > > > > > > > > > > >>> that is optional.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> 4) Templates can be created from the snapshot
> that's
> > on
> > > > the
> > > > > > > > > > > >>> SAN (already works).
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> 5) CloudStack volumes can be created from the
> > snapshot
> > > > > > > > > > > >>> that's on
> > > > > > > > > the
> > > > > > > > > > > >>> SAN (already works as long as the new CloudStack
> > volume
> > > > > ends
> > > > > > > > > > > >>> up on the same primary storage).
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Would we have a need for a storage snapshot API
> then
> > or
> > > > > > > > > > > >>> would that just be the standard volume snapshot
> > without
> > > > the
> > > > > > > > > > > >>> backup to
> > > > > > > > NFS?
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Thanks!
> > > > > > > > > > > >>> Mike
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
> > <javascript:;>>
> > > > > > > > > > > >>> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> Is it possible to have both functionalities
> > (snapshot
> > > on
> > > > > > > > > > > >>>> SAN & Sec
> > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we would like
> to
> > > have
> > > > > > both.
> > > > > > > > > > > >>>> For example, some of our customers want to
> implement
> > > > their
> > > > > > > > > > > >>>> own backup strategies and do encryption to their
> > > backups
> > > > > > > > > > > >>>> which is a perfect use case for Storage Snapshot
> > while
> > > > our
> > > > > > > > > > > >>>> other customers will still keep using the standard
> > > > volume
> > > > > > > > snapshot.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> To keep things backward compatible, we can add a
> > > setting
> > > > > > > > > > > >>>> which
> > > > > > > > > says
> > > > > > > > > > > >>>> to not upload on secondary storage, because, after
> > > all,
> > > > > you
> > > > > > > > > > > >>>> would take a SAN snapshot first when doing a
> Volume
> > > > > > Snapshot.
> > > > > > > > > > > >>>> You could stop the process there and not do the
> > > upload.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> What do you think about this approach?
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Thanks,
> > > > > > > > > > > >>>> -Syed
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > > > > > > > >>>> mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>> wrote:
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> So, this is just me thinking out load here, but
> if
> > a
> > > > > given
> > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need to provide
> > > both
> > > > > the
> > > > > > > > > ability
> > > > > > > > > > > >>>>> to take a SAN snapshot and export it to NFS (if
> > just
> > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we might be
> able
> > > to
> > > > > get
> > > > > > > > > > > >>>>> away with no new
> > > > > > > > > API
> > > > > > > > > > > >>>>> calls and simply implement a new custom snapshot
> > > > strategy
> > > > > > > > > > > >>>>> and
> > > > > > > > > data
> > > > > > > > > > > >>>>> motion strategy to handle the case where the
> > > CloudStack
> > > > > > > > > > > >>>>> cloud
> > > > > > > > > does
> > > > > > > > > > > >>>>> want both a SAN snapshot and exported-to-NFS
> > backup.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> In other words, the "default" behavior would be
> to
> > > use
> > > > > the
> > > > > > > > > > > >>>>> snapshot strategy and data motion strategy that
> we
> > > > > already
> > > > > > > > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> If your CloudStack cloud, however, wants to take
> a
> > > SAN
> > > > > > > > > > > >>>>> snapshot and have the data exported to NFS, then
> we
> > > > could
> > > > > > > > > > > >>>>> have you manipulate a Swing config file to make
> use
> > > of
> > > > a
> > > > > > > > > > > >>>>> new snapshot strategy and data motion strategy
> that
> > > > > > > > > > > >>>>> performs both of these
> > > > > > > > > > > activities.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> This way, the old behavior is still the default
> for
> > > > > users,
> > > > > > > > > > > >>>>> but CloudStack admins can change this behavior
> via
> > > > > > > > configuration.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Thoughts?
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > > > > > > > > >>>>> mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>> wrote:
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>> Right...I think we will need to come up with a
> > > viable
> > > > > > > > > > > >>>>>> upgrade path or some reasonable way for them to
> > move
> > > > > from
> > > > > > > > > > > >>>>>> the old way to the new way (and some obvious way
> > > that
> > > > > > > > > > > >>>>>> they will know they need
> > > > > > > > > to
> > > > > > > > > > > do this).
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > <javascript:;>> wrote:
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> I'm not really sure about the upgrade path
> > however,
> > > > > > > > > > > >>>>>>> customers who are using 4.6 and are on a
> managed
> > > > > storage
> > > > > > > > > > > >>>>>>> would no longer have the same functionality
> with
> > > > Volume
> > > > > > > > Snapshots.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > <javascript:;>> wrote:
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> So if I understand correctly, currently
> taking a
> > > > > Volume
> > > > > > > > > > > >>>>>>>> Snapshots of a volume on a managed storage
> keeps
> > > it
> > > > on
> > > > > > > > > > > >>>>>>>> the storage array. As a part of this feature,
> we
> > > can
> > > > > > > > > > > >>>>>>>> make sure
> > > > > > > > > that
> > > > > > > > > > > >>>>>>>> Volume Snapshots on managed storage are
> uploaded
> > > to
> > > > > the
> > > > > > > > > > > >>>>>>>> secondary storage. This would make the Volume
> > > > Snapshot
> > > > > > > > > > > >>>>>>>> feature behave the same regardless of the
> > storage
> > > > > > > > > > > >>>>>>>> (managed or
> > > > > > > > > > > >>>>>>>> non-managed) And, for utilizing the efficient
> > > > backend
> > > > > > > > > > > >>>>>>>> storage
> > > > > > > > > > > capabilities, we can use the new storage snapshots API.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike
> Tutkowski <
> > > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>>
> > > wrote:
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> Whatever we do here, we need to have a plan
> to
> > > deal
> > > > > > > > > > > >>>>>>>>> with the fact that we already have a feature
> > (in
> > > > 4.6
> > > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a volume
> > snapshot
> > > > (for
> > > > > > > > > > > >>>>>>>>> managed
> > > > > > > > > > > >>>>>>>>> storage) that resides on a backend SAN
> (using a
> > > > > custom
> > > > > > > > > > > >>>>>>>>> snapshot strategy and a custom data motion
> > > > strategy).
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> If these new APIs go in, then how should the
> > > > original
> > > > > > > > > > > >>>>>>>>> implementation (present in 4.6 and later) be
> > > > changed?
> > > > > > > > > > > >>>>>>>>> If it
> > > > > > > > > is
> > > > > > > > > > > >>>>>>>>> changed, how do we support customers who are
> > > > already
> > > > > > > > > > > >>>>>>>>> using
> > > > > > > > > the
> > > > > > > > > > > >>>>>>>>> original volume-snapshot API to take
> snapshots
> > > on a
> > > > > > > > > > > >>>>>>>>> backend
> > > > > > > > > > SAN?
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > > > > >>>>>>>>> Mike
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will
> Stevens <
> > > > > > > > > > > >>>>>>>>> wstevens@cloudops.com <javascript:;>
> > <javascript:;>> wrote:
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> Will you be able to create a Template from a
> > > > > > > > > StorageSnapshot?
> > > > > > > > > > > >>>>>>>>>> If yes, will the template be stored in the
> > > > secondary
> > > > > > > > > > > >>>>>>>>>> storage like normal templates or will that
> be
> > > > > handled
> > > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > > > > > vendor side?
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J
> > 1S6
> > > w
> > > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed
> Mushtaq <
> > > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com <javascript:;>
> > <javascript:;>> wrote:
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will
> Stevens
> > <
> > > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com <javascript:;>
> > <javascript:;>> wrote:
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the
> > > Jira
> > > > > > > > > > > >>>>>>>>>>>> ticket because it was not clear in the
> > > 'mention'
> > > > > > > > > > > >>>>>>>>>>>> section because it shows as a page 'you do
> > not
> > > > > have
> > > > > > > > permission to'.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|*
> H3J
> > > 1S6
> > > > w
> > > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com <javascript:;>
> > <javascript:;>
> > > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to
> do
> > > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage backends
> like
> > > > > > SolidFire.
> > > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed storage stay
> > on
> > > > the
> > > > > > > > > > > >>>>>>>>>>>>> device which is contrary to what
> CloudStack
> > > > calls
> > > > > > > > snpshots.
> > > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage and
> keeping
> > > it
> > > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > > > > > > and
> > > > > > > > > > > >>>>>>>>>>>>> we would ideally like to have both ways
> of
> > > > doing
> > > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs
> to
> > > > > create
> > > > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> What do you guys think of this feature? I
> > > would
> > > > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am working
> on
> > > > > making
> > > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > > > > spec
> > > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a high
> > level
> > > > > > > > > > > >>>>>>>>>>>>> feedback first before starting to work on
> > it.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> --
> > > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> <javascript:;>
> > <javascript:;>
> > > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > > > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > >>>>>>>>> <
> > > > http://solidfire.com/solution/overview/?video=play
> > > > > >*™
> > > > > > > > > > > >>>>>>>>> *
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> --
> > > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>
> > > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > > > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > >>>>>> <
> > http://solidfire.com/solution/overview/?video=play
> > > > >*™*
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> --
> > > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>
> > > > > > > > > > > >>>>> o: 303.746.7302
> > > > > > > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > > > > > > >>>>> <
> > http://solidfire.com/solution/overview/?video=play
> > > > >*™*
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> --
> > > > > > > > > > > >>> *Mike Tutkowski*
> > > > > > > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > >>> e: mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>
> > > > > > > > > > > >>> o: 303.746.7302
> > > > > > > > > > > >>> Advancing the way the world uses the cloud
> > > > > > > > > > > >>> <
> http://solidfire.com/solution/overview/?video=play
> > > >*™*
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>
> > > > > > > > > > > > o: 303.746.7302
> > > > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > > > <http://solidfire.com/solution/overview/?video=play
> > >*™*
> > > > > > > > > > > >
> > > > > > > > > > > Find out more about ShapeBlue and our range of
> CloudStack
> > > > > > > > > > > related
> > > > > > > > > > services:
> > > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > > > CSForge –
> > > > > > > > > > > rapid IaaS deployment framework <
> > > > http://shapeblue.com/csforge/
> > > > > >
> > > > > > > > > > > CloudStack Consulting
> > > > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > > > > > > CloudStack
> > > > > > > > > > > Software Engineering
> > > > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/
> >
> > > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > > > <
> http://shapeblue.com/cloudstack-infrastructure-support/
> > >
> > > |
> > > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Mike Tutkowski*
> > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > <javascript:;>
> > > > > > > > > o: 303.746.7302
> > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > > >
> > > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > related
> > > > > > > services:
> > > > > > > > IaaS Cloud Design & Build
> > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > CSForge –
> > > > > rapid
> > > > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > CloudStack
> > > > > > > > Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > CloudStack
> > > > > > > > Software Engineering
> > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > CloudStack Infrastructure Support
> > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > > CloudStack
> > > > > > > > Bootcamp Training Courses <
> > > > http://shapeblue.com/cloudstack-training/
> > > > > >
> > > > > > > >
> > > > > > > Find out more about ShapeBlue and our range of CloudStack
> related
> > > > > > services:
> > > > > > > IaaS Cloud Design & Build
> > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> CSForge –
> > > > rapid
> > > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > > > CloudStack Consulting <
> > > http://shapeblue.com/cloudstack-consultancy/>
> > > > |
> > > > > > CloudStack
> > > > > > > Software Engineering
> > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > CloudStack Infrastructure Support
> > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > CloudStack
> > > > > > > Bootcamp Training Courses <
> > > http://shapeblue.com/cloudstack-training/
> > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com <javascript:;>
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>
>
> --
> Ian Rae
> CEO | PDG
> c: 514.944.4008
>
> CloudOps | Cloud Infrastructure and Networking Solutions
> www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
>

Re: [Propose][New Feature] Storage Snapshots

[Ian Rae <ir...@cloudops.com>]

I think a service provider backup scenario is more likely to take advantage
of SAN snapshot. There are a few reasons for this. Traditional backups
involve access to the file system, and there is an expectation that this
can be done with reasonably short time frames without negatively impacting
VM performance, and that the backup orchestrator can apply various logic
and or transformations to the data (compress, encrypt, deltas etc...).
While it is true that one could apply a backup process to a cloud snapshot,
this would be slow and inefficient requiring the data to be moved several
times and there are some major bottlenecks with cloud snapshots. With a
cloud snapshot - there seems to be no reasonable expectation of being able
to do differential snapshots (I think this depends on the hypervisor) and
if you do differential snapshots this will make file backups from those
snapshots even more complicated to orchestrate.

Suspect there needs to be a different thread on how to better enable
backups, as a service. As per Paul's suggestion, but it is a related
workflow so relevant to this discussion.

Ian

On Monday, February 8, 2016, Mike Tutkowski <mi...@solidfire.com>
wrote:

> To me it sounds like number two and number three are different uses for the
> same "thing"(which is totally fine).
>
> As for taking a fast SAN snapshot and exporting it asynchronously, do we
> see the SSVM as performing the export?
>
> To be backwards compatible with what we have in 4.6 and later for volume
> snapshots for managed storage, I think it might be easier if we pass in a
> flag that says whether or not to archive the SAN snapshot (which, I think,
> is something that you suggested, as well, Pierre-Luc).
>
> On Monday, February 8, 2016, Pierre-Luc Dion <pdion@cloudops.com
> <javascript:;>> wrote:
>
> > Hi Mike,
> >
> > The reason behind the creation of a SAN snapshot which is exported into
> > secondary storage, is because creating a copy of the .VHD directly would
> > impact uptime of the VM as creating that copy take lots of time. Has
> oppose
> > to a SAN snapshot that is close to instantaneous which can afterward be
> > clone into Secondary Storage asynchronously.
> >
> > I would suspect an extracted VolumeSnapshot taken from a SAN snapshot
> could
> > have is SAN snapshot deleted to avoid duplica and space consumption on
> the
> > Primary Storage side.
> >
> >
> > I see 3 definitions in our current discussion regarding the term snapshot
> > (these are not official terminology but by own interpretation of them):
> >
> > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a snapshot):
> > it's a volume snapshot at the storage level, point in time of your data.
> it
> > reside on the primary storage. Useful and efficient for software side
> > incident.
> > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud backup aws-S3
> > style ): Point in time copy of the Virtual Disk that reside on a
> different
> > storage array then the original Volume. Facilitate data migration between
> > clusters and, in case of primary storage incident, Volume snapshots are
> not
> > impacted and can be reuse.
> > 3. *Backup*: Archival of your Virtual-machines data that also validate
> data
> > integrity, provide a storage efficient archiving method and an
> independent
> > way to restore your data in case of an major infrastructure disaster.
> >
> >
> > Regards,
> >
> > PL
> >
> >
> > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > wrote:
> >
> > > So, let's see if I currently follow the requirements:
> > >
> > > * Augment volume snapshots for managed storage to conditionally export
> > data
> > > to NFS. The current process of taking a snapshot on the SAN is fine,
> but
> > > we'd like the option to export the data to NFS, as well.
> > >
> > > Questions:
> > >
> > > Once the data has been exported to NFS, do we keep the SAN snapshot or
> > > delete it?
> > >
> > > If we are deleting the SAN snapshot, then why don't we just copy the
> VHD
> > > from primary to secondary the way we do today for non-managed (i.e.
> > > traditional) storage? Why create a SAN snapshot in that scenario?
> Perhaps
> > > to have the SSVM mount and perform the VHD copy to secondary storage
> > > instead of a XenServer host?
> > >
> > > Thanks for the clarification.
> > >
> > > By the way, to me a backup is when you copy data from one storage
> system
> > to
> > > another (regardless of features, if any, to restore the data in the
> > > future). A snapshot is a point-in-time view of the data of a volume and
> > > it's stored on the same storage system as the volume.
> > >
> > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <pdion@cloudops.com
> <javascript:;>
> > <javascript:;>>
> > > wrote:
> > >
> > > > That's fun to see that discussion happening. I 100% agree with Paul's
> > > > points of view. VolumeSnapshot are not a backup, but I do consider
> them
> > > as
> > > > a safety vest against Primary Storage failure, because failure append
> > > :-( .
> > > >
> > > > The current proposal around snapshots that reside on the primary
> > storage
> > > or
> > > > snapshots that end in the Secondary Storage  is not to address any
> kind
> > > of
> > > > backups requirement because a snapshot is not a backup, event an
> > > extracted
> > > > VM snapshot.
> > > >
> > > > The main idea, and again this is for managed storage;
> > > >
> > > > 1. StorageSnapshotAPI: Provide storage side snapshot capability for
> > fast
> > > > response time that support rollback to previous timestamp, create new
> > > > volume and maybe create template.
> > > >     not required to be a new API if the work is already done, I think
> > > this
> > > > is a different behaviors than the user expectation of a
> > volume-snapshot.
> > > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior that create
> > an
> > > > extraction of a volume into SecondaryStorage which can be reuse to
> > > create a
> > > > new volume into another Primary Storage. This type of snapshot is a
> > slow
> > > > job since yes it would have to copy the full volume size on the
> > Secondary
> > > > storage.
> > > >
> > > >
> > > > PL
> > > >
> > > >
> > > >
> > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> syed1.mushtaq@gmail.com <javascript:;>
> > <javascript:;>>
> > > > wrote:
> > > >
> > > > > I think I share you view on the 'Ideal world'. Backup (via Volume
> > > > > Snapshots) is a huge bottleneck in Cloudstack. This is amplified
> > > > especially
> > > > > when you have a object storage as your secondary storage because it
> > > > > requires two copies (one to an NFS staging area and from there to
> > > object
> > > > > storage). And not to mention that all these copies are consuming
> > > > hypervisor
> > > > > resources. Xenserver's Dom0 is also a huge bottleneck as all the
> > > Network
> > > > > and I/O flow through it. So our intention of proposing the "Storage
> > > > > Snapshots" is to give a better way of achiving snapshots while
> still
> > > > > keeping the original definition of volume snpashots (ie upload to
> sec
> > > > > storage).
> > > > >
> > > > > But as Erik pointed out volume snapshots are not backups. They
> don't
> > > work
> > > > > form multi-disk LVM volume groups and dynamic disks. I am all in
> for
> > a
> > > > > better backup solution which handles these use cases and utilizes
> the
> > > > > storage's advanced features.
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > In the beginning... there were CloudStack snapshots and they were
> > > > > actually
> > > > > > volume snapshots not hypervisor point-in-time snapshots.
> > > > > > Then VM snapshots were created (which are point-in-time
> hypervisor
> > > > > > snapshots) and we started referring to the original snapshots as
> > > volume
> > > > > > snapshots.
> > > > > >
> > > > > > CloudStack does not offer 'backups', but many people use volume
> > > > snapshots
> > > > > > as backups. However you can't in-place restore volume snapshots
> and
> > > if
> > > > > you
> > > > > > have a VM with multiple volumes, the volume snapshots must be
> done
> > in
> > > > > > series, meaning that the state across all of the volumes is
> > unlikely
> > > to
> > > > > be
> > > > > > consistent.
> > > > > >
> > > > > > 'Actual Backups' would enable all of the restore options which
> > users
> > > > > might
> > > > > > expect as well options as to where they might be stored. In my
> > ideal
> > > > > world
> > > > > > they would also be able to leverage back-end hardware (such as
> > > > Solidfire,
> > > > > > NetApp etc :) ) and software such as Veeam, Commvault etc to
> > > accelerate
> > > > > the
> > > > > > process.
> > > > > >
> > > > > >
> > > > > >
> > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > > Paul Angus
> > > > > > VP Technology ,  ShapeBlue
> > > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |
> m:
> > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > e:  *paul.angus@shapeblue.com <javascript:;> <javascript:;> |
> t: @cloudyangus*
> > > > > > <paul.angus@shapeblue.com <javascript:;>
> <ja...@cloudyangus>
> > |  w:
> > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > > Shape Blue Ltd is a company incorporated in England & Wales.
> > > ShapeBlue
> > > > > > Services India LLP is a company incorporated in India and is
> > operated
> > > > > under
> > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> is
> > a
> > > > > > company incorporated in Brasil and is operated under license from
> > > Shape
> > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > > Republic
> > > > of
> > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > ShapeBlue
> > > > > is
> > > > > > a registered trademark.
> > > > > > This email and any attachments to it may be confidential and are
> > > > intended
> > > > > > solely for the use of the individual to whom it is addressed. Any
> > > views
> > > > > or
> > > > > > opinions expressed are solely those of the author and do not
> > > > necessarily
> > > > > > represent those of Shape Blue Ltd or related companies. If you
> are
> > > not
> > > > > the
> > > > > > intended recipient of this email, you must neither take any
> action
> > > > based
> > > > > > upon its contents, nor copy or show it to anyone. Please contact
> > the
> > > > > sender
> > > > > > if you believe you have received this email in error.
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> <javascript:;> <javascript:;>]
> > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > > To: dev@cloudstack.apache.org <javascript:;> <javascript:;>
> > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > >
> > > > > > Paul,
> > > > > >
> > > > > > When you say actual backups, how would it be different from the
> > > Volume
> > > > > > Snapshots that exist currently. My understanding is that Backups
> > end
> > > up
> > > > > in
> > > > > > Sec Storage whereas Snapshots are just a point-in-time state of
> > your
> > > > > volume
> > > > > > which can be restored back correct?
> > > > > >
> > > > > > -Syed
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > > paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Syed,
> > > > > > >
> > > > > > > As I understand it, the SolidFire plugin will export the
> snapshot
> > > to
> > > > > > > secondary storage if the user requests a template from the
> > snapshot
> > > > or
> > > > > > > wants to download the snapshot from the cloud. This is a good,
> > > > > > > pragmatic approach and yes Mike the SolidFire storage is super
> > > > > > > reliable and snapshots on SolidFire arrays take up next to no
> > > space.
> > > > > > > BUT I think that we are talking about a more general purpose
> API,
> > > and
> > > > > > > other storage systems may not be as awesome as Mike's. That's
> my
> > > > > > > concern. Also, the time to transfer for say 1TB to move from
> > > primary
> > > > > > > to sec storage and then create a VM template out of it may be
> too
> > > > long
> > > > > > for users.
> > > > > > >
> > > > > > > @Mike I don’t think 'we' use the term volume snapshot for
> backup,
> > > > it's
> > > > > > > just that users want to do backups and a volume snapshot is the
> > > only
> > > > > > > type of snapshot that copies the disk elsewhere and can be
> > > scheduled.
> > > > > > >
> > > > > > > I'm 'pondering' the implications of enabling actual backups
> > > (through
> > > > > > > recognised backup providers) and the user requirements around
> > them
> > > > > > > (particularly restoration use cases) as a separate thread of
> > work.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > Technology
> > > > > > > , ShapeBlue
> > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> |
> m:
> > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > e: *paul.angus@shapeblue.com <javascript:;> <javascript:;> |
> t: @cloudyangus*
> > > > > > > <paul.angus@shapeblue.com <javascript:;> <javascript:;>
> > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape
> Blue
> > > Ltd
> > > > > > > is a company incorporated in England & Wales. ShapeBlue
> Services
> > > > India
> > > > > > > LLP is a company incorporated in India and is operated under
> > > license
> > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > > company
> > > > > > > incorporated in Brasil and is operated under license from Shape
> > > Blue
> > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> Republic
> > > of
> > > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > > > ShapeBlue is a registered trademark.
> > > > > > > This email and any attachments to it may be confidential and
> are
> > > > > > > intended solely for the use of the individual to whom it is
> > > > addressed.
> > > > > > > Any views or opinions expressed are solely those of the author
> > and
> > > do
> > > > > > > not necessarily represent those of Shape Blue Ltd or related
> > > > > > > companies. If you are not the intended recipient of this email,
> > you
> > > > > > > must neither take any action based upon its contents, nor copy
> or
> > > > show
> > > > > > > it to anyone. Please contact the sender if you believe you have
> > > > > received
> > > > > > this email in error.
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> <javascript:;>
> > <javascript:;>]
> > > > > > > Sent: 05 February 2016 15:31
> > > > > > > To: dev@cloudstack.apache.org <javascript:;> <javascript:;>
> > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > >
> > > > > > > I think the terminology confusion comes from AWS where they do
> > EBS
> > > > > > > snapshots backed up to S3 and CloudStack sort of followed that.
> > And
> > > > as
> > > > > > > an end user who is oblivious to the internals of my provider,
> my
> > > > > > > expectation would be something similar to what AWS as that is
> my
> > > > > > > biggest reference point.
> > > > > > >
> > > > > > > To your point Mike, I agree that a Primary Storage failure on
> > > > > > > SolidFire is unlikely, there are other motivations for us to
> push
> > > > data
> > > > > > > to secondary storage. Primary storage (atleast for us) costs
> > > around 3
> > > > > > > times as much as secondary storage and snapshots on primary
> > storage
> > > > > > > are rarely used (especially for some of our customers who do
> > daily
> > > > > > backups).
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > > > > > mike.tutkowski@solidfire.com <javascript:;> <javascript:;>>
> wrote:
> > > > > > >
> > > > > > > > Some of the weirdness is around terminology.
> > > > > > > >
> > > > > > > > For most systems I've worked on, a snapshot and a backup are
> > two
> > > > > > > > completely different things (but CloudStack has traditionally
> > > used
> > > > > > > > the term "volume snapshot" to mean backup).
> > > > > > > >
> > > > > > > > I will put in a SolidFire "plug" here and say, though, that
> if
> > > your
> > > > > > > > primary storage is running on SolidFire that it is unlikely
> > > you'll
> > > > > > > > encounter an issue where your primary storage goes offline
> (and
> > > > > > > > you'll even maintain your performance guarantees during
> failure
> > > > > > > > scenarios and upgrades, as well). That being the case, it is
> > less
> > > > > > > > useful to require a backup to Swift (but it's perfectly OK if
> > > > that's
> > > > > > > > what we want to do
> > > > > > > here).
> > > > > > > >
> > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > > > > > <syed1.mushtaq@gmail.com <javascript:;> <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Paul,
> > > > > > > > >
> > > > > > > > > I believe with the current implementation of Snapshots on
> > > managed
> > > > > > > > > storage
> > > > > > > > > (SolidFire) the snapshots are never exported to the
> secondary
> > > > > > storage.
> > > > > > > > > While this solves the problem of having snapshots taking
> > > forever
> > > > > > > > > to get to sec storage, this leaves us with a
> > > > > > > > huge
> > > > > > > > > liability if our primary storage goes down. We see
> snapshots
> > as
> > > > > > > > > our recovery path because we store them in Swift which is
> > > > reliable
> > > > > > > > > and resilient to failures.
> > > > > > > > >
> > > > > > > > > With Storage snpashots our goal is to have Volume snapshots
> > > > always
> > > > > > > > > backed up to secondary storage and Storage Snapshots stay
> on
> > > the
> > > > > > > > > primary
> > > > > > > > storage.
> > > > > > > > > A provider could potentially mix both these and solve the
> > > problem
> > > > > > > > > that you mentioned where you want to meet user's
> expectation
> > > of a
> > > > > > > > > snapshot (ie backup to sec storage) while having an ability
> > to
> > > > > > > > > utilize faster sanpshots (i.e. on the device)
> > > > > > > > >
> > > > > > > > > Hope this clarifies things.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > -Syed
> > > > > > > > >
> > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > > > > > <paul.angus@shapeblue.com <javascript:;> <javascript:;>>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > HI guys,
> > > > > > > > > >
> > > > > > > > > > Could someone point me to the Jira bug of FS for the
> > > > > > > > > > SAN-snapshot
> > > > > > > > feature
> > > > > > > > > > in 4.6 which is mentioned.
> > > > > > > > > >
> > > > > > > > > > From my discussions with users and operators around
> > snapshots
> > > > > > > > > > I'd make
> > > > > > > > > the
> > > > > > > > > > following observations:
> > > > > > > > > > a. 'users' use snapshots as backups (both long-term and
> > short
> > > > > > > > > > term)
> > > > > > > > with
> > > > > > > > > > the expectation that they can use them for recovery if
> > > > required.
> > > > > > > > > > b. operators fall back to snapshots if something has gone
> > > wrong
> > > > > > > > > > with primary storage.
> > > > > > > > > > c. users sometimes want to be able to export snapshots as
> > > well
> > > > > > > > > > as
> > > > > > > > create
> > > > > > > > > > new VMs from their snapshots
> > > > > > > > > > d. snapshots are a currently a massive pain for
> operators,
> > I
> > > > > > > > > > know at
> > > > > > > > > least
> > > > > > > > > > one public cloud who have snapshots which take 2 days to
> > > > > complete.
> > > > > > > > > > e. snapshots (as they are) can't be used for multiple LVM
> > > > disks.
> > > > > > > > > >
> > > > > > > > > > I think the process Mike has used in the SolidFire plugin
> > > (only
> > > > > > > > > > moving
> > > > > > > > > the
> > > > > > > > > > disk image to secondary storage when you absolutely have
> > to)
> > > is
> > > > > > > > > > a very
> > > > > > > > > good
> > > > > > > > > > and pragmatic solution. I wonder what problems an
> operator
> > > > might
> > > > > > > > > experience
> > > > > > > > > > if they have an issue with a given primary storage pool
> in
> > a
> > > > > > cluster.
> > > > > > > > (I
> > > > > > > > > > know that that is REALLY unlikely in the SolidFire case
> > Mike
> > > :)
> > > > > > > > > > ) And
> > > > > > > > if
> > > > > > > > > > the transfer from primary to secondary is slow, the time
> to
> > > > > > > > > > being able
> > > > > > > > to
> > > > > > > > > > create a template or export the volume will be slow.
> > > > > > > > > >
> > > > > > > > > > So for me the issue is around making sure that the end
> > users
> > > > > > > > expectations
> > > > > > > > > > are met (while improving the speed/efficiency of the back
> > > end)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus
> > VP
> > > > > > > > > > Technology , ShapeBlue
> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > > >
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > > |
> > > > m:
> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;>
> <javascript:;> | t:
> > @cloudyangus*
> > > > > > > > > > <paul.angus@shapeblue.com <javascript:;> <javascript:;>
> > %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> Shape
> > > > Blue
> > > > > > > > > > Ltd is a company incorporated in England & Wales.
> ShapeBlue
> > > > > > > > > > Services India LLP is a company incorporated in India and
> > is
> > > > > > > > > > operated
> > > > > > > > > under
> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> Consultoria
> > > Ltda
> > > > > > > > > > is a company incorporated in Brasil and is operated under
> > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a
> > > company
> > > > > > > > > > registered by The Republic
> > > > > > > > of
> > > > > > > > > > South Africa and is traded under license from Shape Blue
> > Ltd.
> > > > > > > > > > ShapeBlue
> > > > > > > > > is
> > > > > > > > > > a registered trademark.
> > > > > > > > > > This email and any attachments to it may be confidential
> > and
> > > > are
> > > > > > > > intended
> > > > > > > > > > solely for the use of the individual to whom it is
> > addressed.
> > > > > > > > > > Any views
> > > > > > > > > or
> > > > > > > > > > opinions expressed are solely those of the author and do
> > not
> > > > > > > > necessarily
> > > > > > > > > > represent those of Shape Blue Ltd or related companies.
> If
> > > you
> > > > > > > > > > are not
> > > > > > > > > the
> > > > > > > > > > intended recipient of this email, you must neither take
> any
> > > > > > > > > > action
> > > > > > > > based
> > > > > > > > > > upon its contents, nor copy or show it to anyone. Please
> > > > contact
> > > > > > > > > > the
> > > > > > > > > sender
> > > > > > > > > > if you believe you have received this email in error.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com
> <javascript:;>
> > <javascript:;>]
> > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> <javascript:;>
> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > > >
> > > > > > > > > > Hi Mike,
> > > > > > > > > >
> > > > > > > > > > The idea of introducing a new API: StorageSnapshot for
> > > managed
> > > > > > > > > > storage
> > > > > > > > is
> > > > > > > > > > because the VolumeSnapshot default, or expected, behavior
> > is
> > > to
> > > > > > > > > > archive snapshots into the Secondary Storage. So a
> > > > > > > > > > StorageSnapshot API would be
> > > > > > > > > for
> > > > > > > > > > snapshot that remain on the managed storage appliance.
> > > > > > > > > >
> > > > > > > > > > Quickly looking at the API doc and I don't see a strong
> > > > > > > > > > requirement for volume snapshots to be moved into
> secondary
> > > > > > > > > > storage. So, maybe StorageSnapshot API is not useful, but
> > > both
> > > > > > > > > > use cases are required. A snapshot that remain on the
> > managed
> > > > > > > > > > storage, and another type of
> > > > > > > > snapshot
> > > > > > > > > > that end up into the secondary storage. Since you've
> done a
> > > lot
> > > > > > > > > > of
> > > > > > > > work,
> > > > > > > > > > might easier to just add a parameter to the current
> > snapshot
> > > > API
> > > > > > > > > > that
> > > > > > > > > would
> > > > > > > > > > trigger an extraction of the storage snapshot into the
> > > > secondary
> > > > > > > > storage?
> > > > > > > > > >
> > > > > > > > > > PL
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I think that all sounds reasonable then - thanks!
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > > > > > syed1.mushtaq@gmail.com <javascript:;> <javascript:;>>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> You are correct Mike in terms of the requirements. One
> > of
> > > > our
> > > > > > > > earlier
> > > > > > > > > > >> iterations on this was to have an argument to the
> create
> > > > > > > > > > >> snapshot
> > > > > > > > API
> > > > > > > > > > >> which decides whether to backup the volume to sec
> > storage
> > > > but
> > > > > > > > > > >> we realized it would make management of snapshots
> quite
> > > > messy
> > > > > > > > > > >> so we proposed a new api instead.
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > > > > > >> <mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Hi,
> > > > > > > > > > >>>
> > > > > > > > > > >>> Just to make sure I understand all the requirements
> > here:
> > > > > > > > > > >>>
> > > > > > > > > > >>> 1) This relates only to managed storage (1:1 mapping
> > > > between
> > > > > > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > > > > > >>>
> > > > > > > > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > > > > > > > >>> functionality, which creates a snapshot on the SAN,
> and
> > > > > > > > > > >>> extend it via a config option (or
> > > > > > > > > > >>> something) to not only take the SAN snapshot, but to
> > copy
> > > > > > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > > > > > >>>
> > > > > > > > > > >>> 3) The SAN snapshot is always taken. It's the backup
> to
> > > NFS
> > > > > > > > > > >>> that is optional.
> > > > > > > > > > >>>
> > > > > > > > > > >>> 4) Templates can be created from the snapshot that's
> on
> > > the
> > > > > > > > > > >>> SAN (already works).
> > > > > > > > > > >>>
> > > > > > > > > > >>> 5) CloudStack volumes can be created from the
> snapshot
> > > > > > > > > > >>> that's on
> > > > > > > > the
> > > > > > > > > > >>> SAN (already works as long as the new CloudStack
> volume
> > > > ends
> > > > > > > > > > >>> up on the same primary storage).
> > > > > > > > > > >>>
> > > > > > > > > > >>> Would we have a need for a storage snapshot API then
> or
> > > > > > > > > > >>> would that just be the standard volume snapshot
> without
> > > the
> > > > > > > > > > >>> backup to
> > > > > > > NFS?
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks!
> > > > > > > > > > >>> Mike
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>
> <javascript:;>>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Is it possible to have both functionalities
> (snapshot
> > on
> > > > > > > > > > >>>> SAN & Sec
> > > > > > > > > > >>>> Storage) coexist? Because Ideally, we would like to
> > have
> > > > > both.
> > > > > > > > > > >>>> For example, some of our customers want to implement
> > > their
> > > > > > > > > > >>>> own backup strategies and do encryption to their
> > backups
> > > > > > > > > > >>>> which is a perfect use case for Storage Snapshot
> while
> > > our
> > > > > > > > > > >>>> other customers will still keep using the standard
> > > volume
> > > > > > > snapshot.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> To keep things backward compatible, we can add a
> > setting
> > > > > > > > > > >>>> which
> > > > > > > > says
> > > > > > > > > > >>>> to not upload on secondary storage, because, after
> > all,
> > > > you
> > > > > > > > > > >>>> would take a SAN snapshot first when doing a Volume
> > > > > Snapshot.
> > > > > > > > > > >>>> You could stop the process there and not do the
> > upload.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> What do you think about this approach?
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Thanks,
> > > > > > > > > > >>>> -Syed
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > > > > > > >>>> mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>> wrote:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> So, this is just me thinking out load here, but if
> a
> > > > given
> > > > > > > > > > >>>>> CloudStack cloud doesn't actually need to provide
> > both
> > > > the
> > > > > > > > ability
> > > > > > > > > > >>>>> to take a SAN snapshot and export it to NFS (if
> just
> > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we might be able
> > to
> > > > get
> > > > > > > > > > >>>>> away with no new
> > > > > > > > API
> > > > > > > > > > >>>>> calls and simply implement a new custom snapshot
> > > strategy
> > > > > > > > > > >>>>> and
> > > > > > > > data
> > > > > > > > > > >>>>> motion strategy to handle the case where the
> > CloudStack
> > > > > > > > > > >>>>> cloud
> > > > > > > > does
> > > > > > > > > > >>>>> want both a SAN snapshot and exported-to-NFS
> backup.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> In other words, the "default" behavior would be to
> > use
> > > > the
> > > > > > > > > > >>>>> snapshot strategy and data motion strategy that we
> > > > already
> > > > > > > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> If your CloudStack cloud, however, wants to take a
> > SAN
> > > > > > > > > > >>>>> snapshot and have the data exported to NFS, then we
> > > could
> > > > > > > > > > >>>>> have you manipulate a Swing config file to make use
> > of
> > > a
> > > > > > > > > > >>>>> new snapshot strategy and data motion strategy that
> > > > > > > > > > >>>>> performs both of these
> > > > > > > > > > activities.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> This way, the old behavior is still the default for
> > > > users,
> > > > > > > > > > >>>>> but CloudStack admins can change this behavior via
> > > > > > > configuration.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Thoughts?
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > > > > > > > >>>>> mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>> wrote:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> Right...I think we will need to come up with a
> > viable
> > > > > > > > > > >>>>>> upgrade path or some reasonable way for them to
> move
> > > > from
> > > > > > > > > > >>>>>> the old way to the new way (and some obvious way
> > that
> > > > > > > > > > >>>>>> they will know they need
> > > > > > > > to
> > > > > > > > > > do this).
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>
> <javascript:;>> wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> I'm not really sure about the upgrade path
> however,
> > > > > > > > > > >>>>>>> customers who are using 4.6 and are on a managed
> > > > storage
> > > > > > > > > > >>>>>>> would no longer have the same functionality with
> > > Volume
> > > > > > > Snapshots.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com <javascript:;>
> <javascript:;>> wrote:
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> So if I understand correctly, currently taking a
> > > > Volume
> > > > > > > > > > >>>>>>>> Snapshots of a volume on a managed storage keeps
> > it
> > > on
> > > > > > > > > > >>>>>>>> the storage array. As a part of this feature, we
> > can
> > > > > > > > > > >>>>>>>> make sure
> > > > > > > > that
> > > > > > > > > > >>>>>>>> Volume Snapshots on managed storage are uploaded
> > to
> > > > the
> > > > > > > > > > >>>>>>>> secondary storage. This would make the Volume
> > > Snapshot
> > > > > > > > > > >>>>>>>> feature behave the same regardless of the
> storage
> > > > > > > > > > >>>>>>>> (managed or
> > > > > > > > > > >>>>>>>> non-managed) And, for utilizing the efficient
> > > backend
> > > > > > > > > > >>>>>>>> storage
> > > > > > > > > > capabilities, we can use the new storage snapshots API.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>>
> > wrote:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Whatever we do here, we need to have a plan to
> > deal
> > > > > > > > > > >>>>>>>>> with the fact that we already have a feature
> (in
> > > 4.6
> > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a volume
> snapshot
> > > (for
> > > > > > > > > > >>>>>>>>> managed
> > > > > > > > > > >>>>>>>>> storage) that resides on a backend SAN (using a
> > > > custom
> > > > > > > > > > >>>>>>>>> snapshot strategy and a custom data motion
> > > strategy).
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> If these new APIs go in, then how should the
> > > original
> > > > > > > > > > >>>>>>>>> implementation (present in 4.6 and later) be
> > > changed?
> > > > > > > > > > >>>>>>>>> If it
> > > > > > > > is
> > > > > > > > > > >>>>>>>>> changed, how do we support customers who are
> > > already
> > > > > > > > > > >>>>>>>>> using
> > > > > > > > the
> > > > > > > > > > >>>>>>>>> original volume-snapshot API to take snapshots
> > on a
> > > > > > > > > > >>>>>>>>> backend
> > > > > > > > > SAN?
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > > > >>>>>>>>> Mike
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > > > > > > > > >>>>>>>>> wstevens@cloudops.com <javascript:;>
> <javascript:;>> wrote:
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Will you be able to create a Template from a
> > > > > > > > StorageSnapshot?
> > > > > > > > > > >>>>>>>>>> If yes, will the template be stored in the
> > > secondary
> > > > > > > > > > >>>>>>>>>> storage like normal templates or will that be
> > > > handled
> > > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > > > > vendor side?
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J
> 1S6
> > w
> > > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com <javascript:;>
> <javascript:;>> wrote:
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens
> <
> > > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com <javascript:;>
> <javascript:;>> wrote:
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the
> > Jira
> > > > > > > > > > >>>>>>>>>>>> ticket because it was not clear in the
> > 'mention'
> > > > > > > > > > >>>>>>>>>>>> section because it shows as a page 'you do
> not
> > > > have
> > > > > > > permission to'.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J
> > 1S6
> > > w
> > > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com <javascript:;>
> <javascript:;>
> > > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do
> > > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage backends like
> > > > > SolidFire.
> > > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed storage stay
> on
> > > the
> > > > > > > > > > >>>>>>>>>>>>> device which is contrary to what CloudStack
> > > calls
> > > > > > > snpshots.
> > > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage and keeping
> > it
> > > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > > > > > and
> > > > > > > > > > >>>>>>>>>>>>> we would ideally like to have both ways of
> > > doing
> > > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to
> > > > create
> > > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> What do you guys think of this feature? I
> > would
> > > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am working on
> > > > making
> > > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > > > spec
> > > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a high
> level
> > > > > > > > > > >>>>>>>>>>>>> feedback first before starting to work on
> it.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> --
> > > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> > > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > > > > > > > >>>>>>>>> <
> > > http://solidfire.com/solution/overview/?video=play
> > > > >*™
> > > > > > > > > > >>>>>>>>> *
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> --
> > > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> > > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > > > > > > >>>>>> <
> http://solidfire.com/solution/overview/?video=play
> > > >*™*
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> --
> > > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> > > > > > > > > > >>>>> o: 303.746.7302
> > > > > > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > > > > > >>>>> <
> http://solidfire.com/solution/overview/?video=play
> > > >*™*
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> --
> > > > > > > > > > >>> *Mike Tutkowski*
> > > > > > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > >>> e: mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> > > > > > > > > > >>> o: 303.746.7302
> > > > > > > > > > >>> Advancing the way the world uses the cloud
> > > > > > > > > > >>> <http://solidfire.com/solution/overview/?video=play
> > >*™*
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> > > > > > > > > > > o: 303.746.7302
> > > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > > <http://solidfire.com/solution/overview/?video=play
> >*™*
> > > > > > > > > > >
> > > > > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > > > > > > > related
> > > > > > > > > services:
> > > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > > CSForge –
> > > > > > > > > > rapid IaaS deployment framework <
> > > http://shapeblue.com/csforge/
> > > > >
> > > > > > > > > > CloudStack Consulting
> > > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > > > > > CloudStack
> > > > > > > > > > Software Engineering
> > > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/
> >
> > |
> > > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Mike Tutkowski*
> > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> <javascript:;>
> > > > > > > > o: 303.746.7302
> > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > >
> > > > > > > Find out more about ShapeBlue and our range of CloudStack
> related
> > > > > > services:
> > > > > > > IaaS Cloud Design & Build
> > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> CSForge –
> > > > rapid
> > > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > CloudStack
> > > > > > > Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> > > > CloudStack
> > > > > > > Software Engineering
> > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > CloudStack Infrastructure Support
> > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > CloudStack
> > > > > > > Bootcamp Training Courses <
> > > http://shapeblue.com/cloudstack-training/
> > > > >
> > > > > > >
> > > > > > Find out more about ShapeBlue and our range of CloudStack related
> > > > > services:
> > > > > > IaaS Cloud Design & Build
> > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > > rapid
> > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > > CloudStack Consulting <
> > http://shapeblue.com/cloudstack-consultancy/>
> > > |
> > > > > CloudStack
> > > > > > Software Engineering
> > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > CloudStack Infrastructure Support
> > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > CloudStack
> > > > > > Bootcamp Training Courses <
> > http://shapeblue.com/cloudstack-training/
> > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com <javascript:;> <javascript:;>
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> >
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com <javascript:;>
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>


-- 
Ian Rae
CEO | PDG
c: 514.944.4008

CloudOps | Cloud Infrastructure and Networking Solutions
www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6

RE: [Propose][New Feature] Storage Snapshots

[Paul Angus <pa...@shapeblue.com>]

Let me see here, I think there are actually more snapshot types in play.

1. VM Snapshot (point-in-time hypervisor based snapshots)
1a. SAN assisted VM snapshots (point-in-time hypervisor snapshot takes place on transparently SAN to avoid performance issue in disk chains)
2. SAN Snapshot (Storage Snapshot) - NEW
3. Volume Snapshot (current old/slow transfer to secstorage)
4. Backup - JUST AN IDEAL.

It sounds to me like we're talking about (vastly) improving the speed of [3] by leveraging the SAN to do the initial snapshot, to give us [2]. But from the users' perspective what are they getting? They get control of their VM back quicker (but we could do that by returning control of the VM back to the user once the initial VM snapshot has been taken in the existing Volume snapshot - type [3]).

Do you have other use cases in mind where the users aren't just trying to do backups?



[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology   ,       ShapeBlue


d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>     |      m:      +44 7711 418784<tel:+44%207711%20418784>

e:      paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus>      |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:imageb7bd2e.png@26e661ab.48ac6e0c]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




-----Original Message-----
From: Mike Tutkowski [mailto:mike.tutkowski@solidfire.com]
Sent: 08 February 2016 14:36
To: dev@cloudstack.apache.org
Subject: Re: [Propose][New Feature] Storage Snapshots

To me it sounds like number two and number three are different uses for the same "thing"(which is totally fine).

As for taking a fast SAN snapshot and exporting it asynchronously, do we see the SSVM as performing the export?

To be backwards compatible with what we have in 4.6 and later for volume snapshots for managed storage, I think it might be easier if we pass in a flag that says whether or not to archive the SAN snapshot (which, I think, is something that you suggested, as well, Pierre-Luc).

On Monday, February 8, 2016, Pierre-Luc Dion <pd...@cloudops.com> wrote:

> Hi Mike,
>
> The reason behind the creation of a SAN snapshot which is exported
> into secondary storage, is because creating a copy of the .VHD
> directly would impact uptime of the VM as creating that copy take lots
> of time. Has oppose to a SAN snapshot that is close to instantaneous
> which can afterward be clone into Secondary Storage asynchronously.
>
> I would suspect an extracted VolumeSnapshot taken from a SAN snapshot
> could have is SAN snapshot deleted to avoid duplica and space
> consumption on the Primary Storage side.
>
>
> I see 3 definitions in our current discussion regarding the term
> snapshot (these are not official terminology but by own interpretation of them):
>
> 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a snapshot):
> it's a volume snapshot at the storage level, point in time of your
> data. it reside on the primary storage. Useful and efficient for
> software side incident.
> 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud backup
> aws-S3 style ): Point in time copy of the Virtual Disk that reside on
> a different storage array then the original Volume. Facilitate data
> migration between clusters and, in case of primary storage incident,
> Volume snapshots are not impacted and can be reuse.
> 3. *Backup*: Archival of your Virtual-machines data that also validate
> data integrity, provide a storage efficient archiving method and an
> independent way to restore your data in case of an major infrastructure disaster.
>
>
> Regards,
>
> PL
>
>
> On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com <javascript:;>
> > wrote:
>
> > So, let's see if I currently follow the requirements:
> >
> > * Augment volume snapshots for managed storage to conditionally
> > export
> data
> > to NFS. The current process of taking a snapshot on the SAN is fine,
> > but we'd like the option to export the data to NFS, as well.
> >
> > Questions:
> >
> > Once the data has been exported to NFS, do we keep the SAN snapshot
> > or delete it?
> >
> > If we are deleting the SAN snapshot, then why don't we just copy the
> > VHD from primary to secondary the way we do today for non-managed (i.e.
> > traditional) storage? Why create a SAN snapshot in that scenario?
> > Perhaps to have the SSVM mount and perform the VHD copy to secondary
> > storage instead of a XenServer host?
> >
> > Thanks for the clarification.
> >
> > By the way, to me a backup is when you copy data from one storage
> > system
> to
> > another (regardless of features, if any, to restore the data in the
> > future). A snapshot is a point-in-time view of the data of a volume
> > and it's stored on the same storage system as the volume.
> >
> > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <pdion@cloudops.com
> <javascript:;>>
> > wrote:
> >
> > > That's fun to see that discussion happening. I 100% agree with
> > > Paul's points of view. VolumeSnapshot are not a backup, but I do
> > > consider them
> > as
> > > a safety vest against Primary Storage failure, because failure
> > > append
> > :-( .
> > >
> > > The current proposal around snapshots that reside on the primary
> storage
> > or
> > > snapshots that end in the Secondary Storage is not to address any
> > > kind
> > of
> > > backups requirement because a snapshot is not a backup, event an
> > extracted
> > > VM snapshot.
> > >
> > > The main idea, and again this is for managed storage;
> > >
> > > 1. StorageSnapshotAPI: Provide storage side snapshot capability
> > > for
> fast
> > > response time that support rollback to previous timestamp, create
> > > new volume and maybe create template.
> > > not required to be a new API if the work is already done, I
> > > think
> > this
> > > is a different behaviors than the user expectation of a
> volume-snapshot.
> > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior that
> > > create
> an
> > > extraction of a volume into SecondaryStorage which can be reuse to
> > create a
> > > new volume into another Primary Storage. This type of snapshot is
> > > a
> slow
> > > job since yes it would have to copy the full volume size on the
> Secondary
> > > storage.
> > >
> > >
> > > PL
> > >
> > >
> > >
> > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq
> > > <syed1.mushtaq@gmail.com
> <javascript:;>>
> > > wrote:
> > >
> > > > I think I share you view on the 'Ideal world'. Backup (via
> > > > Volume
> > > > Snapshots) is a huge bottleneck in Cloudstack. This is amplified
> > > especially
> > > > when you have a object storage as your secondary storage because
> > > > it requires two copies (one to an NFS staging area and from
> > > > there to
> > object
> > > > storage). And not to mention that all these copies are consuming
> > > hypervisor
> > > > resources. Xenserver's Dom0 is also a huge bottleneck as all the
> > Network
> > > > and I/O flow through it. So our intention of proposing the
> > > > "Storage Snapshots" is to give a better way of achiving
> > > > snapshots while still keeping the original definition of volume
> > > > snpashots (ie upload to sec storage).
> > > >
> > > > But as Erik pointed out volume snapshots are not backups. They
> > > > don't
> > work
> > > > form multi-disk LVM volume groups and dynamic disks. I am all in
> > > > for
> a
> > > > better backup solution which handles these use cases and
> > > > utilizes the storage's advanced features.
> > > >
> > > >
> > > >
> > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> paul.angus@shapeblue.com <javascript:;>>
> > > > wrote:
> > > >
> > > > > In the beginning... there were CloudStack snapshots and they
> > > > > were
> > > > actually
> > > > > volume snapshots not hypervisor point-in-time snapshots.
> > > > > Then VM snapshots were created (which are point-in-time
> > > > > hypervisor
> > > > > snapshots) and we started referring to the original snapshots
> > > > > as
> > volume
> > > > > snapshots.
> > > > >
> > > > > CloudStack does not offer 'backups', but many people use
> > > > > volume
> > > snapshots
> > > > > as backups. However you can't in-place restore volume
> > > > > snapshots and
> > if
> > > > you
> > > > > have a VM with multiple volumes, the volume snapshots must be
> > > > > done
> in
> > > > > series, meaning that the state across all of the volumes is
> unlikely
> > to
> > > > be
> > > > > consistent.
> > > > >
> > > > > 'Actual Backups' would enable all of the restore options which
> users
> > > > might
> > > > > expect as well options as to where they might be stored. In my
> ideal
> > > > world
> > > > > they would also be able to leverage back-end hardware (such as
> > > Solidfire,
> > > > > NetApp etc :) ) and software such as Veeam, Commvault etc to
> > accelerate
> > > > the
> > > > > process.
> > > > >
> > > > >
> > > > >
> > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > > Technology , ShapeBlue
> > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > e: *paul.angus@shapeblue.com <javascript:;> | t:
> > > > > @cloudyangus* <paul.angus@shapeblue.com
> > > > > <ja...@cloudyangus>
> | w:
> > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape
> > > > > Blue Ltd is a company incorporated in England & Wales.
> > ShapeBlue
> > > > > Services India LLP is a company incorporated in India and is
> operated
> > > > under
> > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria
> > > > > Ltda is
> a
> > > > > company incorporated in Brasil and is operated under license
> > > > > from
> > Shape
> > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > Republic
> > > of
> > > > > South Africa and is traded under license from Shape Blue Ltd.
> > ShapeBlue
> > > > is
> > > > > a registered trademark.
> > > > > This email and any attachments to it may be confidential and
> > > > > are
> > > intended
> > > > > solely for the use of the individual to whom it is addressed.
> > > > > Any
> > views
> > > > or
> > > > > opinions expressed are solely those of the author and do not
> > > necessarily
> > > > > represent those of Shape Blue Ltd or related companies. If you
> > > > > are
> > not
> > > > the
> > > > > intended recipient of this email, you must neither take any
> > > > > action
> > > based
> > > > > upon its contents, nor copy or show it to anyone. Please
> > > > > contact
> the
> > > > sender
> > > > > if you believe you have received this email in error.
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> > > > > <javascript:;>]
> > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > >
> > > > > Paul,
> > > > >
> > > > > When you say actual backups, how would it be different from
> > > > > the
> > Volume
> > > > > Snapshots that exist currently. My understanding is that
> > > > > Backups
> end
> > up
> > > > in
> > > > > Sec Storage whereas Snapshots are just a point-in-time state
> > > > > of
> your
> > > > volume
> > > > > which can be restored back correct?
> > > > >
> > > > > -Syed
> > > > >
> > > > >
> > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > paul.angus@shapeblue.com <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > Hi Syed,
> > > > > >
> > > > > > As I understand it, the SolidFire plugin will export the
> > > > > > snapshot
> > to
> > > > > > secondary storage if the user requests a template from the
> snapshot
> > > or
> > > > > > wants to download the snapshot from the cloud. This is a
> > > > > > good, pragmatic approach and yes Mike the SolidFire storage
> > > > > > is super reliable and snapshots on SolidFire arrays take up
> > > > > > next to no
> > space.
> > > > > > BUT I think that we are talking about a more general purpose
> > > > > > API,
> > and
> > > > > > other storage systems may not be as awesome as Mike's.
> > > > > > That's my concern. Also, the time to transfer for say 1TB to
> > > > > > move from
> > primary
> > > > > > to sec storage and then create a VM template out of it may
> > > > > > be too
> > > long
> > > > > for users.
> > > > > >
> > > > > > @Mike I don’t think 'we' use the term volume snapshot for
> > > > > > backup,
> > > it's
> > > > > > just that users want to do backups and a volume snapshot is
> > > > > > the
> > only
> > > > > > type of snapshot that copies the disk elsewhere and can be
> > scheduled.
> > > > > >
> > > > > > I'm 'pondering' the implications of enabling actual backups
> > (through
> > > > > > recognised backup providers) and the user requirements
> > > > > > around
> them
> > > > > > (particularly restoration use cases) as a separate thread of
> work.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > Technology
> > > > > > , ShapeBlue
> > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > e: *paul.angus@shapeblue.com <javascript:;> | t:
> > > > > > @cloudyangus* <paul.angus@shapeblue.com <javascript:;>
> %20%7C%20t:%20@cloudyangus> | w:
> > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape
> > > > > > Blue
> > Ltd
> > > > > > is a company incorporated in England & Wales. ShapeBlue
> > > > > > Services
> > > India
> > > > > > LLP is a company incorporated in India and is operated under
> > license
> > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > company
> > > > > > incorporated in Brasil and is operated under license from
> > > > > > Shape
> > Blue
> > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > > > > > Republic
> > of
> > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > > ShapeBlue is a registered trademark.
> > > > > > This email and any attachments to it may be confidential and
> > > > > > are intended solely for the use of the individual to whom it
> > > > > > is
> > > addressed.
> > > > > > Any views or opinions expressed are solely those of the
> > > > > > author
> and
> > do
> > > > > > not necessarily represent those of Shape Blue Ltd or related
> > > > > > companies. If you are not the intended recipient of this
> > > > > > email,
> you
> > > > > > must neither take any action based upon its contents, nor
> > > > > > copy or
> > > show
> > > > > > it to anyone. Please contact the sender if you believe you
> > > > > > have
> > > > received
> > > > > this email in error.
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> <javascript:;>]
> > > > > > Sent: 05 February 2016 15:31
> > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > >
> > > > > > I think the terminology confusion comes from AWS where they
> > > > > > do
> EBS
> > > > > > snapshots backed up to S3 and CloudStack sort of followed that.
> And
> > > as
> > > > > > an end user who is oblivious to the internals of my
> > > > > > provider, my expectation would be something similar to what
> > > > > > AWS as that is my biggest reference point.
> > > > > >
> > > > > > To your point Mike, I agree that a Primary Storage failure
> > > > > > on SolidFire is unlikely, there are other motivations for us
> > > > > > to push
> > > data
> > > > > > to secondary storage. Primary storage (atleast for us) costs
> > around 3
> > > > > > times as much as secondary storage and snapshots on primary
> storage
> > > > > > are rarely used (especially for some of our customers who do
> daily
> > > > > backups).
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > > > > mike.tutkowski@solidfire.com <javascript:;>> wrote:
> > > > > >
> > > > > > > Some of the weirdness is around terminology.
> > > > > > >
> > > > > > > For most systems I've worked on, a snapshot and a backup
> > > > > > > are
> two
> > > > > > > completely different things (but CloudStack has
> > > > > > > traditionally
> > used
> > > > > > > the term "volume snapshot" to mean backup).
> > > > > > >
> > > > > > > I will put in a SolidFire "plug" here and say, though,
> > > > > > > that if
> > your
> > > > > > > primary storage is running on SolidFire that it is
> > > > > > > unlikely
> > you'll
> > > > > > > encounter an issue where your primary storage goes offline
> > > > > > > (and you'll even maintain your performance guarantees
> > > > > > > during failure scenarios and upgrades, as well). That
> > > > > > > being the case, it is
> less
> > > > > > > useful to require a backup to Swift (but it's perfectly OK
> > > > > > > if
> > > that's
> > > > > > > what we want to do
> > > > > > here).
> > > > > > >
> > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > > > > <syed1.mushtaq@gmail.com <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Paul,
> > > > > > > >
> > > > > > > > I believe with the current implementation of Snapshots
> > > > > > > > on
> > managed
> > > > > > > > storage
> > > > > > > > (SolidFire) the snapshots are never exported to the
> > > > > > > > secondary
> > > > > storage.
> > > > > > > > While this solves the problem of having snapshots taking
> > forever
> > > > > > > > to get to sec storage, this leaves us with a
> > > > > > > huge
> > > > > > > > liability if our primary storage goes down. We see
> > > > > > > > snapshots
> as
> > > > > > > > our recovery path because we store them in Swift which
> > > > > > > > is
> > > reliable
> > > > > > > > and resilient to failures.
> > > > > > > >
> > > > > > > > With Storage snpashots our goal is to have Volume
> > > > > > > > snapshots
> > > always
> > > > > > > > backed up to secondary storage and Storage Snapshots
> > > > > > > > stay on
> > the
> > > > > > > > primary
> > > > > > > storage.
> > > > > > > > A provider could potentially mix both these and solve
> > > > > > > > the
> > problem
> > > > > > > > that you mentioned where you want to meet user's
> > > > > > > > expectation
> > of a
> > > > > > > > snapshot (ie backup to sec storage) while having an
> > > > > > > > ability
> to
> > > > > > > > utilize faster sanpshots (i.e. on the device)
> > > > > > > >
> > > > > > > > Hope this clarifies things.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > -Syed
> > > > > > > >
> > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > > > > <paul.angus@shapeblue.com <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > HI guys,
> > > > > > > > >
> > > > > > > > > Could someone point me to the Jira bug of FS for the
> > > > > > > > > SAN-snapshot
> > > > > > > feature
> > > > > > > > > in 4.6 which is mentioned.
> > > > > > > > >
> > > > > > > > > From my discussions with users and operators around
> snapshots
> > > > > > > > > I'd make
> > > > > > > > the
> > > > > > > > > following observations:
> > > > > > > > > a. 'users' use snapshots as backups (both long-term
> > > > > > > > > and
> short
> > > > > > > > > term)
> > > > > > > with
> > > > > > > > > the expectation that they can use them for recovery if
> > > required.
> > > > > > > > > b. operators fall back to snapshots if something has
> > > > > > > > > gone
> > wrong
> > > > > > > > > with primary storage.
> > > > > > > > > c. users sometimes want to be able to export snapshots
> > > > > > > > > as
> > well
> > > > > > > > > as
> > > > > > > create
> > > > > > > > > new VMs from their snapshots d. snapshots are a
> > > > > > > > > currently a massive pain for operators,
> I
> > > > > > > > > know at
> > > > > > > > least
> > > > > > > > > one public cloud who have snapshots which take 2 days
> > > > > > > > > to
> > > > complete.
> > > > > > > > > e. snapshots (as they are) can't be used for multiple
> > > > > > > > > LVM
> > > disks.
> > > > > > > > >
> > > > > > > > > I think the process Mike has used in the SolidFire
> > > > > > > > > plugin
> > (only
> > > > > > > > > moving
> > > > > > > > the
> > > > > > > > > disk image to secondary storage when you absolutely
> > > > > > > > > have
> to)
> > is
> > > > > > > > > a very
> > > > > > > > good
> > > > > > > > > and pragmatic solution. I wonder what problems an
> > > > > > > > > operator
> > > might
> > > > > > > > experience
> > > > > > > > > if they have an issue with a given primary storage
> > > > > > > > > pool in
> a
> > > > > cluster.
> > > > > > > (I
> > > > > > > > > know that that is REALLY unlikely in the SolidFire
> > > > > > > > > case
> Mike
> > :)
> > > > > > > > > ) And
> > > > > > > if
> > > > > > > > > the transfer from primary to secondary is slow, the
> > > > > > > > > time to being able
> > > > > > > to
> > > > > > > > > create a template or export the volume will be slow.
> > > > > > > > >
> > > > > > > > > So for me the issue is around making sure that the end
> users
> > > > > > > expectations
> > > > > > > > > are met (while improving the speed/efficiency of the
> > > > > > > > > back
> > end)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul
> > > > > > > > > Angus
> VP
> > > > > > > > > Technology , ShapeBlue
> > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > |
> > > m:
> > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;> | t:
> @cloudyangus*
> > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > > > > > Shape
> > > Blue
> > > > > > > > > Ltd is a company incorporated in England & Wales.
> > > > > > > > > ShapeBlue Services India LLP is a company incorporated
> > > > > > > > > in India and
> is
> > > > > > > > > operated
> > > > > > > > under
> > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > > > > > > > > Consultoria
> > Ltda
> > > > > > > > > is a company incorporated in Brasil and is operated
> > > > > > > > > under license from Shape Blue Ltd. ShapeBlue SA Pty
> > > > > > > > > Ltd is a
> > company
> > > > > > > > > registered by The Republic
> > > > > > > of
> > > > > > > > > South Africa and is traded under license from Shape
> > > > > > > > > Blue
> Ltd.
> > > > > > > > > ShapeBlue
> > > > > > > > is
> > > > > > > > > a registered trademark.
> > > > > > > > > This email and any attachments to it may be
> > > > > > > > > confidential
> and
> > > are
> > > > > > > intended
> > > > > > > > > solely for the use of the individual to whom it is
> addressed.
> > > > > > > > > Any views
> > > > > > > > or
> > > > > > > > > opinions expressed are solely those of the author and
> > > > > > > > > do
> not
> > > > > > > necessarily
> > > > > > > > > represent those of Shape Blue Ltd or related
> > > > > > > > > companies. If
> > you
> > > > > > > > > are not
> > > > > > > > the
> > > > > > > > > intended recipient of this email, you must neither
> > > > > > > > > take any action
> > > > > > > based
> > > > > > > > > upon its contents, nor copy or show it to anyone.
> > > > > > > > > Please
> > > contact
> > > > > > > > > the
> > > > > > > > sender
> > > > > > > > > if you believe you have received this email in error.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com
> <javascript:;>]
> > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > >
> > > > > > > > > Hi Mike,
> > > > > > > > >
> > > > > > > > > The idea of introducing a new API: StorageSnapshot for
> > managed
> > > > > > > > > storage
> > > > > > > is
> > > > > > > > > because the VolumeSnapshot default, or expected,
> > > > > > > > > behavior
> is
> > to
> > > > > > > > > archive snapshots into the Secondary Storage. So a
> > > > > > > > > StorageSnapshot API would be
> > > > > > > > for
> > > > > > > > > snapshot that remain on the managed storage appliance.
> > > > > > > > >
> > > > > > > > > Quickly looking at the API doc and I don't see a
> > > > > > > > > strong requirement for volume snapshots to be moved
> > > > > > > > > into secondary storage. So, maybe StorageSnapshot API
> > > > > > > > > is not useful, but
> > both
> > > > > > > > > use cases are required. A snapshot that remain on the
> managed
> > > > > > > > > storage, and another type of
> > > > > > > snapshot
> > > > > > > > > that end up into the secondary storage. Since you've
> > > > > > > > > done a
> > lot
> > > > > > > > > of
> > > > > > > work,
> > > > > > > > > might easier to just add a parameter to the current
> snapshot
> > > API
> > > > > > > > > that
> > > > > > > > would
> > > > > > > > > trigger an extraction of the storage snapshot into the
> > > secondary
> > > > > > > storage?
> > > > > > > > >
> > > > > > > > > PL
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I think that all sounds reasonable then - thanks!
> > > > > > > > > >
> > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > > > > syed1.mushtaq@gmail.com <javascript:;>>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> You are correct Mike in terms of the requirements.
> > > > > > > > > >> One
> of
> > > our
> > > > > > > earlier
> > > > > > > > > >> iterations on this was to have an argument to the
> > > > > > > > > >> create snapshot
> > > > > > > API
> > > > > > > > > >> which decides whether to backup the volume to sec
> storage
> > > but
> > > > > > > > > >> we realized it would make management of snapshots
> > > > > > > > > >> quite
> > > messy
> > > > > > > > > >> so we proposed a new api instead.
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > > > > >> <mike.tutkowski@solidfire.com <javascript:;>>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hi,
> > > > > > > > > >>>
> > > > > > > > > >>> Just to make sure I understand all the
> > > > > > > > > >>> requirements
> here:
> > > > > > > > > >>>
> > > > > > > > > >>> 1) This relates only to managed storage (1:1
> > > > > > > > > >>> mapping
> > > between
> > > > > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > > > > >>>
> > > > > > > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > > > > > > >>> functionality, which creates a snapshot on the
> > > > > > > > > >>> SAN, and extend it via a config option (or
> > > > > > > > > >>> something) to not only take the SAN snapshot, but
> > > > > > > > > >>> to
> copy
> > > > > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > > > > >>>
> > > > > > > > > >>> 3) The SAN snapshot is always taken. It's the
> > > > > > > > > >>> backup to
> > NFS
> > > > > > > > > >>> that is optional.
> > > > > > > > > >>>
> > > > > > > > > >>> 4) Templates can be created from the snapshot
> > > > > > > > > >>> that's on
> > the
> > > > > > > > > >>> SAN (already works).
> > > > > > > > > >>>
> > > > > > > > > >>> 5) CloudStack volumes can be created from the
> > > > > > > > > >>> snapshot that's on
> > > > > > > the
> > > > > > > > > >>> SAN (already works as long as the new CloudStack
> > > > > > > > > >>> volume
> > > ends
> > > > > > > > > >>> up on the same primary storage).
> > > > > > > > > >>>
> > > > > > > > > >>> Would we have a need for a storage snapshot API
> > > > > > > > > >>> then or would that just be the standard volume
> > > > > > > > > >>> snapshot without
> > the
> > > > > > > > > >>> backup to
> > > > > > NFS?
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks!
> > > > > > > > > >>> Mike
> > > > > > > > > >>>
> > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Is it possible to have both functionalities
> > > > > > > > > >>>> (snapshot
> on
> > > > > > > > > >>>> SAN & Sec
> > > > > > > > > >>>> Storage) coexist? Because Ideally, we would like
> > > > > > > > > >>>> to
> have
> > > > both.
> > > > > > > > > >>>> For example, some of our customers want to
> > > > > > > > > >>>> implement
> > their
> > > > > > > > > >>>> own backup strategies and do encryption to their
> backups
> > > > > > > > > >>>> which is a perfect use case for Storage Snapshot
> > > > > > > > > >>>> while
> > our
> > > > > > > > > >>>> other customers will still keep using the
> > > > > > > > > >>>> standard
> > volume
> > > > > > snapshot.
> > > > > > > > > >>>>
> > > > > > > > > >>>> To keep things backward compatible, we can add a
> setting
> > > > > > > > > >>>> which
> > > > > > > says
> > > > > > > > > >>>> to not upload on secondary storage, because,
> > > > > > > > > >>>> after
> all,
> > > you
> > > > > > > > > >>>> would take a SAN snapshot first when doing a
> > > > > > > > > >>>> Volume
> > > > Snapshot.
> > > > > > > > > >>>> You could stop the process there and not do the
> upload.
> > > > > > > > > >>>>
> > > > > > > > > >>>> What do you think about this approach?
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thanks,
> > > > > > > > > >>>> -Syed
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > > > > > >>>> mike.tutkowski@solidfire.com <javascript:;>> wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> So, this is just me thinking out load here, but
> > > > > > > > > >>>>> if a
> > > given
> > > > > > > > > >>>>> CloudStack cloud doesn't actually need to
> > > > > > > > > >>>>> provide
> both
> > > the
> > > > > > > ability
> > > > > > > > > >>>>> to take a SAN snapshot and export it to NFS (if
> > > > > > > > > >>>>> just taking a SAN snapshot is OK), then we might
> > > > > > > > > >>>>> be able
> to
> > > get
> > > > > > > > > >>>>> away with no new
> > > > > > > API
> > > > > > > > > >>>>> calls and simply implement a new custom snapshot
> > strategy
> > > > > > > > > >>>>> and
> > > > > > > data
> > > > > > > > > >>>>> motion strategy to handle the case where the
> CloudStack
> > > > > > > > > >>>>> cloud
> > > > > > > does
> > > > > > > > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> In other words, the "default" behavior would be
> > > > > > > > > >>>>> to
> use
> > > the
> > > > > > > > > >>>>> snapshot strategy and data motion strategy that
> > > > > > > > > >>>>> we
> > > already
> > > > > > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> If your CloudStack cloud, however, wants to take
> > > > > > > > > >>>>> a
> SAN
> > > > > > > > > >>>>> snapshot and have the data exported to NFS, then
> > > > > > > > > >>>>> we
> > could
> > > > > > > > > >>>>> have you manipulate a Swing config file to make
> > > > > > > > > >>>>> use
> of
> > a
> > > > > > > > > >>>>> new snapshot strategy and data motion strategy
> > > > > > > > > >>>>> that performs both of these
> > > > > > > > > activities.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> This way, the old behavior is still the default
> > > > > > > > > >>>>> for
> > > users,
> > > > > > > > > >>>>> but CloudStack admins can change this behavior
> > > > > > > > > >>>>> via
> > > > > > configuration.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thoughts?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski
> > > > > > > > > >>>>> < mike.tutkowski@solidfire.com <javascript:;>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Right...I think we will need to come up with a
> viable
> > > > > > > > > >>>>>> upgrade path or some reasonable way for them to
> > > > > > > > > >>>>>> move
> > > from
> > > > > > > > > >>>>>> the old way to the new way (and some obvious
> > > > > > > > > >>>>>> way
> that
> > > > > > > > > >>>>>> they will know they need
> > > > > > > to
> > > > > > > > > do this).
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> I'm not really sure about the upgrade path
> > > > > > > > > >>>>>>> however, customers who are using 4.6 and are
> > > > > > > > > >>>>>>> on a managed
> > > storage
> > > > > > > > > >>>>>>> would no longer have the same functionality
> > > > > > > > > >>>>>>> with
> > Volume
> > > > > > Snapshots.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> So if I understand correctly, currently
> > > > > > > > > >>>>>>>> taking a
> > > Volume
> > > > > > > > > >>>>>>>> Snapshots of a volume on a managed storage
> > > > > > > > > >>>>>>>> keeps
> it
> > on
> > > > > > > > > >>>>>>>> the storage array. As a part of this feature,
> > > > > > > > > >>>>>>>> we
> can
> > > > > > > > > >>>>>>>> make sure
> > > > > > > that
> > > > > > > > > >>>>>>>> Volume Snapshots on managed storage are
> > > > > > > > > >>>>>>>> uploaded
> to
> > > the
> > > > > > > > > >>>>>>>> secondary storage. This would make the Volume
> > Snapshot
> > > > > > > > > >>>>>>>> feature behave the same regardless of the
> > > > > > > > > >>>>>>>> storage (managed or
> > > > > > > > > >>>>>>>> non-managed) And, for utilizing the efficient
> > backend
> > > > > > > > > >>>>>>>> storage
> > > > > > > > > capabilities, we can use the new storage snapshots API.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike
> > > > > > > > > >>>>>>>> Tutkowski < mike.tutkowski@solidfire.com
> > > > > > > > > >>>>>>>> <javascript:;>>
> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Whatever we do here, we need to have a plan
> > > > > > > > > >>>>>>>>> to
> deal
> > > > > > > > > >>>>>>>>> with the fact that we already have a feature
> > > > > > > > > >>>>>>>>> (in
> > 4.6
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a volume
> > > > > > > > > >>>>>>>>> snapshot
> > (for
> > > > > > > > > >>>>>>>>> managed
> > > > > > > > > >>>>>>>>> storage) that resides on a backend SAN
> > > > > > > > > >>>>>>>>> (using a
> > > custom
> > > > > > > > > >>>>>>>>> snapshot strategy and a custom data motion
> > strategy).
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> If these new APIs go in, then how should the
> > original
> > > > > > > > > >>>>>>>>> implementation (present in 4.6 and later) be
> > changed?
> > > > > > > > > >>>>>>>>> If it
> > > > > > > is
> > > > > > > > > >>>>>>>>> changed, how do we support customers who are
> > already
> > > > > > > > > >>>>>>>>> using
> > > > > > > the
> > > > > > > > > >>>>>>>>> original volume-snapshot API to take
> > > > > > > > > >>>>>>>>> snapshots
> on a
> > > > > > > > > >>>>>>>>> backend
> > > > > > > > SAN?
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>> Mike
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will
> > > > > > > > > >>>>>>>>> Stevens < wstevens@cloudops.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>> Will you be able to create a Template from
> > > > > > > > > >>>>>>>>>> a
> > > > > > > StorageSnapshot?
> > > > > > > > > >>>>>>>>>> If yes, will the template be stored in the
> > secondary
> > > > > > > > > >>>>>>>>>> storage like normal templates or will that
> > > > > > > > > >>>>>>>>>> be
> > > handled
> > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > > > vendor side?
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> *Will STEVENS* Lead Developer
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J
> > > > > > > > > >>>>>>>>>> 1S6
> w
> > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed
> > > > > > > > > >>>>>>>>>> Mushtaq < syed1.mushtaq@gmail.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will
> > > > > > > > > >>>>>>>>>>> Stevens < wstevens@cloudops.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in
> > > > > > > > > >>>>>>>>>>>> the
> Jira
> > > > > > > > > >>>>>>>>>>>> ticket because it was not clear in the
> 'mention'
> > > > > > > > > >>>>>>>>>>>> section because it shows as a page 'you
> > > > > > > > > >>>>>>>>>>>> do not
> > > have
> > > > > > permission to'.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> *Will STEVENS* Lead Developer
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|*
> > > > > > > > > >>>>>>>>>>>> H3J
> 1S6
> > w
> > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed
> > > > > > > > > >>>>>>>>>>>> Ahmed <sahmed@cloudops.com <javascript:;>
> > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to
> > > > > > > > > >>>>>>>>>>>>> do snapshots on managed storage backends
> > > > > > > > > >>>>>>>>>>>>> like
> > > > SolidFire.
> > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed storage
> > > > > > > > > >>>>>>>>>>>>> stay on
> > the
> > > > > > > > > >>>>>>>>>>>>> device which is contrary to what
> > > > > > > > > >>>>>>>>>>>>> CloudStack
> > calls
> > > > > > snpshots.
> > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage and
> > > > > > > > > >>>>>>>>>>>>> keeping
> it
> > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > > > > and
> > > > > > > > > >>>>>>>>>>>>> we would ideally like to have both ways
> > > > > > > > > >>>>>>>>>>>>> of
> > doing
> > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs
> > > > > > > > > >>>>>>>>>>>>> to
> > > create
> > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> What do you guys think of this feature?
> > > > > > > > > >>>>>>>>>>>>> I
> would
> > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am working
> > > > > > > > > >>>>>>>>>>>>> on
> > > making
> > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > > spec
> > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a high
> > > > > > > > > >>>>>>>>>>>>> level feedback first before starting to work on it.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> --
> > > > > > > > > >>>>>>>>> *Mike Tutkowski* *Senior CloudStack
> > > > > > > > > >>>>>>>>> Developer, SolidFire Inc.*
> > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > > > > > > > >>>>>>>>> <javascript:;>
> > > > > > > > > >>>>>>>>> o: 303.746.7302 Advancing the way the world
> > > > > > > > > >>>>>>>>> uses the cloud <
> > http://solidfire.com/solution/overview/?video=play
> > > >*™
> > > > > > > > > >>>>>>>>> *
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> --
> > > > > > > > > >>>>>> *Mike Tutkowski* *Senior CloudStack Developer,
> > > > > > > > > >>>>>> SolidFire Inc.*
> > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > >>>>>> o: 303.746.7302 Advancing the way the world
> > > > > > > > > >>>>>> uses the cloud
> > > > > > > > > >>>>>> <http://solidfire.com/solution/overview/?video=
> > > > > > > > > >>>>>> play
> > >*™*
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> --
> > > > > > > > > >>>>> *Mike Tutkowski* *Senior CloudStack Developer,
> > > > > > > > > >>>>> SolidFire Inc.*
> > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > >>>>> o: 303.746.7302
> > > > > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > > > > >>>>> <http://solidfire.com/solution/overview/?video=p
> > > > > > > > > >>>>> lay
> > >*™*
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> --
> > > > > > > > > >>> *Mike Tutkowski*
> > > > > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > >>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > >>> o: 303.746.7302
> > > > > > > > > >>> Advancing the way the world uses the cloud
> > > > > > > > > >>> <http://solidfire.com/solution/overview/?video=pla
> > > > > > > > > >>> y
> >*™*
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > > o: 303.746.7302
> > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > <http://solidfire.com/solution/overview/?video=play>
> > > > > > > > > > *™*
> > > > > > > > > >
> > > > > > > > > Find out more about ShapeBlue and our range of
> > > > > > > > > CloudStack related
> > > > > > > > services:
> > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > CSForge –
> > > > > > > > > rapid IaaS deployment framework <
> > http://shapeblue.com/csforge/
> > > >
> > > > > > > > > CloudStack Consulting
> > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > > > > CloudStack
> > > > > > > > > Software Engineering
> > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/
> > > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-suppor
> > > > > > > > > t/>
> |
> > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Mike Tutkowski*
> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > o: 303.746.7302
> > > > > > > Advancing the way the world uses the cloud
> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >
> > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > > > related
> > > > > services:
> > > > > > IaaS Cloud Design & Build
> > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > > > > CSForge –
> > > rapid
> > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > CloudStack
> > > > > > Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> > > CloudStack
> > > > > > Software Engineering
> > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > CloudStack Infrastructure Support
> > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > CloudStack
> > > > > > Bootcamp Training Courses <
> > http://shapeblue.com/cloudstack-training/
> > > >
> > > > > >
> > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > > related
> > > > services:
> > > > > IaaS Cloud Design & Build
> > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge
> > > > > –
> > rapid
> > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > CloudStack Consulting <
> http://shapeblue.com/cloudstack-consultancy/>
> > |
> > > > CloudStack
> > > > > Software Engineering
> > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > CloudStack Infrastructure Support
> > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > CloudStack
> > > > > Bootcamp Training Courses <
> http://shapeblue.com/cloudstack-training/
> > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com <javascript:;>
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>


--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

To me it sounds like number two and number three are different uses for the
same "thing"(which is totally fine).

As for taking a fast SAN snapshot and exporting it asynchronously, do we
see the SSVM as performing the export?

To be backwards compatible with what we have in 4.6 and later for volume
snapshots for managed storage, I think it might be easier if we pass in a
flag that says whether or not to archive the SAN snapshot (which, I think,
is something that you suggested, as well, Pierre-Luc).

On Monday, February 8, 2016, Pierre-Luc Dion <pd...@cloudops.com> wrote:

> Hi Mike,
>
> The reason behind the creation of a SAN snapshot which is exported into
> secondary storage, is because creating a copy of the .VHD directly would
> impact uptime of the VM as creating that copy take lots of time. Has oppose
> to a SAN snapshot that is close to instantaneous which can afterward be
> clone into Secondary Storage asynchronously.
>
> I would suspect an extracted VolumeSnapshot taken from a SAN snapshot could
> have is SAN snapshot deleted to avoid duplica and space consumption on the
> Primary Storage side.
>
>
> I see 3 definitions in our current discussion regarding the term snapshot
> (these are not official terminology but by own interpretation of them):
>
> 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a snapshot):
> it's a volume snapshot at the storage level, point in time of your data. it
> reside on the primary storage. Useful and efficient for software side
> incident.
> 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud backup aws-S3
> style ): Point in time copy of the Virtual Disk that reside on a different
> storage array then the original Volume. Facilitate data migration between
> clusters and, in case of primary storage incident, Volume snapshots are not
> impacted and can be reuse.
> 3. *Backup*: Archival of your Virtual-machines data that also validate data
> integrity, provide a storage efficient archiving method and an independent
> way to restore your data in case of an major infrastructure disaster.
>
>
> Regards,
>
> PL
>
>
> On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com <javascript:;>
> > wrote:
>
> > So, let's see if I currently follow the requirements:
> >
> > * Augment volume snapshots for managed storage to conditionally export
> data
> > to NFS. The current process of taking a snapshot on the SAN is fine, but
> > we'd like the option to export the data to NFS, as well.
> >
> > Questions:
> >
> > Once the data has been exported to NFS, do we keep the SAN snapshot or
> > delete it?
> >
> > If we are deleting the SAN snapshot, then why don't we just copy the VHD
> > from primary to secondary the way we do today for non-managed (i.e.
> > traditional) storage? Why create a SAN snapshot in that scenario? Perhaps
> > to have the SSVM mount and perform the VHD copy to secondary storage
> > instead of a XenServer host?
> >
> > Thanks for the clarification.
> >
> > By the way, to me a backup is when you copy data from one storage system
> to
> > another (regardless of features, if any, to restore the data in the
> > future). A snapshot is a point-in-time view of the data of a volume and
> > it's stored on the same storage system as the volume.
> >
> > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <pdion@cloudops.com
> <javascript:;>>
> > wrote:
> >
> > > That's fun to see that discussion happening. I 100% agree with Paul's
> > > points of view. VolumeSnapshot are not a backup, but I do consider them
> > as
> > > a safety vest against Primary Storage failure, because failure append
> > :-( .
> > >
> > > The current proposal around snapshots that reside on the primary
> storage
> > or
> > > snapshots that end in the Secondary Storage  is not to address any kind
> > of
> > > backups requirement because a snapshot is not a backup, event an
> > extracted
> > > VM snapshot.
> > >
> > > The main idea, and again this is for managed storage;
> > >
> > > 1. StorageSnapshotAPI: Provide storage side snapshot capability for
> fast
> > > response time that support rollback to previous timestamp, create new
> > > volume and maybe create template.
> > >     not required to be a new API if the work is already done, I think
> > this
> > > is a different behaviors than the user expectation of a
> volume-snapshot.
> > > 2. VolumeSnapshotAPI: Provide current cloudstack behavior that create
> an
> > > extraction of a volume into SecondaryStorage which can be reuse to
> > create a
> > > new volume into another Primary Storage. This type of snapshot is a
> slow
> > > job since yes it would have to copy the full volume size on the
> Secondary
> > > storage.
> > >
> > >
> > > PL
> > >
> > >
> > >
> > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <syed1.mushtaq@gmail.com
> <javascript:;>>
> > > wrote:
> > >
> > > > I think I share you view on the 'Ideal world'. Backup (via Volume
> > > > Snapshots) is a huge bottleneck in Cloudstack. This is amplified
> > > especially
> > > > when you have a object storage as your secondary storage because it
> > > > requires two copies (one to an NFS staging area and from there to
> > object
> > > > storage). And not to mention that all these copies are consuming
> > > hypervisor
> > > > resources. Xenserver's Dom0 is also a huge bottleneck as all the
> > Network
> > > > and I/O flow through it. So our intention of proposing the "Storage
> > > > Snapshots" is to give a better way of achiving snapshots while still
> > > > keeping the original definition of volume snpashots (ie upload to sec
> > > > storage).
> > > >
> > > > But as Erik pointed out volume snapshots are not backups. They don't
> > work
> > > > form multi-disk LVM volume groups and dynamic disks. I am all in for
> a
> > > > better backup solution which handles these use cases and utilizes the
> > > > storage's advanced features.
> > > >
> > > >
> > > >
> > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> paul.angus@shapeblue.com <javascript:;>>
> > > > wrote:
> > > >
> > > > > In the beginning... there were CloudStack snapshots and they were
> > > > actually
> > > > > volume snapshots not hypervisor point-in-time snapshots.
> > > > > Then VM snapshots were created (which are point-in-time hypervisor
> > > > > snapshots) and we started referring to the original snapshots as
> > volume
> > > > > snapshots.
> > > > >
> > > > > CloudStack does not offer 'backups', but many people use volume
> > > snapshots
> > > > > as backups. However you can't in-place restore volume snapshots and
> > if
> > > > you
> > > > > have a VM with multiple volumes, the volume snapshots must be done
> in
> > > > > series, meaning that the state across all of the volumes is
> unlikely
> > to
> > > > be
> > > > > consistent.
> > > > >
> > > > > 'Actual Backups' would enable all of the restore options which
> users
> > > > might
> > > > > expect as well options as to where they might be stored. In my
> ideal
> > > > world
> > > > > they would also be able to leverage back-end hardware (such as
> > > Solidfire,
> > > > > NetApp etc :) ) and software such as Veeam, Commvault etc to
> > accelerate
> > > > the
> > > > > process.
> > > > >
> > > > >
> > > > >
> > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > > Paul Angus
> > > > > VP Technology ,  ShapeBlue
> > > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > e:  *paul.angus@shapeblue.com <javascript:;> | t: @cloudyangus*
> > > > > <paul.angus@shapeblue.com <ja...@cloudyangus>
> |  w:
> > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > > Shape Blue Ltd is a company incorporated in England & Wales.
> > ShapeBlue
> > > > > Services India LLP is a company incorporated in India and is
> operated
> > > > under
> > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
> a
> > > > > company incorporated in Brasil and is operated under license from
> > Shape
> > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> > Republic
> > > of
> > > > > South Africa and is traded under license from Shape Blue Ltd.
> > ShapeBlue
> > > > is
> > > > > a registered trademark.
> > > > > This email and any attachments to it may be confidential and are
> > > intended
> > > > > solely for the use of the individual to whom it is addressed. Any
> > views
> > > > or
> > > > > opinions expressed are solely those of the author and do not
> > > necessarily
> > > > > represent those of Shape Blue Ltd or related companies. If you are
> > not
> > > > the
> > > > > intended recipient of this email, you must neither take any action
> > > based
> > > > > upon its contents, nor copy or show it to anyone. Please contact
> the
> > > > sender
> > > > > if you believe you have received this email in error.
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com <javascript:;>]
> > > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > >
> > > > > Paul,
> > > > >
> > > > > When you say actual backups, how would it be different from the
> > Volume
> > > > > Snapshots that exist currently. My understanding is that Backups
> end
> > up
> > > > in
> > > > > Sec Storage whereas Snapshots are just a point-in-time state of
> your
> > > > volume
> > > > > which can be restored back correct?
> > > > >
> > > > > -Syed
> > > > >
> > > > >
> > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > paul.angus@shapeblue.com <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > Hi Syed,
> > > > > >
> > > > > > As I understand it, the SolidFire plugin will export the snapshot
> > to
> > > > > > secondary storage if the user requests a template from the
> snapshot
> > > or
> > > > > > wants to download the snapshot from the cloud. This is a good,
> > > > > > pragmatic approach and yes Mike the SolidFire storage is super
> > > > > > reliable and snapshots on SolidFire arrays take up next to no
> > space.
> > > > > > BUT I think that we are talking about a more general purpose API,
> > and
> > > > > > other storage systems may not be as awesome as Mike's. That's my
> > > > > > concern. Also, the time to transfer for say 1TB to move from
> > primary
> > > > > > to sec storage and then create a VM template out of it may be too
> > > long
> > > > > for users.
> > > > > >
> > > > > > @Mike I don’t think 'we' use the term volume snapshot for backup,
> > > it's
> > > > > > just that users want to do backups and a volume snapshot is the
> > only
> > > > > > type of snapshot that copies the disk elsewhere and can be
> > scheduled.
> > > > > >
> > > > > > I'm 'pondering' the implications of enabling actual backups
> > (through
> > > > > > recognised backup providers) and the user requirements around
> them
> > > > > > (particularly restoration use cases) as a separate thread of
> work.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > Technology
> > > > > > , ShapeBlue
> > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > e: *paul.angus@shapeblue.com <javascript:;> | t: @cloudyangus*
> > > > > > <paul.angus@shapeblue.com <javascript:;>
> %20%7C%20t:%20@cloudyangus> | w:
> > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue
> > Ltd
> > > > > > is a company incorporated in England & Wales. ShapeBlue Services
> > > India
> > > > > > LLP is a company incorporated in India and is operated under
> > license
> > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > company
> > > > > > incorporated in Brasil and is operated under license from Shape
> > Blue
> > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic
> > of
> > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > > ShapeBlue is a registered trademark.
> > > > > > This email and any attachments to it may be confidential and are
> > > > > > intended solely for the use of the individual to whom it is
> > > addressed.
> > > > > > Any views or opinions expressed are solely those of the author
> and
> > do
> > > > > > not necessarily represent those of Shape Blue Ltd or related
> > > > > > companies. If you are not the intended recipient of this email,
> you
> > > > > > must neither take any action based upon its contents, nor copy or
> > > show
> > > > > > it to anyone. Please contact the sender if you believe you have
> > > > received
> > > > > this email in error.
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com
> <javascript:;>]
> > > > > > Sent: 05 February 2016 15:31
> > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > >
> > > > > > I think the terminology confusion comes from AWS where they do
> EBS
> > > > > > snapshots backed up to S3 and CloudStack sort of followed that.
> And
> > > as
> > > > > > an end user who is oblivious to the internals of my provider, my
> > > > > > expectation would be something similar to what AWS as that is my
> > > > > > biggest reference point.
> > > > > >
> > > > > > To your point Mike, I agree that a Primary Storage failure on
> > > > > > SolidFire is unlikely, there are other motivations for us to push
> > > data
> > > > > > to secondary storage. Primary storage (atleast for us) costs
> > around 3
> > > > > > times as much as secondary storage and snapshots on primary
> storage
> > > > > > are rarely used (especially for some of our customers who do
> daily
> > > > > backups).
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > > > > mike.tutkowski@solidfire.com <javascript:;>> wrote:
> > > > > >
> > > > > > > Some of the weirdness is around terminology.
> > > > > > >
> > > > > > > For most systems I've worked on, a snapshot and a backup are
> two
> > > > > > > completely different things (but CloudStack has traditionally
> > used
> > > > > > > the term "volume snapshot" to mean backup).
> > > > > > >
> > > > > > > I will put in a SolidFire "plug" here and say, though, that if
> > your
> > > > > > > primary storage is running on SolidFire that it is unlikely
> > you'll
> > > > > > > encounter an issue where your primary storage goes offline (and
> > > > > > > you'll even maintain your performance guarantees during failure
> > > > > > > scenarios and upgrades, as well). That being the case, it is
> less
> > > > > > > useful to require a backup to Swift (but it's perfectly OK if
> > > that's
> > > > > > > what we want to do
> > > > > > here).
> > > > > > >
> > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > > > > <syed1.mushtaq@gmail.com <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Paul,
> > > > > > > >
> > > > > > > > I believe with the current implementation of Snapshots on
> > managed
> > > > > > > > storage
> > > > > > > > (SolidFire) the snapshots are never exported to the secondary
> > > > > storage.
> > > > > > > > While this solves the problem of having snapshots taking
> > forever
> > > > > > > > to get to sec storage, this leaves us with a
> > > > > > > huge
> > > > > > > > liability if our primary storage goes down. We see snapshots
> as
> > > > > > > > our recovery path because we store them in Swift which is
> > > reliable
> > > > > > > > and resilient to failures.
> > > > > > > >
> > > > > > > > With Storage snpashots our goal is to have Volume snapshots
> > > always
> > > > > > > > backed up to secondary storage and Storage Snapshots stay on
> > the
> > > > > > > > primary
> > > > > > > storage.
> > > > > > > > A provider could potentially mix both these and solve the
> > problem
> > > > > > > > that you mentioned where you want to meet user's expectation
> > of a
> > > > > > > > snapshot (ie backup to sec storage) while having an ability
> to
> > > > > > > > utilize faster sanpshots (i.e. on the device)
> > > > > > > >
> > > > > > > > Hope this clarifies things.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > -Syed
> > > > > > > >
> > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > > > > <paul.angus@shapeblue.com <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > HI guys,
> > > > > > > > >
> > > > > > > > > Could someone point me to the Jira bug of FS for the
> > > > > > > > > SAN-snapshot
> > > > > > > feature
> > > > > > > > > in 4.6 which is mentioned.
> > > > > > > > >
> > > > > > > > > From my discussions with users and operators around
> snapshots
> > > > > > > > > I'd make
> > > > > > > > the
> > > > > > > > > following observations:
> > > > > > > > > a. 'users' use snapshots as backups (both long-term and
> short
> > > > > > > > > term)
> > > > > > > with
> > > > > > > > > the expectation that they can use them for recovery if
> > > required.
> > > > > > > > > b. operators fall back to snapshots if something has gone
> > wrong
> > > > > > > > > with primary storage.
> > > > > > > > > c. users sometimes want to be able to export snapshots as
> > well
> > > > > > > > > as
> > > > > > > create
> > > > > > > > > new VMs from their snapshots
> > > > > > > > > d. snapshots are a currently a massive pain for operators,
> I
> > > > > > > > > know at
> > > > > > > > least
> > > > > > > > > one public cloud who have snapshots which take 2 days to
> > > > complete.
> > > > > > > > > e. snapshots (as they are) can't be used for multiple LVM
> > > disks.
> > > > > > > > >
> > > > > > > > > I think the process Mike has used in the SolidFire plugin
> > (only
> > > > > > > > > moving
> > > > > > > > the
> > > > > > > > > disk image to secondary storage when you absolutely have
> to)
> > is
> > > > > > > > > a very
> > > > > > > > good
> > > > > > > > > and pragmatic solution. I wonder what problems an operator
> > > might
> > > > > > > > experience
> > > > > > > > > if they have an issue with a given primary storage pool in
> a
> > > > > cluster.
> > > > > > > (I
> > > > > > > > > know that that is REALLY unlikely in the SolidFire case
> Mike
> > :)
> > > > > > > > > ) And
> > > > > > > if
> > > > > > > > > the transfer from primary to secondary is slow, the time to
> > > > > > > > > being able
> > > > > > > to
> > > > > > > > > create a template or export the volume will be slow.
> > > > > > > > >
> > > > > > > > > So for me the issue is around making sure that the end
> users
> > > > > > > expectations
> > > > > > > > > are met (while improving the speed/efficiency of the back
> > end)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus
> VP
> > > > > > > > > Technology , ShapeBlue
> > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > >
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > |
> > > m:
> > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > > e: *paul.angus@shapeblue.com <javascript:;> | t:
> @cloudyangus*
> > > > > > > > > <paul.angus@shapeblue.com <javascript:;>
> %20%7C%20t:%20@cloudyangus> | w:
> > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape
> > > Blue
> > > > > > > > > Ltd is a company incorporated in England & Wales. ShapeBlue
> > > > > > > > > Services India LLP is a company incorporated in India and
> is
> > > > > > > > > operated
> > > > > > > > under
> > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria
> > Ltda
> > > > > > > > > is a company incorporated in Brasil and is operated under
> > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a
> > company
> > > > > > > > > registered by The Republic
> > > > > > > of
> > > > > > > > > South Africa and is traded under license from Shape Blue
> Ltd.
> > > > > > > > > ShapeBlue
> > > > > > > > is
> > > > > > > > > a registered trademark.
> > > > > > > > > This email and any attachments to it may be confidential
> and
> > > are
> > > > > > > intended
> > > > > > > > > solely for the use of the individual to whom it is
> addressed.
> > > > > > > > > Any views
> > > > > > > > or
> > > > > > > > > opinions expressed are solely those of the author and do
> not
> > > > > > > necessarily
> > > > > > > > > represent those of Shape Blue Ltd or related companies. If
> > you
> > > > > > > > > are not
> > > > > > > > the
> > > > > > > > > intended recipient of this email, you must neither take any
> > > > > > > > > action
> > > > > > > based
> > > > > > > > > upon its contents, nor copy or show it to anyone. Please
> > > contact
> > > > > > > > > the
> > > > > > > > sender
> > > > > > > > > if you believe you have received this email in error.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com
> <javascript:;>]
> > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > > >
> > > > > > > > > Hi Mike,
> > > > > > > > >
> > > > > > > > > The idea of introducing a new API: StorageSnapshot for
> > managed
> > > > > > > > > storage
> > > > > > > is
> > > > > > > > > because the VolumeSnapshot default, or expected, behavior
> is
> > to
> > > > > > > > > archive snapshots into the Secondary Storage. So a
> > > > > > > > > StorageSnapshot API would be
> > > > > > > > for
> > > > > > > > > snapshot that remain on the managed storage appliance.
> > > > > > > > >
> > > > > > > > > Quickly looking at the API doc and I don't see a strong
> > > > > > > > > requirement for volume snapshots to be moved into secondary
> > > > > > > > > storage. So, maybe StorageSnapshot API is not useful, but
> > both
> > > > > > > > > use cases are required. A snapshot that remain on the
> managed
> > > > > > > > > storage, and another type of
> > > > > > > snapshot
> > > > > > > > > that end up into the secondary storage. Since you've done a
> > lot
> > > > > > > > > of
> > > > > > > work,
> > > > > > > > > might easier to just add a parameter to the current
> snapshot
> > > API
> > > > > > > > > that
> > > > > > > > would
> > > > > > > > > trigger an extraction of the storage snapshot into the
> > > secondary
> > > > > > > storage?
> > > > > > > > >
> > > > > > > > > PL
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > > > > mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I think that all sounds reasonable then - thanks!
> > > > > > > > > >
> > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > > > > syed1.mushtaq@gmail.com <javascript:;>>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> You are correct Mike in terms of the requirements. One
> of
> > > our
> > > > > > > earlier
> > > > > > > > > >> iterations on this was to have an argument to the create
> > > > > > > > > >> snapshot
> > > > > > > API
> > > > > > > > > >> which decides whether to backup the volume to sec
> storage
> > > but
> > > > > > > > > >> we realized it would make management of snapshots quite
> > > messy
> > > > > > > > > >> so we proposed a new api instead.
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > > > > >> <mike.tutkowski@solidfire.com <javascript:;>>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hi,
> > > > > > > > > >>>
> > > > > > > > > >>> Just to make sure I understand all the requirements
> here:
> > > > > > > > > >>>
> > > > > > > > > >>> 1) This relates only to managed storage (1:1 mapping
> > > between
> > > > > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > > > > >>>
> > > > > > > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > > > > > > >>> functionality, which creates a snapshot on the SAN, and
> > > > > > > > > >>> extend it via a config option (or
> > > > > > > > > >>> something) to not only take the SAN snapshot, but to
> copy
> > > > > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > > > > >>>
> > > > > > > > > >>> 3) The SAN snapshot is always taken. It's the backup to
> > NFS
> > > > > > > > > >>> that is optional.
> > > > > > > > > >>>
> > > > > > > > > >>> 4) Templates can be created from the snapshot that's on
> > the
> > > > > > > > > >>> SAN (already works).
> > > > > > > > > >>>
> > > > > > > > > >>> 5) CloudStack volumes can be created from the snapshot
> > > > > > > > > >>> that's on
> > > > > > > the
> > > > > > > > > >>> SAN (already works as long as the new CloudStack volume
> > > ends
> > > > > > > > > >>> up on the same primary storage).
> > > > > > > > > >>>
> > > > > > > > > >>> Would we have a need for a storage snapshot API then or
> > > > > > > > > >>> would that just be the standard volume snapshot without
> > the
> > > > > > > > > >>> backup to
> > > > > > NFS?
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks!
> > > > > > > > > >>> Mike
> > > > > > > > > >>>
> > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > > > > >>> <syed1.mushtaq@gmail.com <javascript:;>>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Is it possible to have both functionalities (snapshot
> on
> > > > > > > > > >>>> SAN & Sec
> > > > > > > > > >>>> Storage) coexist? Because Ideally, we would like to
> have
> > > > both.
> > > > > > > > > >>>> For example, some of our customers want to implement
> > their
> > > > > > > > > >>>> own backup strategies and do encryption to their
> backups
> > > > > > > > > >>>> which is a perfect use case for Storage Snapshot while
> > our
> > > > > > > > > >>>> other customers will still keep using the standard
> > volume
> > > > > > snapshot.
> > > > > > > > > >>>>
> > > > > > > > > >>>> To keep things backward compatible, we can add a
> setting
> > > > > > > > > >>>> which
> > > > > > > says
> > > > > > > > > >>>> to not upload on secondary storage, because, after
> all,
> > > you
> > > > > > > > > >>>> would take a SAN snapshot first when doing a Volume
> > > > Snapshot.
> > > > > > > > > >>>> You could stop the process there and not do the
> upload.
> > > > > > > > > >>>>
> > > > > > > > > >>>> What do you think about this approach?
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thanks,
> > > > > > > > > >>>> -Syed
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > > > > > >>>> mike.tutkowski@solidfire.com <javascript:;>> wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> So, this is just me thinking out load here, but if a
> > > given
> > > > > > > > > >>>>> CloudStack cloud doesn't actually need to provide
> both
> > > the
> > > > > > > ability
> > > > > > > > > >>>>> to take a SAN snapshot and export it to NFS (if just
> > > > > > > > > >>>>> taking a SAN snapshot is OK), then we might be able
> to
> > > get
> > > > > > > > > >>>>> away with no new
> > > > > > > API
> > > > > > > > > >>>>> calls and simply implement a new custom snapshot
> > strategy
> > > > > > > > > >>>>> and
> > > > > > > data
> > > > > > > > > >>>>> motion strategy to handle the case where the
> CloudStack
> > > > > > > > > >>>>> cloud
> > > > > > > does
> > > > > > > > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> In other words, the "default" behavior would be to
> use
> > > the
> > > > > > > > > >>>>> snapshot strategy and data motion strategy that we
> > > already
> > > > > > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> If your CloudStack cloud, however, wants to take a
> SAN
> > > > > > > > > >>>>> snapshot and have the data exported to NFS, then we
> > could
> > > > > > > > > >>>>> have you manipulate a Swing config file to make use
> of
> > a
> > > > > > > > > >>>>> new snapshot strategy and data motion strategy that
> > > > > > > > > >>>>> performs both of these
> > > > > > > > > activities.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> This way, the old behavior is still the default for
> > > users,
> > > > > > > > > >>>>> but CloudStack admins can change this behavior via
> > > > > > configuration.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thoughts?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > > > > > > >>>>> mike.tutkowski@solidfire.com <javascript:;>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Right...I think we will need to come up with a
> viable
> > > > > > > > > >>>>>> upgrade path or some reasonable way for them to move
> > > from
> > > > > > > > > >>>>>> the old way to the new way (and some obvious way
> that
> > > > > > > > > >>>>>> they will know they need
> > > > > > > to
> > > > > > > > > do this).
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > > > > > > >>>>>> syed1.mushtaq@gmail.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> I'm not really sure about the upgrade path however,
> > > > > > > > > >>>>>>> customers who are using 4.6 and are on a managed
> > > storage
> > > > > > > > > >>>>>>> would no longer have the same functionality with
> > Volume
> > > > > > Snapshots.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > > > > > > >>>>>>> syed1.mushtaq@gmail.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> So if I understand correctly, currently taking a
> > > Volume
> > > > > > > > > >>>>>>>> Snapshots of a volume on a managed storage keeps
> it
> > on
> > > > > > > > > >>>>>>>> the storage array. As a part of this feature, we
> can
> > > > > > > > > >>>>>>>> make sure
> > > > > > > that
> > > > > > > > > >>>>>>>> Volume Snapshots on managed storage are uploaded
> to
> > > the
> > > > > > > > > >>>>>>>> secondary storage. This would make the Volume
> > Snapshot
> > > > > > > > > >>>>>>>> feature behave the same regardless of the storage
> > > > > > > > > >>>>>>>> (managed or
> > > > > > > > > >>>>>>>> non-managed) And, for utilizing the efficient
> > backend
> > > > > > > > > >>>>>>>> storage
> > > > > > > > > capabilities, we can use the new storage snapshots API.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com <javascript:;>>
> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Whatever we do here, we need to have a plan to
> deal
> > > > > > > > > >>>>>>>>> with the fact that we already have a feature (in
> > 4.6
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > > > > > >>>>>>>>> volume-snapshot APIs to create a volume snapshot
> > (for
> > > > > > > > > >>>>>>>>> managed
> > > > > > > > > >>>>>>>>> storage) that resides on a backend SAN (using a
> > > custom
> > > > > > > > > >>>>>>>>> snapshot strategy and a custom data motion
> > strategy).
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> If these new APIs go in, then how should the
> > original
> > > > > > > > > >>>>>>>>> implementation (present in 4.6 and later) be
> > changed?
> > > > > > > > > >>>>>>>>> If it
> > > > > > > is
> > > > > > > > > >>>>>>>>> changed, how do we support customers who are
> > already
> > > > > > > > > >>>>>>>>> using
> > > > > > > the
> > > > > > > > > >>>>>>>>> original volume-snapshot API to take snapshots
> on a
> > > > > > > > > >>>>>>>>> backend
> > > > > > > > SAN?
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>> Mike
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > > > > > > > >>>>>>>>> wstevens@cloudops.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>> Will you be able to create a Template from a
> > > > > > > StorageSnapshot?
> > > > > > > > > >>>>>>>>>> If yes, will the template be stored in the
> > secondary
> > > > > > > > > >>>>>>>>>> storage like normal templates or will that be
> > > handled
> > > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > > > vendor side?
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w
> > > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > > > > > > > >>>>>>>>>>> wstevens@cloudops.com <javascript:;>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the
> Jira
> > > > > > > > > >>>>>>>>>>>> ticket because it was not clear in the
> 'mention'
> > > > > > > > > >>>>>>>>>>>> section because it shows as a page 'you do not
> > > have
> > > > > > permission to'.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J
> 1S6
> > w
> > > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com <javascript:;>
> > > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do
> > > > > > > > > >>>>>>>>>>>>> snapshots on managed storage backends like
> > > > SolidFire.
> > > > > > > > > >>>>>>>>>>>>> Snapshots on current managed storage stay on
> > the
> > > > > > > > > >>>>>>>>>>>>> device which is contrary to what CloudStack
> > calls
> > > > > > snpshots.
> > > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage and keeping
> it
> > > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > > > > and
> > > > > > > > > >>>>>>>>>>>>> we would ideally like to have both ways of
> > doing
> > > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to
> > > create
> > > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> What do you guys think of this feature? I
> would
> > > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am working on
> > > making
> > > > > > > > > >>>>>>>>>>>>> the design
> > > > > > > spec
> > > > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a high level
> > > > > > > > > >>>>>>>>>>>>> feedback first before starting to work on it.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> --
> > > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > > > > > > >>>>>>>>> <
> > http://solidfire.com/solution/overview/?video=play
> > > >*™
> > > > > > > > > >>>>>>>>> *
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> --
> > > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > >>>>>> o: 303.746.7302
> > > > > > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > > > > > >>>>>> <http://solidfire.com/solution/overview/?video=play
> > >*™*
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> --
> > > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > >>>>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > >>>>> o: 303.746.7302
> > > > > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > > > > >>>>> <http://solidfire.com/solution/overview/?video=play
> > >*™*
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> --
> > > > > > > > > >>> *Mike Tutkowski*
> > > > > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > >>> e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > >>> o: 303.746.7302
> > > > > > > > > >>> Advancing the way the world uses the cloud
> > > > > > > > > >>> <http://solidfire.com/solution/overview/?video=play
> >*™*
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Mike Tutkowski*
> > > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > > > > o: 303.746.7302
> > > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > > > >
> > > > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > > > > > > related
> > > > > > > > services:
> > > > > > > > > IaaS Cloud Design & Build
> > > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > > CSForge –
> > > > > > > > > rapid IaaS deployment framework <
> > http://shapeblue.com/csforge/
> > > >
> > > > > > > > > CloudStack Consulting
> > > > > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > > > > CloudStack
> > > > > > > > > Software Engineering
> > > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > > CloudStack Infrastructure Support
> > > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/>
> |
> > > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Mike Tutkowski*
> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > e: mike.tutkowski@solidfire.com <javascript:;>
> > > > > > > o: 303.746.7302
> > > > > > > Advancing the way the world uses the cloud
> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >
> > > > > > Find out more about ShapeBlue and our range of CloudStack related
> > > > > services:
> > > > > > IaaS Cloud Design & Build
> > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > > rapid
> > > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > CloudStack
> > > > > > Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> > > CloudStack
> > > > > > Software Engineering
> > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > CloudStack Infrastructure Support
> > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > CloudStack
> > > > > > Bootcamp Training Courses <
> > http://shapeblue.com/cloudstack-training/
> > > >
> > > > > >
> > > > > Find out more about ShapeBlue and our range of CloudStack related
> > > > services:
> > > > > IaaS Cloud Design & Build
> > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > rapid
> > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > CloudStack Consulting <
> http://shapeblue.com/cloudstack-consultancy/>
> > |
> > > > CloudStack
> > > > > Software Engineering
> > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > CloudStack Infrastructure Support
> > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > CloudStack
> > > > > Bootcamp Training Courses <
> http://shapeblue.com/cloudstack-training/
> > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com <javascript:;>
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>


-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Pierre-Luc Dion <pd...@cloudops.com>]

Hi Mike,

The reason behind the creation of a SAN snapshot which is exported into
secondary storage, is because creating a copy of the .VHD directly would
impact uptime of the VM as creating that copy take lots of time. Has oppose
to a SAN snapshot that is close to instantaneous which can afterward be
clone into Secondary Storage asynchronously.

I would suspect an extracted VolumeSnapshot taken from a SAN snapshot could
have is SAN snapshot deleted to avoid duplica and space consumption on the
Primary Storage side.


I see 3 definitions in our current discussion regarding the term snapshot
(these are not official terminology but by own interpretation of them):

1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of a snapshot):
it's a volume snapshot at the storage level, point in time of your data. it
reside on the primary storage. Useful and efficient for software side
incident.
2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud backup aws-S3
style ): Point in time copy of the Virtual Disk that reside on a different
storage array then the original Volume. Facilitate data migration between
clusters and, in case of primary storage incident, Volume snapshots are not
impacted and can be reuse.
3. *Backup*: Archival of your Virtual-machines data that also validate data
integrity, provide a storage efficient archiving method and an independent
way to restore your data in case of an major infrastructure disaster.


Regards,

PL


On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> So, let's see if I currently follow the requirements:
>
> * Augment volume snapshots for managed storage to conditionally export data
> to NFS. The current process of taking a snapshot on the SAN is fine, but
> we'd like the option to export the data to NFS, as well.
>
> Questions:
>
> Once the data has been exported to NFS, do we keep the SAN snapshot or
> delete it?
>
> If we are deleting the SAN snapshot, then why don't we just copy the VHD
> from primary to secondary the way we do today for non-managed (i.e.
> traditional) storage? Why create a SAN snapshot in that scenario? Perhaps
> to have the SSVM mount and perform the VHD copy to secondary storage
> instead of a XenServer host?
>
> Thanks for the clarification.
>
> By the way, to me a backup is when you copy data from one storage system to
> another (regardless of features, if any, to restore the data in the
> future). A snapshot is a point-in-time view of the data of a volume and
> it's stored on the same storage system as the volume.
>
> On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <pd...@cloudops.com>
> wrote:
>
> > That's fun to see that discussion happening. I 100% agree with Paul's
> > points of view. VolumeSnapshot are not a backup, but I do consider them
> as
> > a safety vest against Primary Storage failure, because failure append
> :-( .
> >
> > The current proposal around snapshots that reside on the primary storage
> or
> > snapshots that end in the Secondary Storage  is not to address any kind
> of
> > backups requirement because a snapshot is not a backup, event an
> extracted
> > VM snapshot.
> >
> > The main idea, and again this is for managed storage;
> >
> > 1. StorageSnapshotAPI: Provide storage side snapshot capability for fast
> > response time that support rollback to previous timestamp, create new
> > volume and maybe create template.
> >     not required to be a new API if the work is already done, I think
> this
> > is a different behaviors than the user expectation of a volume-snapshot.
> > 2. VolumeSnapshotAPI: Provide current cloudstack behavior that create an
> > extraction of a volume into SecondaryStorage which can be reuse to
> create a
> > new volume into another Primary Storage. This type of snapshot is a slow
> > job since yes it would have to copy the full volume size on the Secondary
> > storage.
> >
> >
> > PL
> >
> >
> >
> > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <sy...@gmail.com>
> > wrote:
> >
> > > I think I share you view on the 'Ideal world'. Backup (via Volume
> > > Snapshots) is a huge bottleneck in Cloudstack. This is amplified
> > especially
> > > when you have a object storage as your secondary storage because it
> > > requires two copies (one to an NFS staging area and from there to
> object
> > > storage). And not to mention that all these copies are consuming
> > hypervisor
> > > resources. Xenserver's Dom0 is also a huge bottleneck as all the
> Network
> > > and I/O flow through it. So our intention of proposing the "Storage
> > > Snapshots" is to give a better way of achiving snapshots while still
> > > keeping the original definition of volume snpashots (ie upload to sec
> > > storage).
> > >
> > > But as Erik pointed out volume snapshots are not backups. They don't
> work
> > > form multi-disk LVM volume groups and dynamic disks. I am all in for a
> > > better backup solution which handles these use cases and utilizes the
> > > storage's advanced features.
> > >
> > >
> > >
> > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <pa...@shapeblue.com>
> > > wrote:
> > >
> > > > In the beginning... there were CloudStack snapshots and they were
> > > actually
> > > > volume snapshots not hypervisor point-in-time snapshots.
> > > > Then VM snapshots were created (which are point-in-time hypervisor
> > > > snapshots) and we started referring to the original snapshots as
> volume
> > > > snapshots.
> > > >
> > > > CloudStack does not offer 'backups', but many people use volume
> > snapshots
> > > > as backups. However you can't in-place restore volume snapshots and
> if
> > > you
> > > > have a VM with multiple volumes, the volume snapshots must be done in
> > > > series, meaning that the state across all of the volumes is unlikely
> to
> > > be
> > > > consistent.
> > > >
> > > > 'Actual Backups' would enable all of the restore options which users
> > > might
> > > > expect as well options as to where they might be stored. In my ideal
> > > world
> > > > they would also be able to leverage back-end hardware (such as
> > Solidfire,
> > > > NetApp etc :) ) and software such as Veeam, Commvault etc to
> accelerate
> > > the
> > > > process.
> > > >
> > > >
> > > >
> > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > > Paul Angus
> > > > VP Technology ,  ShapeBlue
> > > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > > > *+44 7711 418784* <+44%207711%20418784>
> > > > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > <pa...@cloudyangus>  |  w:
> > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > > Shape Blue Ltd is a company incorporated in England & Wales.
> ShapeBlue
> > > > Services India LLP is a company incorporated in India and is operated
> > > under
> > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > > > company incorporated in Brasil and is operated under license from
> Shape
> > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> Republic
> > of
> > > > South Africa and is traded under license from Shape Blue Ltd.
> ShapeBlue
> > > is
> > > > a registered trademark.
> > > > This email and any attachments to it may be confidential and are
> > intended
> > > > solely for the use of the individual to whom it is addressed. Any
> views
> > > or
> > > > opinions expressed are solely those of the author and do not
> > necessarily
> > > > represent those of Shape Blue Ltd or related companies. If you are
> not
> > > the
> > > > intended recipient of this email, you must neither take any action
> > based
> > > > upon its contents, nor copy or show it to anyone. Please contact the
> > > sender
> > > > if you believe you have received this email in error.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> > > > Sent: Friday, February 5, 2016 4:58 PM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > >
> > > > Paul,
> > > >
> > > > When you say actual backups, how would it be different from the
> Volume
> > > > Snapshots that exist currently. My understanding is that Backups end
> up
> > > in
> > > > Sec Storage whereas Snapshots are just a point-in-time state of your
> > > volume
> > > > which can be restored back correct?
> > > >
> > > > -Syed
> > > >
> > > >
> > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> paul.angus@shapeblue.com>
> > > > wrote:
> > > >
> > > > > Hi Syed,
> > > > >
> > > > > As I understand it, the SolidFire plugin will export the snapshot
> to
> > > > > secondary storage if the user requests a template from the snapshot
> > or
> > > > > wants to download the snapshot from the cloud. This is a good,
> > > > > pragmatic approach and yes Mike the SolidFire storage is super
> > > > > reliable and snapshots on SolidFire arrays take up next to no
> space.
> > > > > BUT I think that we are talking about a more general purpose API,
> and
> > > > > other storage systems may not be as awesome as Mike's. That's my
> > > > > concern. Also, the time to transfer for say 1TB to move from
> primary
> > > > > to sec storage and then create a VM template out of it may be too
> > long
> > > > for users.
> > > > >
> > > > > @Mike I don’t think 'we' use the term volume snapshot for backup,
> > it's
> > > > > just that users want to do backups and a volume snapshot is the
> only
> > > > > type of snapshot that copies the disk elsewhere and can be
> scheduled.
> > > > >
> > > > > I'm 'pondering' the implications of enabling actual backups
> (through
> > > > > recognised backup providers) and the user requirements around them
> > > > > (particularly restoration use cases) as a separate thread of work.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > Technology
> > > > > , ShapeBlue
> > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > > <pa...@cloudyangus> | w:
> > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue
> Ltd
> > > > > is a company incorporated in England & Wales. ShapeBlue Services
> > India
> > > > > LLP is a company incorporated in India and is operated under
> license
> > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> company
> > > > > incorporated in Brasil and is operated under license from Shape
> Blue
> > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic
> of
> > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > ShapeBlue is a registered trademark.
> > > > > This email and any attachments to it may be confidential and are
> > > > > intended solely for the use of the individual to whom it is
> > addressed.
> > > > > Any views or opinions expressed are solely those of the author and
> do
> > > > > not necessarily represent those of Shape Blue Ltd or related
> > > > > companies. If you are not the intended recipient of this email, you
> > > > > must neither take any action based upon its contents, nor copy or
> > show
> > > > > it to anyone. Please contact the sender if you believe you have
> > > received
> > > > this email in error.
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> > > > > Sent: 05 February 2016 15:31
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > >
> > > > > I think the terminology confusion comes from AWS where they do EBS
> > > > > snapshots backed up to S3 and CloudStack sort of followed that. And
> > as
> > > > > an end user who is oblivious to the internals of my provider, my
> > > > > expectation would be something similar to what AWS as that is my
> > > > > biggest reference point.
> > > > >
> > > > > To your point Mike, I agree that a Primary Storage failure on
> > > > > SolidFire is unlikely, there are other motivations for us to push
> > data
> > > > > to secondary storage. Primary storage (atleast for us) costs
> around 3
> > > > > times as much as secondary storage and snapshots on primary storage
> > > > > are rarely used (especially for some of our customers who do daily
> > > > backups).
> > > > >
> > > > >
> > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com> wrote:
> > > > >
> > > > > > Some of the weirdness is around terminology.
> > > > > >
> > > > > > For most systems I've worked on, a snapshot and a backup are two
> > > > > > completely different things (but CloudStack has traditionally
> used
> > > > > > the term "volume snapshot" to mean backup).
> > > > > >
> > > > > > I will put in a SolidFire "plug" here and say, though, that if
> your
> > > > > > primary storage is running on SolidFire that it is unlikely
> you'll
> > > > > > encounter an issue where your primary storage goes offline (and
> > > > > > you'll even maintain your performance guarantees during failure
> > > > > > scenarios and upgrades, as well). That being the case, it is less
> > > > > > useful to require a backup to Swift (but it's perfectly OK if
> > that's
> > > > > > what we want to do
> > > > > here).
> > > > > >
> > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > > > <sy...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Paul,
> > > > > > >
> > > > > > > I believe with the current implementation of Snapshots on
> managed
> > > > > > > storage
> > > > > > > (SolidFire) the snapshots are never exported to the secondary
> > > > storage.
> > > > > > > While this solves the problem of having snapshots taking
> forever
> > > > > > > to get to sec storage, this leaves us with a
> > > > > > huge
> > > > > > > liability if our primary storage goes down. We see snapshots as
> > > > > > > our recovery path because we store them in Swift which is
> > reliable
> > > > > > > and resilient to failures.
> > > > > > >
> > > > > > > With Storage snpashots our goal is to have Volume snapshots
> > always
> > > > > > > backed up to secondary storage and Storage Snapshots stay on
> the
> > > > > > > primary
> > > > > > storage.
> > > > > > > A provider could potentially mix both these and solve the
> problem
> > > > > > > that you mentioned where you want to meet user's expectation
> of a
> > > > > > > snapshot (ie backup to sec storage) while having an ability to
> > > > > > > utilize faster sanpshots (i.e. on the device)
> > > > > > >
> > > > > > > Hope this clarifies things.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -Syed
> > > > > > >
> > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > > > <pa...@shapeblue.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > HI guys,
> > > > > > > >
> > > > > > > > Could someone point me to the Jira bug of FS for the
> > > > > > > > SAN-snapshot
> > > > > > feature
> > > > > > > > in 4.6 which is mentioned.
> > > > > > > >
> > > > > > > > From my discussions with users and operators around snapshots
> > > > > > > > I'd make
> > > > > > > the
> > > > > > > > following observations:
> > > > > > > > a. 'users' use snapshots as backups (both long-term and short
> > > > > > > > term)
> > > > > > with
> > > > > > > > the expectation that they can use them for recovery if
> > required.
> > > > > > > > b. operators fall back to snapshots if something has gone
> wrong
> > > > > > > > with primary storage.
> > > > > > > > c. users sometimes want to be able to export snapshots as
> well
> > > > > > > > as
> > > > > > create
> > > > > > > > new VMs from their snapshots
> > > > > > > > d. snapshots are a currently a massive pain for operators, I
> > > > > > > > know at
> > > > > > > least
> > > > > > > > one public cloud who have snapshots which take 2 days to
> > > complete.
> > > > > > > > e. snapshots (as they are) can't be used for multiple LVM
> > disks.
> > > > > > > >
> > > > > > > > I think the process Mike has used in the SolidFire plugin
> (only
> > > > > > > > moving
> > > > > > > the
> > > > > > > > disk image to secondary storage when you absolutely have to)
> is
> > > > > > > > a very
> > > > > > > good
> > > > > > > > and pragmatic solution. I wonder what problems an operator
> > might
> > > > > > > experience
> > > > > > > > if they have an issue with a given primary storage pool in a
> > > > cluster.
> > > > > > (I
> > > > > > > > know that that is REALLY unlikely in the SolidFire case Mike
> :)
> > > > > > > > ) And
> > > > > > if
> > > > > > > > the transfer from primary to secondary is slow, the time to
> > > > > > > > being able
> > > > > > to
> > > > > > > > create a template or export the volume will be slow.
> > > > > > > >
> > > > > > > > So for me the issue is around making sure that the end users
> > > > > > expectations
> > > > > > > > are met (while improving the speed/efficiency of the back
> end)
> > > > > > > >
> > > > > > > >
> > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > > > > > Technology , ShapeBlue
> > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> |
> > m:
> > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > > > > > <pa...@cloudyangus> | w:
> > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape
> > Blue
> > > > > > > > Ltd is a company incorporated in England & Wales. ShapeBlue
> > > > > > > > Services India LLP is a company incorporated in India and is
> > > > > > > > operated
> > > > > > > under
> > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria
> Ltda
> > > > > > > > is a company incorporated in Brasil and is operated under
> > > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a
> company
> > > > > > > > registered by The Republic
> > > > > > of
> > > > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > > > > ShapeBlue
> > > > > > > is
> > > > > > > > a registered trademark.
> > > > > > > > This email and any attachments to it may be confidential and
> > are
> > > > > > intended
> > > > > > > > solely for the use of the individual to whom it is addressed.
> > > > > > > > Any views
> > > > > > > or
> > > > > > > > opinions expressed are solely those of the author and do not
> > > > > > necessarily
> > > > > > > > represent those of Shape Blue Ltd or related companies. If
> you
> > > > > > > > are not
> > > > > > > the
> > > > > > > > intended recipient of this email, you must neither take any
> > > > > > > > action
> > > > > > based
> > > > > > > > upon its contents, nor copy or show it to anyone. Please
> > contact
> > > > > > > > the
> > > > > > > sender
> > > > > > > > if you believe you have received this email in error.
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > > >
> > > > > > > > Hi Mike,
> > > > > > > >
> > > > > > > > The idea of introducing a new API: StorageSnapshot for
> managed
> > > > > > > > storage
> > > > > > is
> > > > > > > > because the VolumeSnapshot default, or expected, behavior is
> to
> > > > > > > > archive snapshots into the Secondary Storage. So a
> > > > > > > > StorageSnapshot API would be
> > > > > > > for
> > > > > > > > snapshot that remain on the managed storage appliance.
> > > > > > > >
> > > > > > > > Quickly looking at the API doc and I don't see a strong
> > > > > > > > requirement for volume snapshots to be moved into secondary
> > > > > > > > storage. So, maybe StorageSnapshot API is not useful, but
> both
> > > > > > > > use cases are required. A snapshot that remain on the managed
> > > > > > > > storage, and another type of
> > > > > > snapshot
> > > > > > > > that end up into the secondary storage. Since you've done a
> lot
> > > > > > > > of
> > > > > > work,
> > > > > > > > might easier to just add a parameter to the current snapshot
> > API
> > > > > > > > that
> > > > > > > would
> > > > > > > > trigger an extraction of the storage snapshot into the
> > secondary
> > > > > > storage?
> > > > > > > >
> > > > > > > > PL
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > > > mike.tutkowski@solidfire.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I think that all sounds reasonable then - thanks!
> > > > > > > > >
> > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > > > syed1.mushtaq@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> You are correct Mike in terms of the requirements. One of
> > our
> > > > > > earlier
> > > > > > > > >> iterations on this was to have an argument to the create
> > > > > > > > >> snapshot
> > > > > > API
> > > > > > > > >> which decides whether to backup the volume to sec storage
> > but
> > > > > > > > >> we realized it would make management of snapshots quite
> > messy
> > > > > > > > >> so we proposed a new api instead.
> > > > > > > > >>
> > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > > > >> <mi...@solidfire.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi,
> > > > > > > > >>>
> > > > > > > > >>> Just to make sure I understand all the requirements here:
> > > > > > > > >>>
> > > > > > > > >>> 1) This relates only to managed storage (1:1 mapping
> > between
> > > > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > > > >>>
> > > > > > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > > > > > >>> functionality, which creates a snapshot on the SAN, and
> > > > > > > > >>> extend it via a config option (or
> > > > > > > > >>> something) to not only take the SAN snapshot, but to copy
> > > > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > > > >>>
> > > > > > > > >>> 3) The SAN snapshot is always taken. It's the backup to
> NFS
> > > > > > > > >>> that is optional.
> > > > > > > > >>>
> > > > > > > > >>> 4) Templates can be created from the snapshot that's on
> the
> > > > > > > > >>> SAN (already works).
> > > > > > > > >>>
> > > > > > > > >>> 5) CloudStack volumes can be created from the snapshot
> > > > > > > > >>> that's on
> > > > > > the
> > > > > > > > >>> SAN (already works as long as the new CloudStack volume
> > ends
> > > > > > > > >>> up on the same primary storage).
> > > > > > > > >>>
> > > > > > > > >>> Would we have a need for a storage snapshot API then or
> > > > > > > > >>> would that just be the standard volume snapshot without
> the
> > > > > > > > >>> backup to
> > > > > NFS?
> > > > > > > > >>>
> > > > > > > > >>> Thanks!
> > > > > > > > >>> Mike
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > > > >>> <sy...@gmail.com>
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Is it possible to have both functionalities (snapshot on
> > > > > > > > >>>> SAN & Sec
> > > > > > > > >>>> Storage) coexist? Because Ideally, we would like to have
> > > both.
> > > > > > > > >>>> For example, some of our customers want to implement
> their
> > > > > > > > >>>> own backup strategies and do encryption to their backups
> > > > > > > > >>>> which is a perfect use case for Storage Snapshot while
> our
> > > > > > > > >>>> other customers will still keep using the standard
> volume
> > > > > snapshot.
> > > > > > > > >>>>
> > > > > > > > >>>> To keep things backward compatible, we can add a setting
> > > > > > > > >>>> which
> > > > > > says
> > > > > > > > >>>> to not upload on secondary storage, because, after all,
> > you
> > > > > > > > >>>> would take a SAN snapshot first when doing a Volume
> > > Snapshot.
> > > > > > > > >>>> You could stop the process there and not do the upload.
> > > > > > > > >>>>
> > > > > > > > >>>> What do you think about this approach?
> > > > > > > > >>>>
> > > > > > > > >>>> Thanks,
> > > > > > > > >>>> -Syed
> > > > > > > > >>>>
> > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > > > > >>>> mike.tutkowski@solidfire.com> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> So, this is just me thinking out load here, but if a
> > given
> > > > > > > > >>>>> CloudStack cloud doesn't actually need to provide both
> > the
> > > > > > ability
> > > > > > > > >>>>> to take a SAN snapshot and export it to NFS (if just
> > > > > > > > >>>>> taking a SAN snapshot is OK), then we might be able to
> > get
> > > > > > > > >>>>> away with no new
> > > > > > API
> > > > > > > > >>>>> calls and simply implement a new custom snapshot
> strategy
> > > > > > > > >>>>> and
> > > > > > data
> > > > > > > > >>>>> motion strategy to handle the case where the CloudStack
> > > > > > > > >>>>> cloud
> > > > > > does
> > > > > > > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > > > > > > >>>>>
> > > > > > > > >>>>> In other words, the "default" behavior would be to use
> > the
> > > > > > > > >>>>> snapshot strategy and data motion strategy that we
> > already
> > > > > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > > > > >>>>>
> > > > > > > > >>>>> If your CloudStack cloud, however, wants to take a SAN
> > > > > > > > >>>>> snapshot and have the data exported to NFS, then we
> could
> > > > > > > > >>>>> have you manipulate a Swing config file to make use of
> a
> > > > > > > > >>>>> new snapshot strategy and data motion strategy that
> > > > > > > > >>>>> performs both of these
> > > > > > > > activities.
> > > > > > > > >>>>>
> > > > > > > > >>>>> This way, the old behavior is still the default for
> > users,
> > > > > > > > >>>>> but CloudStack admins can change this behavior via
> > > > > configuration.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thoughts?
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > > > > > >>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Right...I think we will need to come up with a viable
> > > > > > > > >>>>>> upgrade path or some reasonable way for them to move
> > from
> > > > > > > > >>>>>> the old way to the new way (and some obvious way that
> > > > > > > > >>>>>> they will know they need
> > > > > > to
> > > > > > > > do this).
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > > > > > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> I'm not really sure about the upgrade path however,
> > > > > > > > >>>>>>> customers who are using 4.6 and are on a managed
> > storage
> > > > > > > > >>>>>>> would no longer have the same functionality with
> Volume
> > > > > Snapshots.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > > > > > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> So if I understand correctly, currently taking a
> > Volume
> > > > > > > > >>>>>>>> Snapshots of a volume on a managed storage keeps it
> on
> > > > > > > > >>>>>>>> the storage array. As a part of this feature, we can
> > > > > > > > >>>>>>>> make sure
> > > > > > that
> > > > > > > > >>>>>>>> Volume Snapshots on managed storage are uploaded to
> > the
> > > > > > > > >>>>>>>> secondary storage. This would make the Volume
> Snapshot
> > > > > > > > >>>>>>>> feature behave the same regardless of the storage
> > > > > > > > >>>>>>>> (managed or
> > > > > > > > >>>>>>>> non-managed) And, for utilizing the efficient
> backend
> > > > > > > > >>>>>>>> storage
> > > > > > > > capabilities, we can use the new storage snapshots API.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > > > > > > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Whatever we do here, we need to have a plan to deal
> > > > > > > > >>>>>>>>> with the fact that we already have a feature (in
> 4.6
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > > > > >>>>>>>>> volume-snapshot APIs to create a volume snapshot
> (for
> > > > > > > > >>>>>>>>> managed
> > > > > > > > >>>>>>>>> storage) that resides on a backend SAN (using a
> > custom
> > > > > > > > >>>>>>>>> snapshot strategy and a custom data motion
> strategy).
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> If these new APIs go in, then how should the
> original
> > > > > > > > >>>>>>>>> implementation (present in 4.6 and later) be
> changed?
> > > > > > > > >>>>>>>>> If it
> > > > > > is
> > > > > > > > >>>>>>>>> changed, how do we support customers who are
> already
> > > > > > > > >>>>>>>>> using
> > > > > > the
> > > > > > > > >>>>>>>>> original volume-snapshot API to take snapshots on a
> > > > > > > > >>>>>>>>> backend
> > > > > > > SAN?
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>> Mike
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > > > > > > >>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> Will you be able to create a Template from a
> > > > > > StorageSnapshot?
> > > > > > > > >>>>>>>>>> If yes, will the template be stored in the
> secondary
> > > > > > > > >>>>>>>>>> storage like normal templates or will that be
> > handled
> > > > > > > > >>>>>>>>>> somehow on the
> > > > > > > > vendor side?
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > > > > > > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira
> > > > > > > > >>>>>>>>>>>> ticket because it was not clear in the 'mention'
> > > > > > > > >>>>>>>>>>>> section because it shows as a page 'you do not
> > have
> > > > > permission to'.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w
> > > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > > >>>>>>>>>>>>> 27
> > > > > > > > >>>>>>>>>>>>> 8
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do
> > > > > > > > >>>>>>>>>>>>> snapshots on managed storage backends like
> > > SolidFire.
> > > > > > > > >>>>>>>>>>>>> Snapshots on current managed storage stay on
> the
> > > > > > > > >>>>>>>>>>>>> device which is contrary to what CloudStack
> calls
> > > > > snpshots.
> > > > > > > > >>>>>>>>>>>>> But taking snapshots on storage and keeping it
> > > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > > > and
> > > > > > > > >>>>>>>>>>>>> we would ideally like to have both ways of
> doing
> > > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to
> > create
> > > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> What do you guys think of this feature? I would
> > > > > > > > >>>>>>>>>>>>> love to have some feedback. I am working on
> > making
> > > > > > > > >>>>>>>>>>>>> the design
> > > > > > spec
> > > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a high level
> > > > > > > > >>>>>>>>>>>>> feedback first before starting to work on it.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> --
> > > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > > > > > >>>>>>>>> <
> http://solidfire.com/solution/overview/?video=play
> > >*™
> > > > > > > > >>>>>>>>> *
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> --
> > > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > > > > > > >>>>>> o: 303.746.7302
> > > > > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > > > > >>>>>> <http://solidfire.com/solution/overview/?video=play
> >*™*
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> --
> > > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > > > > > >>>>> o: 303.746.7302
> > > > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > > > >>>>> <http://solidfire.com/solution/overview/?video=play
> >*™*
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> --
> > > > > > > > >>> *Mike Tutkowski*
> > > > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > >>> e: mike.tutkowski@solidfire.com
> > > > > > > > >>> o: 303.746.7302
> > > > > > > > >>> Advancing the way the world uses the cloud
> > > > > > > > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Mike Tutkowski*
> > > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > > > o: 303.746.7302
> > > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > > >
> > > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > > > > > related
> > > > > > > services:
> > > > > > > > IaaS Cloud Design & Build
> > > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> > CSForge –
> > > > > > > > rapid IaaS deployment framework <
> http://shapeblue.com/csforge/
> > >
> > > > > > > > CloudStack Consulting
> > > > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > > > CloudStack
> > > > > > > > Software Engineering
> > > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > > CloudStack Infrastructure Support
> > > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > > > > > CloudStack Bootcamp Training Courses
> > > > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > e: mike.tutkowski@solidfire.com
> > > > > > o: 303.746.7302
> > > > > > Advancing the way the world uses the cloud
> > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >
> > > > > Find out more about ShapeBlue and our range of CloudStack related
> > > > services:
> > > > > IaaS Cloud Design & Build
> > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > rapid
> > > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack
> > > > > Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> > CloudStack
> > > > > Software Engineering
> > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > CloudStack Infrastructure Support
> > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > CloudStack
> > > > > Bootcamp Training Courses <
> http://shapeblue.com/cloudstack-training/
> > >
> > > > >
> > > > Find out more about ShapeBlue and our range of CloudStack related
> > > services:
> > > > IaaS Cloud Design & Build
> > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> rapid
> > > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
> |
> > > CloudStack
> > > > Software Engineering
> > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > CloudStack Infrastructure Support
> > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> CloudStack
> > > > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/
> >
> > > >
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

So, let's see if I currently follow the requirements:

* Augment volume snapshots for managed storage to conditionally export data
to NFS. The current process of taking a snapshot on the SAN is fine, but
we'd like the option to export the data to NFS, as well.

Questions:

Once the data has been exported to NFS, do we keep the SAN snapshot or
delete it?

If we are deleting the SAN snapshot, then why don't we just copy the VHD
from primary to secondary the way we do today for non-managed (i.e.
traditional) storage? Why create a SAN snapshot in that scenario? Perhaps
to have the SSVM mount and perform the VHD copy to secondary storage
instead of a XenServer host?

Thanks for the clarification.

By the way, to me a backup is when you copy data from one storage system to
another (regardless of features, if any, to restore the data in the
future). A snapshot is a point-in-time view of the data of a volume and
it's stored on the same storage system as the volume.

On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <pd...@cloudops.com> wrote:

> That's fun to see that discussion happening. I 100% agree with Paul's
> points of view. VolumeSnapshot are not a backup, but I do consider them as
> a safety vest against Primary Storage failure, because failure append :-( .
>
> The current proposal around snapshots that reside on the primary storage or
> snapshots that end in the Secondary Storage  is not to address any kind of
> backups requirement because a snapshot is not a backup, event an extracted
> VM snapshot.
>
> The main idea, and again this is for managed storage;
>
> 1. StorageSnapshotAPI: Provide storage side snapshot capability for fast
> response time that support rollback to previous timestamp, create new
> volume and maybe create template.
>     not required to be a new API if the work is already done, I think this
> is a different behaviors than the user expectation of a volume-snapshot.
> 2. VolumeSnapshotAPI: Provide current cloudstack behavior that create an
> extraction of a volume into SecondaryStorage which can be reuse to create a
> new volume into another Primary Storage. This type of snapshot is a slow
> job since yes it would have to copy the full volume size on the Secondary
> storage.
>
>
> PL
>
>
>
> On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
> > I think I share you view on the 'Ideal world'. Backup (via Volume
> > Snapshots) is a huge bottleneck in Cloudstack. This is amplified
> especially
> > when you have a object storage as your secondary storage because it
> > requires two copies (one to an NFS staging area and from there to object
> > storage). And not to mention that all these copies are consuming
> hypervisor
> > resources. Xenserver's Dom0 is also a huge bottleneck as all the Network
> > and I/O flow through it. So our intention of proposing the "Storage
> > Snapshots" is to give a better way of achiving snapshots while still
> > keeping the original definition of volume snpashots (ie upload to sec
> > storage).
> >
> > But as Erik pointed out volume snapshots are not backups. They don't work
> > form multi-disk LVM volume groups and dynamic disks. I am all in for a
> > better backup solution which handles these use cases and utilizes the
> > storage's advanced features.
> >
> >
> >
> > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <pa...@shapeblue.com>
> > wrote:
> >
> > > In the beginning... there were CloudStack snapshots and they were
> > actually
> > > volume snapshots not hypervisor point-in-time snapshots.
> > > Then VM snapshots were created (which are point-in-time hypervisor
> > > snapshots) and we started referring to the original snapshots as volume
> > > snapshots.
> > >
> > > CloudStack does not offer 'backups', but many people use volume
> snapshots
> > > as backups. However you can't in-place restore volume snapshots and if
> > you
> > > have a VM with multiple volumes, the volume snapshots must be done in
> > > series, meaning that the state across all of the volumes is unlikely to
> > be
> > > consistent.
> > >
> > > 'Actual Backups' would enable all of the restore options which users
> > might
> > > expect as well options as to where they might be stored. In my ideal
> > world
> > > they would also be able to leverage back-end hardware (such as
> Solidfire,
> > > NetApp etc :) ) and software such as Veeam, Commvault etc to accelerate
> > the
> > > process.
> > >
> > >
> > >
> > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > Paul Angus
> > > VP Technology ,  ShapeBlue
> > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > > *+44 7711 418784* <+44%207711%20418784>
> > > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > > <pa...@cloudyangus>  |  w:
> > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> > > Services India LLP is a company incorporated in India and is operated
> > under
> > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > > company incorporated in Brasil and is operated under license from Shape
> > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic
> of
> > > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
> > is
> > > a registered trademark.
> > > This email and any attachments to it may be confidential and are
> intended
> > > solely for the use of the individual to whom it is addressed. Any views
> > or
> > > opinions expressed are solely those of the author and do not
> necessarily
> > > represent those of Shape Blue Ltd or related companies. If you are not
> > the
> > > intended recipient of this email, you must neither take any action
> based
> > > upon its contents, nor copy or show it to anyone. Please contact the
> > sender
> > > if you believe you have received this email in error.
> > >
> > >
> > > -----Original Message-----
> > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> > > Sent: Friday, February 5, 2016 4:58 PM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > >
> > > Paul,
> > >
> > > When you say actual backups, how would it be different from the Volume
> > > Snapshots that exist currently. My understanding is that Backups end up
> > in
> > > Sec Storage whereas Snapshots are just a point-in-time state of your
> > volume
> > > which can be restored back correct?
> > >
> > > -Syed
> > >
> > >
> > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <pa...@shapeblue.com>
> > > wrote:
> > >
> > > > Hi Syed,
> > > >
> > > > As I understand it, the SolidFire plugin will export the snapshot to
> > > > secondary storage if the user requests a template from the snapshot
> or
> > > > wants to download the snapshot from the cloud. This is a good,
> > > > pragmatic approach and yes Mike the SolidFire storage is super
> > > > reliable and snapshots on SolidFire arrays take up next to no space.
> > > > BUT I think that we are talking about a more general purpose API, and
> > > > other storage systems may not be as awesome as Mike's. That's my
> > > > concern. Also, the time to transfer for say 1TB to move from primary
> > > > to sec storage and then create a VM template out of it may be too
> long
> > > for users.
> > > >
> > > > @Mike I don’t think 'we' use the term volume snapshot for backup,
> it's
> > > > just that users want to do backups and a volume snapshot is the only
> > > > type of snapshot that copies the disk elsewhere and can be scheduled.
> > > >
> > > > I'm 'pondering' the implications of enabling actual backups (through
> > > > recognised backup providers) and the user requirements around them
> > > > (particularly restoration use cases) as a separate thread of work.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> Technology
> > > > , ShapeBlue
> > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > *+44 7711 418784* <+44%207711%20418784>
> > > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > <pa...@cloudyangus> | w:
> > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue Ltd
> > > > is a company incorporated in England & Wales. ShapeBlue Services
> India
> > > > LLP is a company incorporated in India and is operated under license
> > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company
> > > > incorporated in Brasil and is operated under license from Shape Blue
> > > > Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > ShapeBlue is a registered trademark.
> > > > This email and any attachments to it may be confidential and are
> > > > intended solely for the use of the individual to whom it is
> addressed.
> > > > Any views or opinions expressed are solely those of the author and do
> > > > not necessarily represent those of Shape Blue Ltd or related
> > > > companies. If you are not the intended recipient of this email, you
> > > > must neither take any action based upon its contents, nor copy or
> show
> > > > it to anyone. Please contact the sender if you believe you have
> > received
> > > this email in error.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> > > > Sent: 05 February 2016 15:31
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > >
> > > > I think the terminology confusion comes from AWS where they do EBS
> > > > snapshots backed up to S3 and CloudStack sort of followed that. And
> as
> > > > an end user who is oblivious to the internals of my provider, my
> > > > expectation would be something similar to what AWS as that is my
> > > > biggest reference point.
> > > >
> > > > To your point Mike, I agree that a Primary Storage failure on
> > > > SolidFire is unlikely, there are other motivations for us to push
> data
> > > > to secondary storage. Primary storage (atleast for us) costs around 3
> > > > times as much as secondary storage and snapshots on primary storage
> > > > are rarely used (especially for some of our customers who do daily
> > > backups).
> > > >
> > > >
> > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com> wrote:
> > > >
> > > > > Some of the weirdness is around terminology.
> > > > >
> > > > > For most systems I've worked on, a snapshot and a backup are two
> > > > > completely different things (but CloudStack has traditionally used
> > > > > the term "volume snapshot" to mean backup).
> > > > >
> > > > > I will put in a SolidFire "plug" here and say, though, that if your
> > > > > primary storage is running on SolidFire that it is unlikely you'll
> > > > > encounter an issue where your primary storage goes offline (and
> > > > > you'll even maintain your performance guarantees during failure
> > > > > scenarios and upgrades, as well). That being the case, it is less
> > > > > useful to require a backup to Swift (but it's perfectly OK if
> that's
> > > > > what we want to do
> > > > here).
> > > > >
> > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > > <sy...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Paul,
> > > > > >
> > > > > > I believe with the current implementation of Snapshots on managed
> > > > > > storage
> > > > > > (SolidFire) the snapshots are never exported to the secondary
> > > storage.
> > > > > > While this solves the problem of having snapshots taking forever
> > > > > > to get to sec storage, this leaves us with a
> > > > > huge
> > > > > > liability if our primary storage goes down. We see snapshots as
> > > > > > our recovery path because we store them in Swift which is
> reliable
> > > > > > and resilient to failures.
> > > > > >
> > > > > > With Storage snpashots our goal is to have Volume snapshots
> always
> > > > > > backed up to secondary storage and Storage Snapshots stay on the
> > > > > > primary
> > > > > storage.
> > > > > > A provider could potentially mix both these and solve the problem
> > > > > > that you mentioned where you want to meet user's expectation of a
> > > > > > snapshot (ie backup to sec storage) while having an ability to
> > > > > > utilize faster sanpshots (i.e. on the device)
> > > > > >
> > > > > > Hope this clarifies things.
> > > > > >
> > > > > > Thanks,
> > > > > > -Syed
> > > > > >
> > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > > <pa...@shapeblue.com>
> > > > > > wrote:
> > > > > >
> > > > > > > HI guys,
> > > > > > >
> > > > > > > Could someone point me to the Jira bug of FS for the
> > > > > > > SAN-snapshot
> > > > > feature
> > > > > > > in 4.6 which is mentioned.
> > > > > > >
> > > > > > > From my discussions with users and operators around snapshots
> > > > > > > I'd make
> > > > > > the
> > > > > > > following observations:
> > > > > > > a. 'users' use snapshots as backups (both long-term and short
> > > > > > > term)
> > > > > with
> > > > > > > the expectation that they can use them for recovery if
> required.
> > > > > > > b. operators fall back to snapshots if something has gone wrong
> > > > > > > with primary storage.
> > > > > > > c. users sometimes want to be able to export snapshots as well
> > > > > > > as
> > > > > create
> > > > > > > new VMs from their snapshots
> > > > > > > d. snapshots are a currently a massive pain for operators, I
> > > > > > > know at
> > > > > > least
> > > > > > > one public cloud who have snapshots which take 2 days to
> > complete.
> > > > > > > e. snapshots (as they are) can't be used for multiple LVM
> disks.
> > > > > > >
> > > > > > > I think the process Mike has used in the SolidFire plugin (only
> > > > > > > moving
> > > > > > the
> > > > > > > disk image to secondary storage when you absolutely have to) is
> > > > > > > a very
> > > > > > good
> > > > > > > and pragmatic solution. I wonder what problems an operator
> might
> > > > > > experience
> > > > > > > if they have an issue with a given primary storage pool in a
> > > cluster.
> > > > > (I
> > > > > > > know that that is REALLY unlikely in the SolidFire case Mike :)
> > > > > > > ) And
> > > > > if
> > > > > > > the transfer from primary to secondary is slow, the time to
> > > > > > > being able
> > > > > to
> > > > > > > create a template or export the volume will be slow.
> > > > > > >
> > > > > > > So for me the issue is around making sure that the end users
> > > > > expectations
> > > > > > > are met (while improving the speed/efficiency of the back end)
> > > > > > >
> > > > > > >
> > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > > > > Technology , ShapeBlue
> > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> |
> m:
> > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > > > > <pa...@cloudyangus> | w:
> > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape
> Blue
> > > > > > > Ltd is a company incorporated in England & Wales. ShapeBlue
> > > > > > > Services India LLP is a company incorporated in India and is
> > > > > > > operated
> > > > > > under
> > > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> > > > > > > is a company incorporated in Brasil and is operated under
> > > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > > > > > registered by The Republic
> > > > > of
> > > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > > > ShapeBlue
> > > > > > is
> > > > > > > a registered trademark.
> > > > > > > This email and any attachments to it may be confidential and
> are
> > > > > intended
> > > > > > > solely for the use of the individual to whom it is addressed.
> > > > > > > Any views
> > > > > > or
> > > > > > > opinions expressed are solely those of the author and do not
> > > > > necessarily
> > > > > > > represent those of Shape Blue Ltd or related companies. If you
> > > > > > > are not
> > > > > > the
> > > > > > > intended recipient of this email, you must neither take any
> > > > > > > action
> > > > > based
> > > > > > > upon its contents, nor copy or show it to anyone. Please
> contact
> > > > > > > the
> > > > > > sender
> > > > > > > if you believe you have received this email in error.
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > > To: dev@cloudstack.apache.org
> > > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > > >
> > > > > > > Hi Mike,
> > > > > > >
> > > > > > > The idea of introducing a new API: StorageSnapshot for managed
> > > > > > > storage
> > > > > is
> > > > > > > because the VolumeSnapshot default, or expected, behavior is to
> > > > > > > archive snapshots into the Secondary Storage. So a
> > > > > > > StorageSnapshot API would be
> > > > > > for
> > > > > > > snapshot that remain on the managed storage appliance.
> > > > > > >
> > > > > > > Quickly looking at the API doc and I don't see a strong
> > > > > > > requirement for volume snapshots to be moved into secondary
> > > > > > > storage. So, maybe StorageSnapshot API is not useful, but both
> > > > > > > use cases are required. A snapshot that remain on the managed
> > > > > > > storage, and another type of
> > > > > snapshot
> > > > > > > that end up into the secondary storage. Since you've done a lot
> > > > > > > of
> > > > > work,
> > > > > > > might easier to just add a parameter to the current snapshot
> API
> > > > > > > that
> > > > > > would
> > > > > > > trigger an extraction of the storage snapshot into the
> secondary
> > > > > storage?
> > > > > > >
> > > > > > > PL
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > > mike.tutkowski@solidfire.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > I think that all sounds reasonable then - thanks!
> > > > > > > >
> > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > > syed1.mushtaq@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> You are correct Mike in terms of the requirements. One of
> our
> > > > > earlier
> > > > > > > >> iterations on this was to have an argument to the create
> > > > > > > >> snapshot
> > > > > API
> > > > > > > >> which decides whether to backup the volume to sec storage
> but
> > > > > > > >> we realized it would make management of snapshots quite
> messy
> > > > > > > >> so we proposed a new api instead.
> > > > > > > >>
> > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > > >> <mi...@solidfire.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Hi,
> > > > > > > >>>
> > > > > > > >>> Just to make sure I understand all the requirements here:
> > > > > > > >>>
> > > > > > > >>> 1) This relates only to managed storage (1:1 mapping
> between
> > > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > > >>>
> > > > > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > > > > >>> functionality, which creates a snapshot on the SAN, and
> > > > > > > >>> extend it via a config option (or
> > > > > > > >>> something) to not only take the SAN snapshot, but to copy
> > > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > > >>>
> > > > > > > >>> 3) The SAN snapshot is always taken. It's the backup to NFS
> > > > > > > >>> that is optional.
> > > > > > > >>>
> > > > > > > >>> 4) Templates can be created from the snapshot that's on the
> > > > > > > >>> SAN (already works).
> > > > > > > >>>
> > > > > > > >>> 5) CloudStack volumes can be created from the snapshot
> > > > > > > >>> that's on
> > > > > the
> > > > > > > >>> SAN (already works as long as the new CloudStack volume
> ends
> > > > > > > >>> up on the same primary storage).
> > > > > > > >>>
> > > > > > > >>> Would we have a need for a storage snapshot API then or
> > > > > > > >>> would that just be the standard volume snapshot without the
> > > > > > > >>> backup to
> > > > NFS?
> > > > > > > >>>
> > > > > > > >>> Thanks!
> > > > > > > >>> Mike
> > > > > > > >>>
> > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > > >>> <sy...@gmail.com>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> Is it possible to have both functionalities (snapshot on
> > > > > > > >>>> SAN & Sec
> > > > > > > >>>> Storage) coexist? Because Ideally, we would like to have
> > both.
> > > > > > > >>>> For example, some of our customers want to implement their
> > > > > > > >>>> own backup strategies and do encryption to their backups
> > > > > > > >>>> which is a perfect use case for Storage Snapshot while our
> > > > > > > >>>> other customers will still keep using the standard volume
> > > > snapshot.
> > > > > > > >>>>
> > > > > > > >>>> To keep things backward compatible, we can add a setting
> > > > > > > >>>> which
> > > > > says
> > > > > > > >>>> to not upload on secondary storage, because, after all,
> you
> > > > > > > >>>> would take a SAN snapshot first when doing a Volume
> > Snapshot.
> > > > > > > >>>> You could stop the process there and not do the upload.
> > > > > > > >>>>
> > > > > > > >>>> What do you think about this approach?
> > > > > > > >>>>
> > > > > > > >>>> Thanks,
> > > > > > > >>>> -Syed
> > > > > > > >>>>
> > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > > > >>>> mike.tutkowski@solidfire.com> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> So, this is just me thinking out load here, but if a
> given
> > > > > > > >>>>> CloudStack cloud doesn't actually need to provide both
> the
> > > > > ability
> > > > > > > >>>>> to take a SAN snapshot and export it to NFS (if just
> > > > > > > >>>>> taking a SAN snapshot is OK), then we might be able to
> get
> > > > > > > >>>>> away with no new
> > > > > API
> > > > > > > >>>>> calls and simply implement a new custom snapshot strategy
> > > > > > > >>>>> and
> > > > > data
> > > > > > > >>>>> motion strategy to handle the case where the CloudStack
> > > > > > > >>>>> cloud
> > > > > does
> > > > > > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > > > > > >>>>>
> > > > > > > >>>>> In other words, the "default" behavior would be to use
> the
> > > > > > > >>>>> snapshot strategy and data motion strategy that we
> already
> > > > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > > > >>>>>
> > > > > > > >>>>> If your CloudStack cloud, however, wants to take a SAN
> > > > > > > >>>>> snapshot and have the data exported to NFS, then we could
> > > > > > > >>>>> have you manipulate a Swing config file to make use of a
> > > > > > > >>>>> new snapshot strategy and data motion strategy that
> > > > > > > >>>>> performs both of these
> > > > > > > activities.
> > > > > > > >>>>>
> > > > > > > >>>>> This way, the old behavior is still the default for
> users,
> > > > > > > >>>>> but CloudStack admins can change this behavior via
> > > > configuration.
> > > > > > > >>>>>
> > > > > > > >>>>> Thoughts?
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > > > > >>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Right...I think we will need to come up with a viable
> > > > > > > >>>>>> upgrade path or some reasonable way for them to move
> from
> > > > > > > >>>>>> the old way to the new way (and some obvious way that
> > > > > > > >>>>>> they will know they need
> > > > > to
> > > > > > > do this).
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > > > > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> I'm not really sure about the upgrade path however,
> > > > > > > >>>>>>> customers who are using 4.6 and are on a managed
> storage
> > > > > > > >>>>>>> would no longer have the same functionality with Volume
> > > > Snapshots.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > > > > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> So if I understand correctly, currently taking a
> Volume
> > > > > > > >>>>>>>> Snapshots of a volume on a managed storage keeps it on
> > > > > > > >>>>>>>> the storage array. As a part of this feature, we can
> > > > > > > >>>>>>>> make sure
> > > > > that
> > > > > > > >>>>>>>> Volume Snapshots on managed storage are uploaded to
> the
> > > > > > > >>>>>>>> secondary storage. This would make the Volume Snapshot
> > > > > > > >>>>>>>> feature behave the same regardless of the storage
> > > > > > > >>>>>>>> (managed or
> > > > > > > >>>>>>>> non-managed) And, for utilizing the efficient backend
> > > > > > > >>>>>>>> storage
> > > > > > > capabilities, we can use the new storage snapshots API.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > > > > > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Hi everyone,
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Whatever we do here, we need to have a plan to deal
> > > > > > > >>>>>>>>> with the fact that we already have a feature (in 4.6
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > > > >>>>>>>>> volume-snapshot APIs to create a volume snapshot (for
> > > > > > > >>>>>>>>> managed
> > > > > > > >>>>>>>>> storage) that resides on a backend SAN (using a
> custom
> > > > > > > >>>>>>>>> snapshot strategy and a custom data motion strategy).
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> If these new APIs go in, then how should the original
> > > > > > > >>>>>>>>> implementation (present in 4.6 and later) be changed?
> > > > > > > >>>>>>>>> If it
> > > > > is
> > > > > > > >>>>>>>>> changed, how do we support customers who are already
> > > > > > > >>>>>>>>> using
> > > > > the
> > > > > > > >>>>>>>>> original volume-snapshot API to take snapshots on a
> > > > > > > >>>>>>>>> backend
> > > > > > SAN?
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Thanks,
> > > > > > > >>>>>>>>> Mike
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > > > > > >>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> Will you be able to create a Template from a
> > > > > StorageSnapshot?
> > > > > > > >>>>>>>>>> If yes, will the template be stored in the secondary
> > > > > > > >>>>>>>>>> storage like normal templates or will that be
> handled
> > > > > > > >>>>>>>>>> somehow on the
> > > > > > > vendor side?
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > > >>>>>>>>>> Lead Developer
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > > > > > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira
> > > > > > > >>>>>>>>>>>> ticket because it was not clear in the 'mention'
> > > > > > > >>>>>>>>>>>> section because it shows as a page 'you do not
> have
> > > > permission to'.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > > > >>>>>>>>>>>> > wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > > >>>>>>>>>>>>> 27
> > > > > > > >>>>>>>>>>>>> 8
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do
> > > > > > > >>>>>>>>>>>>> snapshots on managed storage backends like
> > SolidFire.
> > > > > > > >>>>>>>>>>>>> Snapshots on current managed storage stay on the
> > > > > > > >>>>>>>>>>>>> device which is contrary to what CloudStack calls
> > > > snpshots.
> > > > > > > >>>>>>>>>>>>> But taking snapshots on storage and keeping it
> > > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > > and
> > > > > > > >>>>>>>>>>>>> we would ideally like to have both ways of doing
> > > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to
> create
> > > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> What do you guys think of this feature? I would
> > > > > > > >>>>>>>>>>>>> love to have some feedback. I am working on
> making
> > > > > > > >>>>>>>>>>>>> the design
> > > > > spec
> > > > > > > >>>>>>>>>>>>> more concrete but wanted to have a high level
> > > > > > > >>>>>>>>>>>>> feedback first before starting to work on it.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>> -Syed
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> --
> > > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > > > > >>>>>>>>> <http://solidfire.com/solution/overview/?video=play
> >*™
> > > > > > > >>>>>>>>> *
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> --
> > > > > > > >>>>>> *Mike Tutkowski*
> > > > > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > > > > > >>>>>> o: 303.746.7302
> > > > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > > > >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> --
> > > > > > > >>>>> *Mike Tutkowski*
> > > > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > > > > >>>>> o: 303.746.7302
> > > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > > >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> *Mike Tutkowski*
> > > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > >>> e: mike.tutkowski@solidfire.com
> > > > > > > >>> o: 303.746.7302
> > > > > > > >>> Advancing the way the world uses the cloud
> > > > > > > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Mike Tutkowski*
> > > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > > o: 303.746.7302
> > > > > > > > Advancing the way the world uses the cloud
> > > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > > >
> > > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > > > > related
> > > > > > services:
> > > > > > > IaaS Cloud Design & Build
> > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> |
> CSForge –
> > > > > > > rapid IaaS deployment framework <http://shapeblue.com/csforge/
> >
> > > > > > > CloudStack Consulting
> > > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > > CloudStack
> > > > > > > Software Engineering
> > > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > > CloudStack Infrastructure Support
> > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > > > > CloudStack Bootcamp Training Courses
> > > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > > Find out more about ShapeBlue and our range of CloudStack related
> > > services:
> > > > IaaS Cloud Design & Build
> > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> rapid
> > > > IaaS deployment framework <http://shapeblue.com/csforge/> CloudStack
> > > > Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack
> > > > Software Engineering
> > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > CloudStack Infrastructure Support
> > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> CloudStack
> > > > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/
> >
> > > >
> > > Find out more about ShapeBlue and our range of CloudStack related
> > services:
> > > IaaS Cloud Design & Build
> > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> > CloudStack
> > > Software Engineering
> > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > CloudStack Infrastructure Support
> > > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
> > >
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Pierre-Luc Dion <pd...@cloudops.com>]

That's fun to see that discussion happening. I 100% agree with Paul's
points of view. VolumeSnapshot are not a backup, but I do consider them as
a safety vest against Primary Storage failure, because failure append :-( .

The current proposal around snapshots that reside on the primary storage or
snapshots that end in the Secondary Storage  is not to address any kind of
backups requirement because a snapshot is not a backup, event an extracted
VM snapshot.

The main idea, and again this is for managed storage;

1. StorageSnapshotAPI: Provide storage side snapshot capability for fast
response time that support rollback to previous timestamp, create new
volume and maybe create template.
    not required to be a new API if the work is already done, I think this
is a different behaviors than the user expectation of a volume-snapshot.
2. VolumeSnapshotAPI: Provide current cloudstack behavior that create an
extraction of a volume into SecondaryStorage which can be reuse to create a
new volume into another Primary Storage. This type of snapshot is a slow
job since yes it would have to copy the full volume size on the Secondary
storage.


PL



On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <sy...@gmail.com>
wrote:

> I think I share you view on the 'Ideal world'. Backup (via Volume
> Snapshots) is a huge bottleneck in Cloudstack. This is amplified especially
> when you have a object storage as your secondary storage because it
> requires two copies (one to an NFS staging area and from there to object
> storage). And not to mention that all these copies are consuming hypervisor
> resources. Xenserver's Dom0 is also a huge bottleneck as all the Network
> and I/O flow through it. So our intention of proposing the "Storage
> Snapshots" is to give a better way of achiving snapshots while still
> keeping the original definition of volume snpashots (ie upload to sec
> storage).
>
> But as Erik pointed out volume snapshots are not backups. They don't work
> form multi-disk LVM volume groups and dynamic disks. I am all in for a
> better backup solution which handles these use cases and utilizes the
> storage's advanced features.
>
>
>
> On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <pa...@shapeblue.com>
> wrote:
>
> > In the beginning... there were CloudStack snapshots and they were
> actually
> > volume snapshots not hypervisor point-in-time snapshots.
> > Then VM snapshots were created (which are point-in-time hypervisor
> > snapshots) and we started referring to the original snapshots as volume
> > snapshots.
> >
> > CloudStack does not offer 'backups', but many people use volume snapshots
> > as backups. However you can't in-place restore volume snapshots and if
> you
> > have a VM with multiple volumes, the volume snapshots must be done in
> > series, meaning that the state across all of the volumes is unlikely to
> be
> > consistent.
> >
> > 'Actual Backups' would enable all of the restore options which users
> might
> > expect as well options as to where they might be stored. In my ideal
> world
> > they would also be able to leverage back-end hardware (such as Solidfire,
> > NetApp etc :) ) and software such as Veeam, Commvault etc to accelerate
> the
> > process.
> >
> >
> >
> > [image: ShapeBlue] <http://www.shapeblue.com>
> > Paul Angus
> > VP Technology ,  ShapeBlue
> > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > *+44 7711 418784* <+44%207711%20418784>
> > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > <pa...@cloudyangus>  |  w:
> > *www.shapeblue.com* <http://www.shapeblue.com>
> > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> > Services India LLP is a company incorporated in India and is operated
> under
> > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > company incorporated in Brasil and is operated under license from Shape
> > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
> is
> > a registered trademark.
> > This email and any attachments to it may be confidential and are intended
> > solely for the use of the individual to whom it is addressed. Any views
> or
> > opinions expressed are solely those of the author and do not necessarily
> > represent those of Shape Blue Ltd or related companies. If you are not
> the
> > intended recipient of this email, you must neither take any action based
> > upon its contents, nor copy or show it to anyone. Please contact the
> sender
> > if you believe you have received this email in error.
> >
> >
> > -----Original Message-----
> > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> > Sent: Friday, February 5, 2016 4:58 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [Propose][New Feature] Storage Snapshots
> >
> > Paul,
> >
> > When you say actual backups, how would it be different from the Volume
> > Snapshots that exist currently. My understanding is that Backups end up
> in
> > Sec Storage whereas Snapshots are just a point-in-time state of your
> volume
> > which can be restored back correct?
> >
> > -Syed
> >
> >
> > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <pa...@shapeblue.com>
> > wrote:
> >
> > > Hi Syed,
> > >
> > > As I understand it, the SolidFire plugin will export the snapshot to
> > > secondary storage if the user requests a template from the snapshot or
> > > wants to download the snapshot from the cloud. This is a good,
> > > pragmatic approach and yes Mike the SolidFire storage is super
> > > reliable and snapshots on SolidFire arrays take up next to no space.
> > > BUT I think that we are talking about a more general purpose API, and
> > > other storage systems may not be as awesome as Mike's. That's my
> > > concern. Also, the time to transfer for say 1TB to move from primary
> > > to sec storage and then create a VM template out of it may be too long
> > for users.
> > >
> > > @Mike I don’t think 'we' use the term volume snapshot for backup, it's
> > > just that users want to do backups and a volume snapshot is the only
> > > type of snapshot that copies the disk elsewhere and can be scheduled.
> > >
> > > I'm 'pondering' the implications of enabling actual backups (through
> > > recognised backup providers) and the user requirements around them
> > > (particularly restoration use cases) as a separate thread of work.
> > >
> > >
> > >
> > >
> > >
> > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP Technology
> > > , ShapeBlue
> > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > *+44 7711 418784* <+44%207711%20418784>
> > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > <pa...@cloudyangus> | w:
> > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue Ltd
> > > is a company incorporated in England & Wales. ShapeBlue Services India
> > > LLP is a company incorporated in India and is operated under license
> > > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company
> > > incorporated in Brasil and is operated under license from Shape Blue
> > > Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> > > South Africa and is traded under license from Shape Blue Ltd.
> > > ShapeBlue is a registered trademark.
> > > This email and any attachments to it may be confidential and are
> > > intended solely for the use of the individual to whom it is addressed.
> > > Any views or opinions expressed are solely those of the author and do
> > > not necessarily represent those of Shape Blue Ltd or related
> > > companies. If you are not the intended recipient of this email, you
> > > must neither take any action based upon its contents, nor copy or show
> > > it to anyone. Please contact the sender if you believe you have
> received
> > this email in error.
> > >
> > >
> > > -----Original Message-----
> > > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> > > Sent: 05 February 2016 15:31
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > >
> > > I think the terminology confusion comes from AWS where they do EBS
> > > snapshots backed up to S3 and CloudStack sort of followed that. And as
> > > an end user who is oblivious to the internals of my provider, my
> > > expectation would be something similar to what AWS as that is my
> > > biggest reference point.
> > >
> > > To your point Mike, I agree that a Primary Storage failure on
> > > SolidFire is unlikely, there are other motivations for us to push data
> > > to secondary storage. Primary storage (atleast for us) costs around 3
> > > times as much as secondary storage and snapshots on primary storage
> > > are rarely used (especially for some of our customers who do daily
> > backups).
> > >
> > >
> > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com> wrote:
> > >
> > > > Some of the weirdness is around terminology.
> > > >
> > > > For most systems I've worked on, a snapshot and a backup are two
> > > > completely different things (but CloudStack has traditionally used
> > > > the term "volume snapshot" to mean backup).
> > > >
> > > > I will put in a SolidFire "plug" here and say, though, that if your
> > > > primary storage is running on SolidFire that it is unlikely you'll
> > > > encounter an issue where your primary storage goes offline (and
> > > > you'll even maintain your performance guarantees during failure
> > > > scenarios and upgrades, as well). That being the case, it is less
> > > > useful to require a backup to Swift (but it's perfectly OK if that's
> > > > what we want to do
> > > here).
> > > >
> > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > > <sy...@gmail.com>
> > > > wrote:
> > > >
> > > > > Paul,
> > > > >
> > > > > I believe with the current implementation of Snapshots on managed
> > > > > storage
> > > > > (SolidFire) the snapshots are never exported to the secondary
> > storage.
> > > > > While this solves the problem of having snapshots taking forever
> > > > > to get to sec storage, this leaves us with a
> > > > huge
> > > > > liability if our primary storage goes down. We see snapshots as
> > > > > our recovery path because we store them in Swift which is reliable
> > > > > and resilient to failures.
> > > > >
> > > > > With Storage snpashots our goal is to have Volume snapshots always
> > > > > backed up to secondary storage and Storage Snapshots stay on the
> > > > > primary
> > > > storage.
> > > > > A provider could potentially mix both these and solve the problem
> > > > > that you mentioned where you want to meet user's expectation of a
> > > > > snapshot (ie backup to sec storage) while having an ability to
> > > > > utilize faster sanpshots (i.e. on the device)
> > > > >
> > > > > Hope this clarifies things.
> > > > >
> > > > > Thanks,
> > > > > -Syed
> > > > >
> > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > > <pa...@shapeblue.com>
> > > > > wrote:
> > > > >
> > > > > > HI guys,
> > > > > >
> > > > > > Could someone point me to the Jira bug of FS for the
> > > > > > SAN-snapshot
> > > > feature
> > > > > > in 4.6 which is mentioned.
> > > > > >
> > > > > > From my discussions with users and operators around snapshots
> > > > > > I'd make
> > > > > the
> > > > > > following observations:
> > > > > > a. 'users' use snapshots as backups (both long-term and short
> > > > > > term)
> > > > with
> > > > > > the expectation that they can use them for recovery if required.
> > > > > > b. operators fall back to snapshots if something has gone wrong
> > > > > > with primary storage.
> > > > > > c. users sometimes want to be able to export snapshots as well
> > > > > > as
> > > > create
> > > > > > new VMs from their snapshots
> > > > > > d. snapshots are a currently a massive pain for operators, I
> > > > > > know at
> > > > > least
> > > > > > one public cloud who have snapshots which take 2 days to
> complete.
> > > > > > e. snapshots (as they are) can't be used for multiple LVM disks.
> > > > > >
> > > > > > I think the process Mike has used in the SolidFire plugin (only
> > > > > > moving
> > > > > the
> > > > > > disk image to secondary storage when you absolutely have to) is
> > > > > > a very
> > > > > good
> > > > > > and pragmatic solution. I wonder what problems an operator might
> > > > > experience
> > > > > > if they have an issue with a given primary storage pool in a
> > cluster.
> > > > (I
> > > > > > know that that is REALLY unlikely in the SolidFire case Mike :)
> > > > > > ) And
> > > > if
> > > > > > the transfer from primary to secondary is slow, the time to
> > > > > > being able
> > > > to
> > > > > > create a template or export the volume will be slow.
> > > > > >
> > > > > > So for me the issue is around making sure that the end users
> > > > expectations
> > > > > > are met (while improving the speed/efficiency of the back end)
> > > > > >
> > > > > >
> > > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > > > Technology , ShapeBlue
> > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > > > <pa...@cloudyangus> | w:
> > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue
> > > > > > Ltd is a company incorporated in England & Wales. ShapeBlue
> > > > > > Services India LLP is a company incorporated in India and is
> > > > > > operated
> > > > > under
> > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> > > > > > is a company incorporated in Brasil and is operated under
> > > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > > > > registered by The Republic
> > > > of
> > > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > > ShapeBlue
> > > > > is
> > > > > > a registered trademark.
> > > > > > This email and any attachments to it may be confidential and are
> > > > intended
> > > > > > solely for the use of the individual to whom it is addressed.
> > > > > > Any views
> > > > > or
> > > > > > opinions expressed are solely those of the author and do not
> > > > necessarily
> > > > > > represent those of Shape Blue Ltd or related companies. If you
> > > > > > are not
> > > > > the
> > > > > > intended recipient of this email, you must neither take any
> > > > > > action
> > > > based
> > > > > > upon its contents, nor copy or show it to anyone. Please contact
> > > > > > the
> > > > > sender
> > > > > > if you believe you have received this email in error.
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > > >
> > > > > > Hi Mike,
> > > > > >
> > > > > > The idea of introducing a new API: StorageSnapshot for managed
> > > > > > storage
> > > > is
> > > > > > because the VolumeSnapshot default, or expected, behavior is to
> > > > > > archive snapshots into the Secondary Storage. So a
> > > > > > StorageSnapshot API would be
> > > > > for
> > > > > > snapshot that remain on the managed storage appliance.
> > > > > >
> > > > > > Quickly looking at the API doc and I don't see a strong
> > > > > > requirement for volume snapshots to be moved into secondary
> > > > > > storage. So, maybe StorageSnapshot API is not useful, but both
> > > > > > use cases are required. A snapshot that remain on the managed
> > > > > > storage, and another type of
> > > > snapshot
> > > > > > that end up into the secondary storage. Since you've done a lot
> > > > > > of
> > > > work,
> > > > > > might easier to just add a parameter to the current snapshot API
> > > > > > that
> > > > > would
> > > > > > trigger an extraction of the storage snapshot into the secondary
> > > > storage?
> > > > > >
> > > > > > PL
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > > mike.tutkowski@solidfire.com
> > > > > > > wrote:
> > > > > >
> > > > > > > I think that all sounds reasonable then - thanks!
> > > > > > >
> > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > > syed1.mushtaq@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> You are correct Mike in terms of the requirements. One of our
> > > > earlier
> > > > > > >> iterations on this was to have an argument to the create
> > > > > > >> snapshot
> > > > API
> > > > > > >> which decides whether to backup the volume to sec storage but
> > > > > > >> we realized it would make management of snapshots quite messy
> > > > > > >> so we proposed a new api instead.
> > > > > > >>
> > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > > >> <mi...@solidfire.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Hi,
> > > > > > >>>
> > > > > > >>> Just to make sure I understand all the requirements here:
> > > > > > >>>
> > > > > > >>> 1) This relates only to managed storage (1:1 mapping between
> > > > > > >>> a virtual disk and a backend SAN volume).
> > > > > > >>>
> > > > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > > > >>> functionality, which creates a snapshot on the SAN, and
> > > > > > >>> extend it via a config option (or
> > > > > > >>> something) to not only take the SAN snapshot, but to copy
> > > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > > >>>
> > > > > > >>> 3) The SAN snapshot is always taken. It's the backup to NFS
> > > > > > >>> that is optional.
> > > > > > >>>
> > > > > > >>> 4) Templates can be created from the snapshot that's on the
> > > > > > >>> SAN (already works).
> > > > > > >>>
> > > > > > >>> 5) CloudStack volumes can be created from the snapshot
> > > > > > >>> that's on
> > > > the
> > > > > > >>> SAN (already works as long as the new CloudStack volume ends
> > > > > > >>> up on the same primary storage).
> > > > > > >>>
> > > > > > >>> Would we have a need for a storage snapshot API then or
> > > > > > >>> would that just be the standard volume snapshot without the
> > > > > > >>> backup to
> > > NFS?
> > > > > > >>>
> > > > > > >>> Thanks!
> > > > > > >>> Mike
> > > > > > >>>
> > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > > >>> <sy...@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> Is it possible to have both functionalities (snapshot on
> > > > > > >>>> SAN & Sec
> > > > > > >>>> Storage) coexist? Because Ideally, we would like to have
> both.
> > > > > > >>>> For example, some of our customers want to implement their
> > > > > > >>>> own backup strategies and do encryption to their backups
> > > > > > >>>> which is a perfect use case for Storage Snapshot while our
> > > > > > >>>> other customers will still keep using the standard volume
> > > snapshot.
> > > > > > >>>>
> > > > > > >>>> To keep things backward compatible, we can add a setting
> > > > > > >>>> which
> > > > says
> > > > > > >>>> to not upload on secondary storage, because, after all, you
> > > > > > >>>> would take a SAN snapshot first when doing a Volume
> Snapshot.
> > > > > > >>>> You could stop the process there and not do the upload.
> > > > > > >>>>
> > > > > > >>>> What do you think about this approach?
> > > > > > >>>>
> > > > > > >>>> Thanks,
> > > > > > >>>> -Syed
> > > > > > >>>>
> > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > > >>>> mike.tutkowski@solidfire.com> wrote:
> > > > > > >>>>
> > > > > > >>>>> So, this is just me thinking out load here, but if a given
> > > > > > >>>>> CloudStack cloud doesn't actually need to provide both the
> > > > ability
> > > > > > >>>>> to take a SAN snapshot and export it to NFS (if just
> > > > > > >>>>> taking a SAN snapshot is OK), then we might be able to get
> > > > > > >>>>> away with no new
> > > > API
> > > > > > >>>>> calls and simply implement a new custom snapshot strategy
> > > > > > >>>>> and
> > > > data
> > > > > > >>>>> motion strategy to handle the case where the CloudStack
> > > > > > >>>>> cloud
> > > > does
> > > > > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > > > > >>>>>
> > > > > > >>>>> In other words, the "default" behavior would be to use the
> > > > > > >>>>> snapshot strategy and data motion strategy that we already
> > > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > > >>>>>
> > > > > > >>>>> If your CloudStack cloud, however, wants to take a SAN
> > > > > > >>>>> snapshot and have the data exported to NFS, then we could
> > > > > > >>>>> have you manipulate a Swing config file to make use of a
> > > > > > >>>>> new snapshot strategy and data motion strategy that
> > > > > > >>>>> performs both of these
> > > > > > activities.
> > > > > > >>>>>
> > > > > > >>>>> This way, the old behavior is still the default for users,
> > > > > > >>>>> but CloudStack admins can change this behavior via
> > > configuration.
> > > > > > >>>>>
> > > > > > >>>>> Thoughts?
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > > > >>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Right...I think we will need to come up with a viable
> > > > > > >>>>>> upgrade path or some reasonable way for them to move from
> > > > > > >>>>>> the old way to the new way (and some obvious way that
> > > > > > >>>>>> they will know they need
> > > > to
> > > > > > do this).
> > > > > > >>>>>>
> > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > > > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> I'm not really sure about the upgrade path however,
> > > > > > >>>>>>> customers who are using 4.6 and are on a managed storage
> > > > > > >>>>>>> would no longer have the same functionality with Volume
> > > Snapshots.
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > > > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> So if I understand correctly, currently taking a Volume
> > > > > > >>>>>>>> Snapshots of a volume on a managed storage keeps it on
> > > > > > >>>>>>>> the storage array. As a part of this feature, we can
> > > > > > >>>>>>>> make sure
> > > > that
> > > > > > >>>>>>>> Volume Snapshots on managed storage are uploaded to the
> > > > > > >>>>>>>> secondary storage. This would make the Volume Snapshot
> > > > > > >>>>>>>> feature behave the same regardless of the storage
> > > > > > >>>>>>>> (managed or
> > > > > > >>>>>>>> non-managed) And, for utilizing the efficient backend
> > > > > > >>>>>>>> storage
> > > > > > capabilities, we can use the new storage snapshots API.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > > > > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Hi everyone,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Whatever we do here, we need to have a plan to deal
> > > > > > >>>>>>>>> with the fact that we already have a feature (in 4.6
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > > >>>>>>>>> volume-snapshot APIs to create a volume snapshot (for
> > > > > > >>>>>>>>> managed
> > > > > > >>>>>>>>> storage) that resides on a backend SAN (using a custom
> > > > > > >>>>>>>>> snapshot strategy and a custom data motion strategy).
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> If these new APIs go in, then how should the original
> > > > > > >>>>>>>>> implementation (present in 4.6 and later) be changed?
> > > > > > >>>>>>>>> If it
> > > > is
> > > > > > >>>>>>>>> changed, how do we support customers who are already
> > > > > > >>>>>>>>> using
> > > > the
> > > > > > >>>>>>>>> original volume-snapshot API to take snapshots on a
> > > > > > >>>>>>>>> backend
> > > > > SAN?
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks,
> > > > > > >>>>>>>>> Mike
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > > > > >>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Will you be able to create a Template from a
> > > > StorageSnapshot?
> > > > > > >>>>>>>>>> If yes, will the template be stored in the secondary
> > > > > > >>>>>>>>>> storage like normal templates or will that be handled
> > > > > > >>>>>>>>>> somehow on the
> > > > > > vendor side?
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> *Will STEVENS*
> > > > > > >>>>>>>>>> Lead Developer
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > > > > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > > > > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira
> > > > > > >>>>>>>>>>>> ticket because it was not clear in the 'mention'
> > > > > > >>>>>>>>>>>> section because it shows as a page 'you do not have
> > > permission to'.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > > >>>>>>>>>>>> Lead Developer
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > > >>>>>>>>>>>> > wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Design Spec:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > > >>>>>>>>>>>>> 27
> > > > > > >>>>>>>>>>>>> 8
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Hi All,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do
> > > > > > >>>>>>>>>>>>> snapshots on managed storage backends like
> SolidFire.
> > > > > > >>>>>>>>>>>>> Snapshots on current managed storage stay on the
> > > > > > >>>>>>>>>>>>> device which is contrary to what CloudStack calls
> > > snpshots.
> > > > > > >>>>>>>>>>>>> But taking snapshots on storage and keeping it
> > > > > > >>>>>>>>>>>>> there has its own advantages
> > > > and
> > > > > > >>>>>>>>>>>>> we would ideally like to have both ways of doing
> > > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
> > > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> What do you guys think of this feature? I would
> > > > > > >>>>>>>>>>>>> love to have some feedback. I am working on making
> > > > > > >>>>>>>>>>>>> the design
> > > > spec
> > > > > > >>>>>>>>>>>>> more concrete but wanted to have a high level
> > > > > > >>>>>>>>>>>>> feedback first before starting to work on it.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>> -Syed
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> --
> > > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > > > > >>>>>>>>> o: 303.746.7302
> > > > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > > > >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™
> > > > > > >>>>>>>>> *
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> --
> > > > > > >>>>>> *Mike Tutkowski*
> > > > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > > > > >>>>>> o: 303.746.7302
> > > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > > >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> --
> > > > > > >>>>> *Mike Tutkowski*
> > > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > > > >>>>> o: 303.746.7302
> > > > > > >>>>> Advancing the way the world uses the cloud
> > > > > > >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> *Mike Tutkowski*
> > > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > >>> e: mike.tutkowski@solidfire.com
> > > > > > >>> o: 303.746.7302
> > > > > > >>> Advancing the way the world uses the cloud
> > > > > > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Mike Tutkowski*
> > > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > > e: mike.tutkowski@solidfire.com
> > > > > > > o: 303.746.7302
> > > > > > > Advancing the way the world uses the cloud
> > > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > > >
> > > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > > > related
> > > > > services:
> > > > > > IaaS Cloud Design & Build
> > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > > > > > rapid IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > > CloudStack Consulting
> > > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > > CloudStack
> > > > > > Software Engineering
> > > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > > CloudStack Infrastructure Support
> > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > > > CloudStack Bootcamp Training Courses
> > > > > > <http://shapeblue.com/cloudstack-training/>
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > > Find out more about ShapeBlue and our range of CloudStack related
> > services:
> > > IaaS Cloud Design & Build
> > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > > IaaS deployment framework <http://shapeblue.com/csforge/> CloudStack
> > > Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack
> > > Software Engineering
> > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > CloudStack Infrastructure Support
> > > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
> > >
> > Find out more about ShapeBlue and our range of CloudStack related
> services:
> > IaaS Cloud Design & Build
> > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > IaaS deployment framework <http://shapeblue.com/csforge/>
> > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack
> > Software Engineering
> > <http://shapeblue.com/cloudstack-software-engineering/>
> > CloudStack Infrastructure Support
> > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
> >
>

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

I think I share you view on the 'Ideal world'. Backup (via Volume
Snapshots) is a huge bottleneck in Cloudstack. This is amplified especially
when you have a object storage as your secondary storage because it
requires two copies (one to an NFS staging area and from there to object
storage). And not to mention that all these copies are consuming hypervisor
resources. Xenserver's Dom0 is also a huge bottleneck as all the Network
and I/O flow through it. So our intention of proposing the "Storage
Snapshots" is to give a better way of achiving snapshots while still
keeping the original definition of volume snpashots (ie upload to sec
storage).

But as Erik pointed out volume snapshots are not backups. They don't work
form multi-disk LVM volume groups and dynamic disks. I am all in for a
better backup solution which handles these use cases and utilizes the
storage's advanced features.



On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <pa...@shapeblue.com>
wrote:

> In the beginning... there were CloudStack snapshots and they were actually
> volume snapshots not hypervisor point-in-time snapshots.
> Then VM snapshots were created (which are point-in-time hypervisor
> snapshots) and we started referring to the original snapshots as volume
> snapshots.
>
> CloudStack does not offer 'backups', but many people use volume snapshots
> as backups. However you can't in-place restore volume snapshots and if you
> have a VM with multiple volumes, the volume snapshots must be done in
> series, meaning that the state across all of the volumes is unlikely to be
> consistent.
>
> 'Actual Backups' would enable all of the restore options which users might
> expect as well options as to where they might be stored. In my ideal world
> they would also be able to leverage back-end hardware (such as Solidfire,
> NetApp etc :) ) and software such as Veeam, Commvault etc to accelerate the
> process.
>
>
>
> [image: ShapeBlue] <http://www.shapeblue.com>
> Paul Angus
> VP Technology ,  ShapeBlue
> d:  *+44 203 617 0528 | s: +44 203 603 0540*
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> *+44 7711 418784* <+44%207711%20418784>
> e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> <pa...@cloudyangus>  |  w:
> *www.shapeblue.com* <http://www.shapeblue.com>
> a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> Services India LLP is a company incorporated in India and is operated under
> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> company incorporated in Brasil and is operated under license from Shape
> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
> a registered trademark.
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error.
>
>
> -----Original Message-----
> From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> Sent: Friday, February 5, 2016 4:58 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [Propose][New Feature] Storage Snapshots
>
> Paul,
>
> When you say actual backups, how would it be different from the Volume
> Snapshots that exist currently. My understanding is that Backups end up in
> Sec Storage whereas Snapshots are just a point-in-time state of your volume
> which can be restored back correct?
>
> -Syed
>
>
> On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <pa...@shapeblue.com>
> wrote:
>
> > Hi Syed,
> >
> > As I understand it, the SolidFire plugin will export the snapshot to
> > secondary storage if the user requests a template from the snapshot or
> > wants to download the snapshot from the cloud. This is a good,
> > pragmatic approach and yes Mike the SolidFire storage is super
> > reliable and snapshots on SolidFire arrays take up next to no space.
> > BUT I think that we are talking about a more general purpose API, and
> > other storage systems may not be as awesome as Mike's. That's my
> > concern. Also, the time to transfer for say 1TB to move from primary
> > to sec storage and then create a VM template out of it may be too long
> for users.
> >
> > @Mike I don’t think 'we' use the term volume snapshot for backup, it's
> > just that users want to do backups and a volume snapshot is the only
> > type of snapshot that copies the disk elsewhere and can be scheduled.
> >
> > I'm 'pondering' the implications of enabling actual backups (through
> > recognised backup providers) and the user requirements around them
> > (particularly restoration use cases) as a separate thread of work.
> >
> >
> >
> >
> >
> > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP Technology
> > , ShapeBlue
> > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > *+44 7711 418784* <+44%207711%20418784>
> > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > <pa...@cloudyangus> | w:
> > *www.shapeblue.com* <http://www.shapeblue.com>
> > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue Ltd
> > is a company incorporated in England & Wales. ShapeBlue Services India
> > LLP is a company incorporated in India and is operated under license
> > from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company
> > incorporated in Brasil and is operated under license from Shape Blue
> > Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> > South Africa and is traded under license from Shape Blue Ltd.
> > ShapeBlue is a registered trademark.
> > This email and any attachments to it may be confidential and are
> > intended solely for the use of the individual to whom it is addressed.
> > Any views or opinions expressed are solely those of the author and do
> > not necessarily represent those of Shape Blue Ltd or related
> > companies. If you are not the intended recipient of this email, you
> > must neither take any action based upon its contents, nor copy or show
> > it to anyone. Please contact the sender if you believe you have received
> this email in error.
> >
> >
> > -----Original Message-----
> > From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> > Sent: 05 February 2016 15:31
> > To: dev@cloudstack.apache.org
> > Subject: Re: [Propose][New Feature] Storage Snapshots
> >
> > I think the terminology confusion comes from AWS where they do EBS
> > snapshots backed up to S3 and CloudStack sort of followed that. And as
> > an end user who is oblivious to the internals of my provider, my
> > expectation would be something similar to what AWS as that is my
> > biggest reference point.
> >
> > To your point Mike, I agree that a Primary Storage failure on
> > SolidFire is unlikely, there are other motivations for us to push data
> > to secondary storage. Primary storage (atleast for us) costs around 3
> > times as much as secondary storage and snapshots on primary storage
> > are rarely used (especially for some of our customers who do daily
> backups).
> >
> >
> > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com> wrote:
> >
> > > Some of the weirdness is around terminology.
> > >
> > > For most systems I've worked on, a snapshot and a backup are two
> > > completely different things (but CloudStack has traditionally used
> > > the term "volume snapshot" to mean backup).
> > >
> > > I will put in a SolidFire "plug" here and say, though, that if your
> > > primary storage is running on SolidFire that it is unlikely you'll
> > > encounter an issue where your primary storage goes offline (and
> > > you'll even maintain your performance guarantees during failure
> > > scenarios and upgrades, as well). That being the case, it is less
> > > useful to require a backup to Swift (but it's perfectly OK if that's
> > > what we want to do
> > here).
> > >
> > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > > <sy...@gmail.com>
> > > wrote:
> > >
> > > > Paul,
> > > >
> > > > I believe with the current implementation of Snapshots on managed
> > > > storage
> > > > (SolidFire) the snapshots are never exported to the secondary
> storage.
> > > > While this solves the problem of having snapshots taking forever
> > > > to get to sec storage, this leaves us with a
> > > huge
> > > > liability if our primary storage goes down. We see snapshots as
> > > > our recovery path because we store them in Swift which is reliable
> > > > and resilient to failures.
> > > >
> > > > With Storage snpashots our goal is to have Volume snapshots always
> > > > backed up to secondary storage and Storage Snapshots stay on the
> > > > primary
> > > storage.
> > > > A provider could potentially mix both these and solve the problem
> > > > that you mentioned where you want to meet user's expectation of a
> > > > snapshot (ie backup to sec storage) while having an ability to
> > > > utilize faster sanpshots (i.e. on the device)
> > > >
> > > > Hope this clarifies things.
> > > >
> > > > Thanks,
> > > > -Syed
> > > >
> > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > > <pa...@shapeblue.com>
> > > > wrote:
> > > >
> > > > > HI guys,
> > > > >
> > > > > Could someone point me to the Jira bug of FS for the
> > > > > SAN-snapshot
> > > feature
> > > > > in 4.6 which is mentioned.
> > > > >
> > > > > From my discussions with users and operators around snapshots
> > > > > I'd make
> > > > the
> > > > > following observations:
> > > > > a. 'users' use snapshots as backups (both long-term and short
> > > > > term)
> > > with
> > > > > the expectation that they can use them for recovery if required.
> > > > > b. operators fall back to snapshots if something has gone wrong
> > > > > with primary storage.
> > > > > c. users sometimes want to be able to export snapshots as well
> > > > > as
> > > create
> > > > > new VMs from their snapshots
> > > > > d. snapshots are a currently a massive pain for operators, I
> > > > > know at
> > > > least
> > > > > one public cloud who have snapshots which take 2 days to complete.
> > > > > e. snapshots (as they are) can't be used for multiple LVM disks.
> > > > >
> > > > > I think the process Mike has used in the SolidFire plugin (only
> > > > > moving
> > > > the
> > > > > disk image to secondary storage when you absolutely have to) is
> > > > > a very
> > > > good
> > > > > and pragmatic solution. I wonder what problems an operator might
> > > > experience
> > > > > if they have an issue with a given primary storage pool in a
> cluster.
> > > (I
> > > > > know that that is REALLY unlikely in the SolidFire case Mike :)
> > > > > ) And
> > > if
> > > > > the transfer from primary to secondary is slow, the time to
> > > > > being able
> > > to
> > > > > create a template or export the volume will be slow.
> > > > >
> > > > > So for me the issue is around making sure that the end users
> > > expectations
> > > > > are met (while improving the speed/efficiency of the back end)
> > > > >
> > > > >
> > > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > > Technology , ShapeBlue
> > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > > *+44 7711 418784* <+44%207711%20418784>
> > > > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > > <pa...@cloudyangus> | w:
> > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue
> > > > > Ltd is a company incorporated in England & Wales. ShapeBlue
> > > > > Services India LLP is a company incorporated in India and is
> > > > > operated
> > > > under
> > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> > > > > is a company incorporated in Brasil and is operated under
> > > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > > > registered by The Republic
> > > of
> > > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > > ShapeBlue
> > > > is
> > > > > a registered trademark.
> > > > > This email and any attachments to it may be confidential and are
> > > intended
> > > > > solely for the use of the individual to whom it is addressed.
> > > > > Any views
> > > > or
> > > > > opinions expressed are solely those of the author and do not
> > > necessarily
> > > > > represent those of Shape Blue Ltd or related companies. If you
> > > > > are not
> > > > the
> > > > > intended recipient of this email, you must neither take any
> > > > > action
> > > based
> > > > > upon its contents, nor copy or show it to anyone. Please contact
> > > > > the
> > > > sender
> > > > > if you believe you have received this email in error.
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > > >
> > > > > Hi Mike,
> > > > >
> > > > > The idea of introducing a new API: StorageSnapshot for managed
> > > > > storage
> > > is
> > > > > because the VolumeSnapshot default, or expected, behavior is to
> > > > > archive snapshots into the Secondary Storage. So a
> > > > > StorageSnapshot API would be
> > > > for
> > > > > snapshot that remain on the managed storage appliance.
> > > > >
> > > > > Quickly looking at the API doc and I don't see a strong
> > > > > requirement for volume snapshots to be moved into secondary
> > > > > storage. So, maybe StorageSnapshot API is not useful, but both
> > > > > use cases are required. A snapshot that remain on the managed
> > > > > storage, and another type of
> > > snapshot
> > > > > that end up into the secondary storage. Since you've done a lot
> > > > > of
> > > work,
> > > > > might easier to just add a parameter to the current snapshot API
> > > > > that
> > > > would
> > > > > trigger an extraction of the storage snapshot into the secondary
> > > storage?
> > > > >
> > > > > PL
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > > mike.tutkowski@solidfire.com
> > > > > > wrote:
> > > > >
> > > > > > I think that all sounds reasonable then - thanks!
> > > > > >
> > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > > syed1.mushtaq@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> You are correct Mike in terms of the requirements. One of our
> > > earlier
> > > > > >> iterations on this was to have an argument to the create
> > > > > >> snapshot
> > > API
> > > > > >> which decides whether to backup the volume to sec storage but
> > > > > >> we realized it would make management of snapshots quite messy
> > > > > >> so we proposed a new api instead.
> > > > > >>
> > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > > >> <mi...@solidfire.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hi,
> > > > > >>>
> > > > > >>> Just to make sure I understand all the requirements here:
> > > > > >>>
> > > > > >>> 1) This relates only to managed storage (1:1 mapping between
> > > > > >>> a virtual disk and a backend SAN volume).
> > > > > >>>
> > > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > > >>> functionality, which creates a snapshot on the SAN, and
> > > > > >>> extend it via a config option (or
> > > > > >>> something) to not only take the SAN snapshot, but to copy
> > > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > > >>>
> > > > > >>> 3) The SAN snapshot is always taken. It's the backup to NFS
> > > > > >>> that is optional.
> > > > > >>>
> > > > > >>> 4) Templates can be created from the snapshot that's on the
> > > > > >>> SAN (already works).
> > > > > >>>
> > > > > >>> 5) CloudStack volumes can be created from the snapshot
> > > > > >>> that's on
> > > the
> > > > > >>> SAN (already works as long as the new CloudStack volume ends
> > > > > >>> up on the same primary storage).
> > > > > >>>
> > > > > >>> Would we have a need for a storage snapshot API then or
> > > > > >>> would that just be the standard volume snapshot without the
> > > > > >>> backup to
> > NFS?
> > > > > >>>
> > > > > >>> Thanks!
> > > > > >>> Mike
> > > > > >>>
> > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > > >>> <sy...@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Is it possible to have both functionalities (snapshot on
> > > > > >>>> SAN & Sec
> > > > > >>>> Storage) coexist? Because Ideally, we would like to have both.
> > > > > >>>> For example, some of our customers want to implement their
> > > > > >>>> own backup strategies and do encryption to their backups
> > > > > >>>> which is a perfect use case for Storage Snapshot while our
> > > > > >>>> other customers will still keep using the standard volume
> > snapshot.
> > > > > >>>>
> > > > > >>>> To keep things backward compatible, we can add a setting
> > > > > >>>> which
> > > says
> > > > > >>>> to not upload on secondary storage, because, after all, you
> > > > > >>>> would take a SAN snapshot first when doing a Volume Snapshot.
> > > > > >>>> You could stop the process there and not do the upload.
> > > > > >>>>
> > > > > >>>> What do you think about this approach?
> > > > > >>>>
> > > > > >>>> Thanks,
> > > > > >>>> -Syed
> > > > > >>>>
> > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > > >>>> mike.tutkowski@solidfire.com> wrote:
> > > > > >>>>
> > > > > >>>>> So, this is just me thinking out load here, but if a given
> > > > > >>>>> CloudStack cloud doesn't actually need to provide both the
> > > ability
> > > > > >>>>> to take a SAN snapshot and export it to NFS (if just
> > > > > >>>>> taking a SAN snapshot is OK), then we might be able to get
> > > > > >>>>> away with no new
> > > API
> > > > > >>>>> calls and simply implement a new custom snapshot strategy
> > > > > >>>>> and
> > > data
> > > > > >>>>> motion strategy to handle the case where the CloudStack
> > > > > >>>>> cloud
> > > does
> > > > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > > > >>>>>
> > > > > >>>>> In other words, the "default" behavior would be to use the
> > > > > >>>>> snapshot strategy and data motion strategy that we already
> > > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > > >>>>>
> > > > > >>>>> If your CloudStack cloud, however, wants to take a SAN
> > > > > >>>>> snapshot and have the data exported to NFS, then we could
> > > > > >>>>> have you manipulate a Swing config file to make use of a
> > > > > >>>>> new snapshot strategy and data motion strategy that
> > > > > >>>>> performs both of these
> > > > > activities.
> > > > > >>>>>
> > > > > >>>>> This way, the old behavior is still the default for users,
> > > > > >>>>> but CloudStack admins can change this behavior via
> > configuration.
> > > > > >>>>>
> > > > > >>>>> Thoughts?
> > > > > >>>>>
> > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > > >>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > >>>>>
> > > > > >>>>>> Right...I think we will need to come up with a viable
> > > > > >>>>>> upgrade path or some reasonable way for them to move from
> > > > > >>>>>> the old way to the new way (and some obvious way that
> > > > > >>>>>> they will know they need
> > > to
> > > > > do this).
> > > > > >>>>>>
> > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> I'm not really sure about the upgrade path however,
> > > > > >>>>>>> customers who are using 4.6 and are on a managed storage
> > > > > >>>>>>> would no longer have the same functionality with Volume
> > Snapshots.
> > > > > >>>>>>>
> > > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> So if I understand correctly, currently taking a Volume
> > > > > >>>>>>>> Snapshots of a volume on a managed storage keeps it on
> > > > > >>>>>>>> the storage array. As a part of this feature, we can
> > > > > >>>>>>>> make sure
> > > that
> > > > > >>>>>>>> Volume Snapshots on managed storage are uploaded to the
> > > > > >>>>>>>> secondary storage. This would make the Volume Snapshot
> > > > > >>>>>>>> feature behave the same regardless of the storage
> > > > > >>>>>>>> (managed or
> > > > > >>>>>>>> non-managed) And, for utilizing the efficient backend
> > > > > >>>>>>>> storage
> > > > > capabilities, we can use the new storage snapshots API.
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > > > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Hi everyone,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Whatever we do here, we need to have a plan to deal
> > > > > >>>>>>>>> with the fact that we already have a feature (in 4.6
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>> later) that allows you to use the existing
> > > > > >>>>>>>>> volume-snapshot APIs to create a volume snapshot (for
> > > > > >>>>>>>>> managed
> > > > > >>>>>>>>> storage) that resides on a backend SAN (using a custom
> > > > > >>>>>>>>> snapshot strategy and a custom data motion strategy).
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> If these new APIs go in, then how should the original
> > > > > >>>>>>>>> implementation (present in 4.6 and later) be changed?
> > > > > >>>>>>>>> If it
> > > is
> > > > > >>>>>>>>> changed, how do we support customers who are already
> > > > > >>>>>>>>> using
> > > the
> > > > > >>>>>>>>> original volume-snapshot API to take snapshots on a
> > > > > >>>>>>>>> backend
> > > > SAN?
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks,
> > > > > >>>>>>>>> Mike
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > > > >>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Will you be able to create a Template from a
> > > StorageSnapshot?
> > > > > >>>>>>>>>> If yes, will the template be stored in the secondary
> > > > > >>>>>>>>>> storage like normal templates or will that be handled
> > > > > >>>>>>>>>> somehow on the
> > > > > vendor side?
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> *Will STEVENS*
> > > > > >>>>>>>>>> Lead Developer
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > > > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks Will!!!
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > > > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira
> > > > > >>>>>>>>>>>> ticket because it was not clear in the 'mention'
> > > > > >>>>>>>>>>>> section because it shows as a page 'you do not have
> > permission to'.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> *Will STEVENS*
> > > > > >>>>>>>>>>>> Lead Developer
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > > >>>>>>>>>>>> > wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Design Spec:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Jira Ticket
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > > >>>>>>>>>>>>> 27
> > > > > >>>>>>>>>>>>> 8
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Hi All,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do
> > > > > >>>>>>>>>>>>> snapshots on managed storage backends like SolidFire.
> > > > > >>>>>>>>>>>>> Snapshots on current managed storage stay on the
> > > > > >>>>>>>>>>>>> device which is contrary to what CloudStack calls
> > snpshots.
> > > > > >>>>>>>>>>>>> But taking snapshots on storage and keeping it
> > > > > >>>>>>>>>>>>> there has its own advantages
> > > and
> > > > > >>>>>>>>>>>>> we would ideally like to have both ways of doing
> > > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
> > > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> What do you guys think of this feature? I would
> > > > > >>>>>>>>>>>>> love to have some feedback. I am working on making
> > > > > >>>>>>>>>>>>> the design
> > > spec
> > > > > >>>>>>>>>>>>> more concrete but wanted to have a high level
> > > > > >>>>>>>>>>>>> feedback first before starting to work on it.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>> -Syed
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> --
> > > > > >>>>>>>>> *Mike Tutkowski*
> > > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > > > >>>>>>>>> o: 303.746.7302
> > > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > > >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™
> > > > > >>>>>>>>> *
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> --
> > > > > >>>>>> *Mike Tutkowski*
> > > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > > > >>>>>> o: 303.746.7302
> > > > > >>>>>> Advancing the way the world uses the cloud
> > > > > >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> --
> > > > > >>>>> *Mike Tutkowski*
> > > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > > >>>>> o: 303.746.7302
> > > > > >>>>> Advancing the way the world uses the cloud
> > > > > >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> *Mike Tutkowski*
> > > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > > >>> e: mike.tutkowski@solidfire.com
> > > > > >>> o: 303.746.7302
> > > > > >>> Advancing the way the world uses the cloud
> > > > > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >>>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Mike Tutkowski*
> > > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > > e: mike.tutkowski@solidfire.com
> > > > > > o: 303.746.7302
> > > > > > Advancing the way the world uses the cloud
> > > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > > >
> > > > > Find out more about ShapeBlue and our range of CloudStack
> > > > > related
> > > > services:
> > > > > IaaS Cloud Design & Build
> > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > > > > rapid IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > > CloudStack Consulting
> > > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > > CloudStack
> > > > > Software Engineering
> > > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > > CloudStack Infrastructure Support
> > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > > CloudStack Bootcamp Training Courses
> > > > > <http://shapeblue.com/cloudstack-training/>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> > Find out more about ShapeBlue and our range of CloudStack related
> services:
> > IaaS Cloud Design & Build
> > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > IaaS deployment framework <http://shapeblue.com/csforge/> CloudStack
> > Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack
> > Software Engineering
> > <http://shapeblue.com/cloudstack-software-engineering/>
> > CloudStack Infrastructure Support
> > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
> >
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack
> Software Engineering
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support
> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
>

RE: [Propose][New Feature] Storage Snapshots

[Paul Angus <pa...@shapeblue.com>]

In the beginning... there were CloudStack snapshots and they were actually volume snapshots not hypervisor point-in-time snapshots.
Then VM snapshots were created (which are point-in-time hypervisor snapshots) and we started referring to the original snapshots as volume snapshots.

CloudStack does not offer 'backups', but many people use volume snapshots as backups. However you can't in-place restore volume snapshots and if you have a VM with multiple volumes, the volume snapshots must be done in series, meaning that the state across all of the volumes is unlikely to be consistent.

'Actual Backups' would enable all of the restore options which users might expect as well options as to where they might be stored. In my ideal world they would also be able to leverage back-end hardware (such as Solidfire, NetApp etc :) ) and software such as Veeam, Commvault etc to accelerate the process.



[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology   ,       ShapeBlue


d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>     |      m:      +44 7711 418784<tel:+44%207711%20418784>

e:      paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus>      |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:image20867f.png@a1abedce.4198548d]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




-----Original Message-----
From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
Sent: Friday, February 5, 2016 4:58 PM
To: dev@cloudstack.apache.org
Subject: Re: [Propose][New Feature] Storage Snapshots

Paul,

When you say actual backups, how would it be different from the Volume Snapshots that exist currently. My understanding is that Backups end up in Sec Storage whereas Snapshots are just a point-in-time state of your volume which can be restored back correct?

-Syed


On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <pa...@shapeblue.com>
wrote:

> Hi Syed,
>
> As I understand it, the SolidFire plugin will export the snapshot to
> secondary storage if the user requests a template from the snapshot or
> wants to download the snapshot from the cloud. This is a good,
> pragmatic approach and yes Mike the SolidFire storage is super
> reliable and snapshots on SolidFire arrays take up next to no space.
> BUT I think that we are talking about a more general purpose API, and
> other storage systems may not be as awesome as Mike's. That's my
> concern. Also, the time to transfer for say 1TB to move from primary
> to sec storage and then create a VM template out of it may be too long for users.
>
> @Mike I don’t think 'we' use the term volume snapshot for backup, it's
> just that users want to do backups and a volume snapshot is the only
> type of snapshot that copies the disk elsewhere and can be scheduled.
>
> I'm 'pondering' the implications of enabling actual backups (through
> recognised backup providers) and the user requirements around them
> (particularly restoration use cases) as a separate thread of work.
>
>
>
>
>
> [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP Technology
> , ShapeBlue
> d: *+44 203 617 0528 | s: +44 203 603 0540*
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> *+44 7711 418784* <+44%207711%20418784>
> e: *paul.angus@shapeblue.com | t: @cloudyangus*
> <pa...@cloudyangus> | w:
> *www.shapeblue.com* <http://www.shapeblue.com>
> a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue Ltd
> is a company incorporated in England & Wales. ShapeBlue Services India
> LLP is a company incorporated in India and is operated under license
> from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company
> incorporated in Brasil and is operated under license from Shape Blue
> Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> South Africa and is traded under license from Shape Blue Ltd.
> ShapeBlue is a registered trademark.
> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed.
> Any views or opinions expressed are solely those of the author and do
> not necessarily represent those of Shape Blue Ltd or related
> companies. If you are not the intended recipient of this email, you
> must neither take any action based upon its contents, nor copy or show
> it to anyone. Please contact the sender if you believe you have received this email in error.
>
>
> -----Original Message-----
> From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> Sent: 05 February 2016 15:31
> To: dev@cloudstack.apache.org
> Subject: Re: [Propose][New Feature] Storage Snapshots
>
> I think the terminology confusion comes from AWS where they do EBS
> snapshots backed up to S3 and CloudStack sort of followed that. And as
> an end user who is oblivious to the internals of my provider, my
> expectation would be something similar to what AWS as that is my
> biggest reference point.
>
> To your point Mike, I agree that a Primary Storage failure on
> SolidFire is unlikely, there are other motivations for us to push data
> to secondary storage. Primary storage (atleast for us) costs around 3
> times as much as secondary storage and snapshots on primary storage
> are rarely used (especially for some of our customers who do daily backups).
>
>
> On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
> > Some of the weirdness is around terminology.
> >
> > For most systems I've worked on, a snapshot and a backup are two
> > completely different things (but CloudStack has traditionally used
> > the term "volume snapshot" to mean backup).
> >
> > I will put in a SolidFire "plug" here and say, though, that if your
> > primary storage is running on SolidFire that it is unlikely you'll
> > encounter an issue where your primary storage goes offline (and
> > you'll even maintain your performance guarantees during failure
> > scenarios and upgrades, as well). That being the case, it is less
> > useful to require a backup to Swift (but it's perfectly OK if that's
> > what we want to do
> here).
> >
> > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > <sy...@gmail.com>
> > wrote:
> >
> > > Paul,
> > >
> > > I believe with the current implementation of Snapshots on managed
> > > storage
> > > (SolidFire) the snapshots are never exported to the secondary storage.
> > > While this solves the problem of having snapshots taking forever
> > > to get to sec storage, this leaves us with a
> > huge
> > > liability if our primary storage goes down. We see snapshots as
> > > our recovery path because we store them in Swift which is reliable
> > > and resilient to failures.
> > >
> > > With Storage snpashots our goal is to have Volume snapshots always
> > > backed up to secondary storage and Storage Snapshots stay on the
> > > primary
> > storage.
> > > A provider could potentially mix both these and solve the problem
> > > that you mentioned where you want to meet user's expectation of a
> > > snapshot (ie backup to sec storage) while having an ability to
> > > utilize faster sanpshots (i.e. on the device)
> > >
> > > Hope this clarifies things.
> > >
> > > Thanks,
> > > -Syed
> > >
> > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > <pa...@shapeblue.com>
> > > wrote:
> > >
> > > > HI guys,
> > > >
> > > > Could someone point me to the Jira bug of FS for the
> > > > SAN-snapshot
> > feature
> > > > in 4.6 which is mentioned.
> > > >
> > > > From my discussions with users and operators around snapshots
> > > > I'd make
> > > the
> > > > following observations:
> > > > a. 'users' use snapshots as backups (both long-term and short
> > > > term)
> > with
> > > > the expectation that they can use them for recovery if required.
> > > > b. operators fall back to snapshots if something has gone wrong
> > > > with primary storage.
> > > > c. users sometimes want to be able to export snapshots as well
> > > > as
> > create
> > > > new VMs from their snapshots
> > > > d. snapshots are a currently a massive pain for operators, I
> > > > know at
> > > least
> > > > one public cloud who have snapshots which take 2 days to complete.
> > > > e. snapshots (as they are) can't be used for multiple LVM disks.
> > > >
> > > > I think the process Mike has used in the SolidFire plugin (only
> > > > moving
> > > the
> > > > disk image to secondary storage when you absolutely have to) is
> > > > a very
> > > good
> > > > and pragmatic solution. I wonder what problems an operator might
> > > experience
> > > > if they have an issue with a given primary storage pool in a cluster.
> > (I
> > > > know that that is REALLY unlikely in the SolidFire case Mike :)
> > > > ) And
> > if
> > > > the transfer from primary to secondary is slow, the time to
> > > > being able
> > to
> > > > create a template or export the volume will be slow.
> > > >
> > > > So for me the issue is around making sure that the end users
> > expectations
> > > > are met (while improving the speed/efficiency of the back end)
> > > >
> > > >
> > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > Technology , ShapeBlue
> > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > *+44 7711 418784* <+44%207711%20418784>
> > > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > <pa...@cloudyangus> | w:
> > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue
> > > > Ltd is a company incorporated in England & Wales. ShapeBlue
> > > > Services India LLP is a company incorporated in India and is
> > > > operated
> > > under
> > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> > > > is a company incorporated in Brasil and is operated under
> > > > license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
> > > > registered by The Republic
> > of
> > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > ShapeBlue
> > > is
> > > > a registered trademark.
> > > > This email and any attachments to it may be confidential and are
> > intended
> > > > solely for the use of the individual to whom it is addressed.
> > > > Any views
> > > or
> > > > opinions expressed are solely those of the author and do not
> > necessarily
> > > > represent those of Shape Blue Ltd or related companies. If you
> > > > are not
> > > the
> > > > intended recipient of this email, you must neither take any
> > > > action
> > based
> > > > upon its contents, nor copy or show it to anyone. Please contact
> > > > the
> > > sender
> > > > if you believe you have received this email in error.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > >
> > > > Hi Mike,
> > > >
> > > > The idea of introducing a new API: StorageSnapshot for managed
> > > > storage
> > is
> > > > because the VolumeSnapshot default, or expected, behavior is to
> > > > archive snapshots into the Secondary Storage. So a
> > > > StorageSnapshot API would be
> > > for
> > > > snapshot that remain on the managed storage appliance.
> > > >
> > > > Quickly looking at the API doc and I don't see a strong
> > > > requirement for volume snapshots to be moved into secondary
> > > > storage. So, maybe StorageSnapshot API is not useful, but both
> > > > use cases are required. A snapshot that remain on the managed
> > > > storage, and another type of
> > snapshot
> > > > that end up into the secondary storage. Since you've done a lot
> > > > of
> > work,
> > > > might easier to just add a parameter to the current snapshot API
> > > > that
> > > would
> > > > trigger an extraction of the storage snapshot into the secondary
> > storage?
> > > >
> > > > PL
> > > >
> > > >
> > > >
> > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com
> > > > > wrote:
> > > >
> > > > > I think that all sounds reasonable then - thanks!
> > > > >
> > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > syed1.mushtaq@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> You are correct Mike in terms of the requirements. One of our
> > earlier
> > > > >> iterations on this was to have an argument to the create
> > > > >> snapshot
> > API
> > > > >> which decides whether to backup the volume to sec storage but
> > > > >> we realized it would make management of snapshots quite messy
> > > > >> so we proposed a new api instead.
> > > > >>
> > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > >> <mi...@solidfire.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> Just to make sure I understand all the requirements here:
> > > > >>>
> > > > >>> 1) This relates only to managed storage (1:1 mapping between
> > > > >>> a virtual disk and a backend SAN volume).
> > > > >>>
> > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > >>> functionality, which creates a snapshot on the SAN, and
> > > > >>> extend it via a config option (or
> > > > >>> something) to not only take the SAN snapshot, but to copy
> > > > >>> the underlying VHD (XenServer only) to NFS.
> > > > >>>
> > > > >>> 3) The SAN snapshot is always taken. It's the backup to NFS
> > > > >>> that is optional.
> > > > >>>
> > > > >>> 4) Templates can be created from the snapshot that's on the
> > > > >>> SAN (already works).
> > > > >>>
> > > > >>> 5) CloudStack volumes can be created from the snapshot
> > > > >>> that's on
> > the
> > > > >>> SAN (already works as long as the new CloudStack volume ends
> > > > >>> up on the same primary storage).
> > > > >>>
> > > > >>> Would we have a need for a storage snapshot API then or
> > > > >>> would that just be the standard volume snapshot without the
> > > > >>> backup to
> NFS?
> > > > >>>
> > > > >>> Thanks!
> > > > >>> Mike
> > > > >>>
> > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > >>> <sy...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Is it possible to have both functionalities (snapshot on
> > > > >>>> SAN & Sec
> > > > >>>> Storage) coexist? Because Ideally, we would like to have both.
> > > > >>>> For example, some of our customers want to implement their
> > > > >>>> own backup strategies and do encryption to their backups
> > > > >>>> which is a perfect use case for Storage Snapshot while our
> > > > >>>> other customers will still keep using the standard volume
> snapshot.
> > > > >>>>
> > > > >>>> To keep things backward compatible, we can add a setting
> > > > >>>> which
> > says
> > > > >>>> to not upload on secondary storage, because, after all, you
> > > > >>>> would take a SAN snapshot first when doing a Volume Snapshot.
> > > > >>>> You could stop the process there and not do the upload.
> > > > >>>>
> > > > >>>> What do you think about this approach?
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> -Syed
> > > > >>>>
> > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > >>>> mike.tutkowski@solidfire.com> wrote:
> > > > >>>>
> > > > >>>>> So, this is just me thinking out load here, but if a given
> > > > >>>>> CloudStack cloud doesn't actually need to provide both the
> > ability
> > > > >>>>> to take a SAN snapshot and export it to NFS (if just
> > > > >>>>> taking a SAN snapshot is OK), then we might be able to get
> > > > >>>>> away with no new
> > API
> > > > >>>>> calls and simply implement a new custom snapshot strategy
> > > > >>>>> and
> > data
> > > > >>>>> motion strategy to handle the case where the CloudStack
> > > > >>>>> cloud
> > does
> > > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > > >>>>>
> > > > >>>>> In other words, the "default" behavior would be to use the
> > > > >>>>> snapshot strategy and data motion strategy that we already
> > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > >>>>>
> > > > >>>>> If your CloudStack cloud, however, wants to take a SAN
> > > > >>>>> snapshot and have the data exported to NFS, then we could
> > > > >>>>> have you manipulate a Swing config file to make use of a
> > > > >>>>> new snapshot strategy and data motion strategy that
> > > > >>>>> performs both of these
> > > > activities.
> > > > >>>>>
> > > > >>>>> This way, the old behavior is still the default for users,
> > > > >>>>> but CloudStack admins can change this behavior via
> configuration.
> > > > >>>>>
> > > > >>>>> Thoughts?
> > > > >>>>>
> > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > >>>>> mike.tutkowski@solidfire.com> wrote:
> > > > >>>>>
> > > > >>>>>> Right...I think we will need to come up with a viable
> > > > >>>>>> upgrade path or some reasonable way for them to move from
> > > > >>>>>> the old way to the new way (and some obvious way that
> > > > >>>>>> they will know they need
> > to
> > > > do this).
> > > > >>>>>>
> > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > >>>>>>
> > > > >>>>>>> I'm not really sure about the upgrade path however,
> > > > >>>>>>> customers who are using 4.6 and are on a managed storage
> > > > >>>>>>> would no longer have the same functionality with Volume
> Snapshots.
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> So if I understand correctly, currently taking a Volume
> > > > >>>>>>>> Snapshots of a volume on a managed storage keeps it on
> > > > >>>>>>>> the storage array. As a part of this feature, we can
> > > > >>>>>>>> make sure
> > that
> > > > >>>>>>>> Volume Snapshots on managed storage are uploaded to the
> > > > >>>>>>>> secondary storage. This would make the Volume Snapshot
> > > > >>>>>>>> feature behave the same regardless of the storage
> > > > >>>>>>>> (managed or
> > > > >>>>>>>> non-managed) And, for utilizing the efficient backend
> > > > >>>>>>>> storage
> > > > capabilities, we can use the new storage snapshots API.
> > > > >>>>>>>>
> > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi everyone,
> > > > >>>>>>>>>
> > > > >>>>>>>>> Whatever we do here, we need to have a plan to deal
> > > > >>>>>>>>> with the fact that we already have a feature (in 4.6
> > > > >>>>>>>>> and
> > > > >>>>>>>>> later) that allows you to use the existing
> > > > >>>>>>>>> volume-snapshot APIs to create a volume snapshot (for
> > > > >>>>>>>>> managed
> > > > >>>>>>>>> storage) that resides on a backend SAN (using a custom
> > > > >>>>>>>>> snapshot strategy and a custom data motion strategy).
> > > > >>>>>>>>>
> > > > >>>>>>>>> If these new APIs go in, then how should the original
> > > > >>>>>>>>> implementation (present in 4.6 and later) be changed?
> > > > >>>>>>>>> If it
> > is
> > > > >>>>>>>>> changed, how do we support customers who are already
> > > > >>>>>>>>> using
> > the
> > > > >>>>>>>>> original volume-snapshot API to take snapshots on a
> > > > >>>>>>>>> backend
> > > SAN?
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks,
> > > > >>>>>>>>> Mike
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > > >>>>>>>>> wstevens@cloudops.com> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Will you be able to create a Template from a
> > StorageSnapshot?
> > > > >>>>>>>>>> If yes, will the template be stored in the secondary
> > > > >>>>>>>>>> storage like normal templates or will that be handled
> > > > >>>>>>>>>> somehow on the
> > > > vendor side?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> *Will STEVENS*
> > > > >>>>>>>>>> Lead Developer
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Thanks Will!!!
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira
> > > > >>>>>>>>>>>> ticket because it was not clear in the 'mention'
> > > > >>>>>>>>>>>> section because it shows as a page 'you do not have
> permission to'.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> *Will STEVENS*
> > > > >>>>>>>>>>>> Lead Developer
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > >>>>>>>>>>>> > wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Design Spec:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Jira Ticket
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9
> > > > >>>>>>>>>>>>> 27
> > > > >>>>>>>>>>>>> 8
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Hi All,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do
> > > > >>>>>>>>>>>>> snapshots on managed storage backends like SolidFire.
> > > > >>>>>>>>>>>>> Snapshots on current managed storage stay on the
> > > > >>>>>>>>>>>>> device which is contrary to what CloudStack calls
> snpshots.
> > > > >>>>>>>>>>>>> But taking snapshots on storage and keeping it
> > > > >>>>>>>>>>>>> there has its own advantages
> > and
> > > > >>>>>>>>>>>>> we would ideally like to have both ways of doing
> > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
> > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> What do you guys think of this feature? I would
> > > > >>>>>>>>>>>>> love to have some feedback. I am working on making
> > > > >>>>>>>>>>>>> the design
> > spec
> > > > >>>>>>>>>>>>> more concrete but wanted to have a high level
> > > > >>>>>>>>>>>>> feedback first before starting to work on it.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>> -Syed
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> --
> > > > >>>>>>>>> *Mike Tutkowski*
> > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > > >>>>>>>>> o: 303.746.7302
> > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™
> > > > >>>>>>>>> *
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>> *Mike Tutkowski*
> > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > > >>>>>> o: 303.746.7302
> > > > >>>>>> Advancing the way the world uses the cloud
> > > > >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> *Mike Tutkowski*
> > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > >>>>> o: 303.746.7302
> > > > >>>>> Advancing the way the world uses the cloud
> > > > >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> *Mike Tutkowski*
> > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > >>> e: mike.tutkowski@solidfire.com
> > > > >>> o: 303.746.7302
> > > > >>> Advancing the way the world uses the cloud
> > > > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > > Find out more about ShapeBlue and our range of CloudStack
> > > > related
> > > services:
> > > > IaaS Cloud Design & Build
> > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > > > rapid IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > CloudStack Consulting
> > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > CloudStack
> > > > Software Engineering
> > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > CloudStack Infrastructure Support
> > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > CloudStack Bootcamp Training Courses
> > > > <http://shapeblue.com/cloudstack-training/>
> > > >
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework <http://shapeblue.com/csforge/> CloudStack
> Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack
> Software Engineering
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support
> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

Paul,

When you say actual backups, how would it be different from the Volume
Snapshots that exist currently. My understanding is that Backups end up in
Sec Storage whereas Snapshots are just a point-in-time state of your volume
which can be restored back correct?

-Syed


On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <pa...@shapeblue.com>
wrote:

> Hi Syed,
>
> As I understand it, the SolidFire plugin will export the snapshot to
> secondary storage if the user requests a template from the snapshot or
> wants to download the snapshot from the cloud. This is a good, pragmatic
> approach and yes Mike the SolidFire storage is super reliable and snapshots
> on SolidFire arrays take up next to no space. BUT I think that we are
> talking about a more general purpose API, and other storage systems may not
> be as awesome as Mike's. That's my concern. Also, the time to transfer for
> say 1TB to move from primary to sec storage and then create a VM template
> out of it may be too long for users.
>
> @Mike I don’t think 'we' use the term volume snapshot for backup, it's
> just that users want to do backups and a volume snapshot is the only type
> of snapshot that copies the disk elsewhere and can be scheduled.
>
> I'm 'pondering' the implications of enabling actual backups (through
> recognised backup providers) and the user requirements around them
> (particularly restoration use cases) as a separate thread of work.
>
>
>
>
>
> [image: ShapeBlue] <http://www.shapeblue.com>
> Paul Angus
> VP Technology ,  ShapeBlue
> d:  *+44 203 617 0528 | s: +44 203 603 0540*
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> *+44 7711 418784* <+44%207711%20418784>
> e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> <pa...@cloudyangus>  |  w:
> *www.shapeblue.com* <http://www.shapeblue.com>
> a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> Services India LLP is a company incorporated in India and is operated under
> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> company incorporated in Brasil and is operated under license from Shape
> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
> a registered trademark.
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error.
>
>
> -----Original Message-----
> From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
> Sent: 05 February 2016 15:31
> To: dev@cloudstack.apache.org
> Subject: Re: [Propose][New Feature] Storage Snapshots
>
> I think the terminology confusion comes from AWS where they do EBS
> snapshots backed up to S3 and CloudStack sort of followed that. And as an
> end user who is oblivious to the internals of my provider, my expectation
> would be something similar to what AWS as that is my biggest reference
> point.
>
> To your point Mike, I agree that a Primary Storage failure on SolidFire is
> unlikely, there are other motivations for us to push data to secondary
> storage. Primary storage (atleast for us) costs around 3 times as much as
> secondary storage and snapshots on primary storage are rarely used
> (especially for some of our customers who do daily backups).
>
>
> On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
> > Some of the weirdness is around terminology.
> >
> > For most systems I've worked on, a snapshot and a backup are two
> > completely different things (but CloudStack has traditionally used the
> > term "volume snapshot" to mean backup).
> >
> > I will put in a SolidFire "plug" here and say, though, that if your
> > primary storage is running on SolidFire that it is unlikely you'll
> > encounter an issue where your primary storage goes offline (and you'll
> > even maintain your performance guarantees during failure scenarios and
> > upgrades, as well). That being the case, it is less useful to require
> > a backup to Swift (but it's perfectly OK if that's what we want to do
> here).
> >
> > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq <sy...@gmail.com>
> > wrote:
> >
> > > Paul,
> > >
> > > I believe with the current implementation of Snapshots on managed
> > > storage
> > > (SolidFire) the snapshots are never exported to the secondary storage.
> > > While this solves the problem of having snapshots taking forever to
> > > get to sec storage, this leaves us with a
> > huge
> > > liability if our primary storage goes down. We see snapshots as our
> > > recovery path because we store them in Swift which is reliable and
> > > resilient to failures.
> > >
> > > With Storage snpashots our goal is to have Volume snapshots always
> > > backed up to secondary storage and Storage Snapshots stay on the
> > > primary
> > storage.
> > > A provider could potentially mix both these and solve the problem
> > > that you mentioned where you want to meet user's expectation of a
> > > snapshot (ie backup to sec storage) while having an ability to
> > > utilize faster sanpshots (i.e. on the device)
> > >
> > > Hope this clarifies things.
> > >
> > > Thanks,
> > > -Syed
> > >
> > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > > <pa...@shapeblue.com>
> > > wrote:
> > >
> > > > HI guys,
> > > >
> > > > Could someone point me to the Jira bug of FS for the SAN-snapshot
> > feature
> > > > in 4.6 which is mentioned.
> > > >
> > > > From my discussions with users and operators around snapshots I'd
> > > > make
> > > the
> > > > following observations:
> > > > a. 'users' use snapshots as backups (both long-term and short
> > > > term)
> > with
> > > > the expectation that they can use them for recovery if required.
> > > > b. operators fall back to snapshots if something has gone wrong
> > > > with primary storage.
> > > > c. users sometimes want to be able to export snapshots as well as
> > create
> > > > new VMs from their snapshots
> > > > d. snapshots are a currently a massive pain for operators, I know
> > > > at
> > > least
> > > > one public cloud who have snapshots which take 2 days to complete.
> > > > e. snapshots (as they are) can't be used for multiple LVM disks.
> > > >
> > > > I think the process Mike has used in the SolidFire plugin (only
> > > > moving
> > > the
> > > > disk image to secondary storage when you absolutely have to) is a
> > > > very
> > > good
> > > > and pragmatic solution. I wonder what problems an operator might
> > > experience
> > > > if they have an issue with a given primary storage pool in a cluster.
> > (I
> > > > know that that is REALLY unlikely in the SolidFire case Mike :) )
> > > > And
> > if
> > > > the transfer from primary to secondary is slow, the time to being
> > > > able
> > to
> > > > create a template or export the volume will be slow.
> > > >
> > > > So for me the issue is around making sure that the end users
> > expectations
> > > > are met (while improving the speed/efficiency of the back end)
> > > >
> > > >
> > > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > > Technology , ShapeBlue
> > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > > *+44 7711 418784* <+44%207711%20418784>
> > > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > > <pa...@cloudyangus> | w:
> > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue
> > > > Ltd is a company incorporated in England & Wales. ShapeBlue
> > > > Services India LLP is a company incorporated in India and is
> > > > operated
> > > under
> > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
> > > > a company incorporated in Brasil and is operated under license
> > > > from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> > > > by The Republic
> > of
> > > > South Africa and is traded under license from Shape Blue Ltd.
> > > > ShapeBlue
> > > is
> > > > a registered trademark.
> > > > This email and any attachments to it may be confidential and are
> > intended
> > > > solely for the use of the individual to whom it is addressed. Any
> > > > views
> > > or
> > > > opinions expressed are solely those of the author and do not
> > necessarily
> > > > represent those of Shape Blue Ltd or related companies. If you are
> > > > not
> > > the
> > > > intended recipient of this email, you must neither take any action
> > based
> > > > upon its contents, nor copy or show it to anyone. Please contact
> > > > the
> > > sender
> > > > if you believe you have received this email in error.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > > Sent: Friday, February 5, 2016 12:56 PM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > > >
> > > > Hi Mike,
> > > >
> > > > The idea of introducing a new API: StorageSnapshot for managed
> > > > storage
> > is
> > > > because the VolumeSnapshot default, or expected, behavior is to
> > > > archive snapshots into the Secondary Storage. So a StorageSnapshot
> > > > API would be
> > > for
> > > > snapshot that remain on the managed storage appliance.
> > > >
> > > > Quickly looking at the API doc and I don't see a strong
> > > > requirement for volume snapshots to be moved into secondary
> > > > storage. So, maybe StorageSnapshot API is not useful, but both use
> > > > cases are required. A snapshot that remain on the managed storage,
> > > > and another type of
> > snapshot
> > > > that end up into the secondary storage. Since you've done a lot of
> > work,
> > > > might easier to just add a parameter to the current snapshot API
> > > > that
> > > would
> > > > trigger an extraction of the storage snapshot into the secondary
> > storage?
> > > >
> > > > PL
> > > >
> > > >
> > > >
> > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > > mike.tutkowski@solidfire.com
> > > > > wrote:
> > > >
> > > > > I think that all sounds reasonable then - thanks!
> > > > >
> > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> > syed1.mushtaq@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> You are correct Mike in terms of the requirements. One of our
> > earlier
> > > > >> iterations on this was to have an argument to the create
> > > > >> snapshot
> > API
> > > > >> which decides whether to backup the volume to sec storage but
> > > > >> we realized it would make management of snapshots quite messy
> > > > >> so we proposed a new api instead.
> > > > >>
> > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > > >> <mi...@solidfire.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> Just to make sure I understand all the requirements here:
> > > > >>>
> > > > >>> 1) This relates only to managed storage (1:1 mapping between a
> > > > >>> virtual disk and a backend SAN volume).
> > > > >>>
> > > > >>> 2) We want to take the current (introduced in 4.6)
> > > > >>> functionality, which creates a snapshot on the SAN, and extend
> > > > >>> it via a config option (or
> > > > >>> something) to not only take the SAN snapshot, but to copy the
> > > > >>> underlying VHD (XenServer only) to NFS.
> > > > >>>
> > > > >>> 3) The SAN snapshot is always taken. It's the backup to NFS
> > > > >>> that is optional.
> > > > >>>
> > > > >>> 4) Templates can be created from the snapshot that's on the
> > > > >>> SAN (already works).
> > > > >>>
> > > > >>> 5) CloudStack volumes can be created from the snapshot that's
> > > > >>> on
> > the
> > > > >>> SAN (already works as long as the new CloudStack volume ends
> > > > >>> up on the same primary storage).
> > > > >>>
> > > > >>> Would we have a need for a storage snapshot API then or would
> > > > >>> that just be the standard volume snapshot without the backup to
> NFS?
> > > > >>>
> > > > >>> Thanks!
> > > > >>> Mike
> > > > >>>
> > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > > >>> <sy...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Is it possible to have both functionalities (snapshot on SAN
> > > > >>>> & Sec
> > > > >>>> Storage) coexist? Because Ideally, we would like to have both.
> > > > >>>> For example, some of our customers want to implement their
> > > > >>>> own backup strategies and do encryption to their backups
> > > > >>>> which is a perfect use case for Storage Snapshot while our
> > > > >>>> other customers will still keep using the standard volume
> snapshot.
> > > > >>>>
> > > > >>>> To keep things backward compatible, we can add a setting
> > > > >>>> which
> > says
> > > > >>>> to not upload on secondary storage, because, after all, you
> > > > >>>> would take a SAN snapshot first when doing a Volume Snapshot.
> > > > >>>> You could stop the process there and not do the upload.
> > > > >>>>
> > > > >>>> What do you think about this approach?
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> -Syed
> > > > >>>>
> > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > > >>>> mike.tutkowski@solidfire.com> wrote:
> > > > >>>>
> > > > >>>>> So, this is just me thinking out load here, but if a given
> > > > >>>>> CloudStack cloud doesn't actually need to provide both the
> > ability
> > > > >>>>> to take a SAN snapshot and export it to NFS (if just taking
> > > > >>>>> a SAN snapshot is OK), then we might be able to get away
> > > > >>>>> with no new
> > API
> > > > >>>>> calls and simply implement a new custom snapshot strategy
> > > > >>>>> and
> > data
> > > > >>>>> motion strategy to handle the case where the CloudStack
> > > > >>>>> cloud
> > does
> > > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > > >>>>>
> > > > >>>>> In other words, the "default" behavior would be to use the
> > > > >>>>> snapshot strategy and data motion strategy that we already
> > > > >>>>> have (the one that only takes a SAN snapshot).
> > > > >>>>>
> > > > >>>>> If your CloudStack cloud, however, wants to take a SAN
> > > > >>>>> snapshot and have the data exported to NFS, then we could
> > > > >>>>> have you manipulate a Swing config file to make use of a new
> > > > >>>>> snapshot strategy and data motion strategy that performs
> > > > >>>>> both of these
> > > > activities.
> > > > >>>>>
> > > > >>>>> This way, the old behavior is still the default for users,
> > > > >>>>> but CloudStack admins can change this behavior via
> configuration.
> > > > >>>>>
> > > > >>>>> Thoughts?
> > > > >>>>>
> > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > > >>>>> mike.tutkowski@solidfire.com> wrote:
> > > > >>>>>
> > > > >>>>>> Right...I think we will need to come up with a viable
> > > > >>>>>> upgrade path or some reasonable way for them to move from
> > > > >>>>>> the old way to the new way (and some obvious way that they
> > > > >>>>>> will know they need
> > to
> > > > do this).
> > > > >>>>>>
> > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > >>>>>>
> > > > >>>>>>> I'm not really sure about the upgrade path however,
> > > > >>>>>>> customers who are using 4.6 and are on a managed storage
> > > > >>>>>>> would no longer have the same functionality with Volume
> Snapshots.
> > > > >>>>>>>
> > > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> So if I understand correctly, currently taking a Volume
> > > > >>>>>>>> Snapshots of a volume on a managed storage keeps it on
> > > > >>>>>>>> the storage array. As a part of this feature, we can make
> > > > >>>>>>>> sure
> > that
> > > > >>>>>>>> Volume Snapshots on managed storage are uploaded to the
> > > > >>>>>>>> secondary storage. This would make the Volume Snapshot
> > > > >>>>>>>> feature behave the same regardless of the storage
> > > > >>>>>>>> (managed or
> > > > >>>>>>>> non-managed) And, for utilizing the efficient backend
> > > > >>>>>>>> storage
> > > > capabilities, we can use the new storage snapshots API.
> > > > >>>>>>>>
> > > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi everyone,
> > > > >>>>>>>>>
> > > > >>>>>>>>> Whatever we do here, we need to have a plan to deal with
> > > > >>>>>>>>> the fact that we already have a feature (in 4.6 and
> > > > >>>>>>>>> later) that allows you to use the existing
> > > > >>>>>>>>> volume-snapshot APIs to create a volume snapshot (for
> > > > >>>>>>>>> managed
> > > > >>>>>>>>> storage) that resides on a backend SAN (using a custom
> > > > >>>>>>>>> snapshot strategy and a custom data motion strategy).
> > > > >>>>>>>>>
> > > > >>>>>>>>> If these new APIs go in, then how should the original
> > > > >>>>>>>>> implementation (present in 4.6 and later) be changed? If
> > > > >>>>>>>>> it
> > is
> > > > >>>>>>>>> changed, how do we support customers who are already
> > > > >>>>>>>>> using
> > the
> > > > >>>>>>>>> original volume-snapshot API to take snapshots on a
> > > > >>>>>>>>> backend
> > > SAN?
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks,
> > > > >>>>>>>>> Mike
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > > >>>>>>>>> wstevens@cloudops.com> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Will you be able to create a Template from a
> > StorageSnapshot?
> > > > >>>>>>>>>> If yes, will the template be stored in the secondary
> > > > >>>>>>>>>> storage like normal templates or will that be handled
> > > > >>>>>>>>>> somehow on the
> > > > vendor side?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> *Will STEVENS*
> > > > >>>>>>>>>> Lead Developer
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Thanks Will!!!
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira
> > > > >>>>>>>>>>>> ticket because it was not clear in the 'mention'
> > > > >>>>>>>>>>>> section because it shows as a page 'you do not have
> permission to'.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> *Will STEVENS*
> > > > >>>>>>>>>>>> Lead Developer
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > > >>>>>>>>>>>> > wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Design Spec:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > > >>>>>>>>>>>>> rageSnapshot++API
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Jira Ticket
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-927
> > > > >>>>>>>>>>>>> 8
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Hi All,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots
> > > > >>>>>>>>>>>>> on managed storage backends like SolidFire.
> > > > >>>>>>>>>>>>> Snapshots on current managed storage stay on the
> > > > >>>>>>>>>>>>> device which is contrary to what CloudStack calls
> snpshots.
> > > > >>>>>>>>>>>>> But taking snapshots on storage and keeping it there
> > > > >>>>>>>>>>>>> has its own advantages
> > and
> > > > >>>>>>>>>>>>> we would ideally like to have both ways of doing
> > > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
> > > > >>>>>>>>>>>>> snapshots on backend storage.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> What do you guys think of this feature? I would love
> > > > >>>>>>>>>>>>> to have some feedback. I am working on making the
> > > > >>>>>>>>>>>>> design
> > spec
> > > > >>>>>>>>>>>>> more concrete but wanted to have a high level
> > > > >>>>>>>>>>>>> feedback first before starting to work on it.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>> -Syed
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> --
> > > > >>>>>>>>> *Mike Tutkowski*
> > > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > > >>>>>>>>> o: 303.746.7302
> > > > >>>>>>>>> Advancing the way the world uses the cloud
> > > > >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>> *Mike Tutkowski*
> > > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > >>>>>> e: mike.tutkowski@solidfire.com
> > > > >>>>>> o: 303.746.7302
> > > > >>>>>> Advancing the way the world uses the cloud
> > > > >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> *Mike Tutkowski*
> > > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > >>>>> e: mike.tutkowski@solidfire.com
> > > > >>>>> o: 303.746.7302
> > > > >>>>> Advancing the way the world uses the cloud
> > > > >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> *Mike Tutkowski*
> > > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > > >>> e: mike.tutkowski@solidfire.com
> > > > >>> o: 303.746.7302
> > > > >>> Advancing the way the world uses the cloud
> > > > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > *Mike Tutkowski*
> > > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > > e: mike.tutkowski@solidfire.com
> > > > > o: 303.746.7302
> > > > > Advancing the way the world uses the cloud
> > > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > > >
> > > > Find out more about ShapeBlue and our range of CloudStack related
> > > services:
> > > > IaaS Cloud Design & Build
> > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > > > rapid IaaS deployment framework <http://shapeblue.com/csforge/>
> > > > CloudStack Consulting
> > > > <http://shapeblue.com/cloudstack-consultancy/> |
> > > CloudStack
> > > > Software Engineering
> > > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > > CloudStack Infrastructure Support
> > > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > > CloudStack Bootcamp Training Courses
> > > > <http://shapeblue.com/cloudstack-training/>
> > > >
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack
> Software Engineering
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support
> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
>

RE: [Propose][New Feature] Storage Snapshots

[Paul Angus <pa...@shapeblue.com>]

Hi Syed,

As I understand it, the SolidFire plugin will export the snapshot to secondary storage if the user requests a template from the snapshot or wants to download the snapshot from the cloud. This is a good, pragmatic approach and yes Mike the SolidFire storage is super reliable and snapshots on SolidFire arrays take up next to no space. BUT I think that we are talking about a more general purpose API, and other storage systems may not be as awesome as Mike's. That's my concern. Also, the time to transfer for say 1TB to move from primary to sec storage and then create a VM template out of it may be too long for users.

@Mike I don’t think 'we' use the term volume snapshot for backup, it's just that users want to do backups and a volume snapshot is the only type of snapshot that copies the disk elsewhere and can be scheduled.

I'm 'pondering' the implications of enabling actual backups (through recognised backup providers) and the user requirements around them (particularly restoration use cases) as a separate thread of work.





[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology   ,       ShapeBlue


d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>     |      m:      +44 7711 418784<tel:+44%207711%20418784>

e:      paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus>      |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:imaged6b85d.png@10e7fa55.46a9676b]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




-----Original Message-----
From: Syed Mushtaq [mailto:syed1.mushtaq@gmail.com]
Sent: 05 February 2016 15:31
To: dev@cloudstack.apache.org
Subject: Re: [Propose][New Feature] Storage Snapshots

I think the terminology confusion comes from AWS where they do EBS snapshots backed up to S3 and CloudStack sort of followed that. And as an end user who is oblivious to the internals of my provider, my expectation would be something similar to what AWS as that is my biggest reference point.

To your point Mike, I agree that a Primary Storage failure on SolidFire is unlikely, there are other motivations for us to push data to secondary storage. Primary storage (atleast for us) costs around 3 times as much as secondary storage and snapshots on primary storage are rarely used (especially for some of our customers who do daily backups).


On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski < mike.tutkowski@solidfire.com> wrote:

> Some of the weirdness is around terminology.
>
> For most systems I've worked on, a snapshot and a backup are two
> completely different things (but CloudStack has traditionally used the
> term "volume snapshot" to mean backup).
>
> I will put in a SolidFire "plug" here and say, though, that if your
> primary storage is running on SolidFire that it is unlikely you'll
> encounter an issue where your primary storage goes offline (and you'll
> even maintain your performance guarantees during failure scenarios and
> upgrades, as well). That being the case, it is less useful to require
> a backup to Swift (but it's perfectly OK if that's what we want to do here).
>
> On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
> > Paul,
> >
> > I believe with the current implementation of Snapshots on managed
> > storage
> > (SolidFire) the snapshots are never exported to the secondary storage.
> > While this solves the problem of having snapshots taking forever to
> > get to sec storage, this leaves us with a
> huge
> > liability if our primary storage goes down. We see snapshots as our
> > recovery path because we store them in Swift which is reliable and
> > resilient to failures.
> >
> > With Storage snpashots our goal is to have Volume snapshots always
> > backed up to secondary storage and Storage Snapshots stay on the
> > primary
> storage.
> > A provider could potentially mix both these and solve the problem
> > that you mentioned where you want to meet user's expectation of a
> > snapshot (ie backup to sec storage) while having an ability to
> > utilize faster sanpshots (i.e. on the device)
> >
> > Hope this clarifies things.
> >
> > Thanks,
> > -Syed
> >
> > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > <pa...@shapeblue.com>
> > wrote:
> >
> > > HI guys,
> > >
> > > Could someone point me to the Jira bug of FS for the SAN-snapshot
> feature
> > > in 4.6 which is mentioned.
> > >
> > > From my discussions with users and operators around snapshots I'd
> > > make
> > the
> > > following observations:
> > > a. 'users' use snapshots as backups (both long-term and short
> > > term)
> with
> > > the expectation that they can use them for recovery if required.
> > > b. operators fall back to snapshots if something has gone wrong
> > > with primary storage.
> > > c. users sometimes want to be able to export snapshots as well as
> create
> > > new VMs from their snapshots
> > > d. snapshots are a currently a massive pain for operators, I know
> > > at
> > least
> > > one public cloud who have snapshots which take 2 days to complete.
> > > e. snapshots (as they are) can't be used for multiple LVM disks.
> > >
> > > I think the process Mike has used in the SolidFire plugin (only
> > > moving
> > the
> > > disk image to secondary storage when you absolutely have to) is a
> > > very
> > good
> > > and pragmatic solution. I wonder what problems an operator might
> > experience
> > > if they have an issue with a given primary storage pool in a cluster.
> (I
> > > know that that is REALLY unlikely in the SolidFire case Mike :) )
> > > And
> if
> > > the transfer from primary to secondary is slow, the time to being
> > > able
> to
> > > create a template or export the volume will be slow.
> > >
> > > So for me the issue is around making sure that the end users
> expectations
> > > are met (while improving the speed/efficiency of the back end)
> > >
> > >
> > > [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP
> > > Technology , ShapeBlue
> > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> > > *+44 7711 418784* <+44%207711%20418784>
> > > e: *paul.angus@shapeblue.com | t: @cloudyangus*
> > > <pa...@cloudyangus> | w:
> > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue
> > > Ltd is a company incorporated in England & Wales. ShapeBlue
> > > Services India LLP is a company incorporated in India and is
> > > operated
> > under
> > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is
> > > a company incorporated in Brasil and is operated under license
> > > from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
> > > by The Republic
> of
> > > South Africa and is traded under license from Shape Blue Ltd.
> > > ShapeBlue
> > is
> > > a registered trademark.
> > > This email and any attachments to it may be confidential and are
> intended
> > > solely for the use of the individual to whom it is addressed. Any
> > > views
> > or
> > > opinions expressed are solely those of the author and do not
> necessarily
> > > represent those of Shape Blue Ltd or related companies. If you are
> > > not
> > the
> > > intended recipient of this email, you must neither take any action
> based
> > > upon its contents, nor copy or show it to anyone. Please contact
> > > the
> > sender
> > > if you believe you have received this email in error.
> > >
> > >
> > > -----Original Message-----
> > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > Sent: Friday, February 5, 2016 12:56 PM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > >
> > > Hi Mike,
> > >
> > > The idea of introducing a new API: StorageSnapshot for managed
> > > storage
> is
> > > because the VolumeSnapshot default, or expected, behavior is to
> > > archive snapshots into the Secondary Storage. So a StorageSnapshot
> > > API would be
> > for
> > > snapshot that remain on the managed storage appliance.
> > >
> > > Quickly looking at the API doc and I don't see a strong
> > > requirement for volume snapshots to be moved into secondary
> > > storage. So, maybe StorageSnapshot API is not useful, but both use
> > > cases are required. A snapshot that remain on the managed storage,
> > > and another type of
> snapshot
> > > that end up into the secondary storage. Since you've done a lot of
> work,
> > > might easier to just add a parameter to the current snapshot API
> > > that
> > would
> > > trigger an extraction of the storage snapshot into the secondary
> storage?
> > >
> > > PL
> > >
> > >
> > >
> > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com
> > > > wrote:
> > >
> > > > I think that all sounds reasonable then - thanks!
> > > >
> > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> syed1.mushtaq@gmail.com>
> > > > wrote:
> > > >
> > > >> You are correct Mike in terms of the requirements. One of our
> earlier
> > > >> iterations on this was to have an argument to the create
> > > >> snapshot
> API
> > > >> which decides whether to backup the volume to sec storage but
> > > >> we realized it would make management of snapshots quite messy
> > > >> so we proposed a new api instead.
> > > >>
> > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > >> <mi...@solidfire.com>
> > > >> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> Just to make sure I understand all the requirements here:
> > > >>>
> > > >>> 1) This relates only to managed storage (1:1 mapping between a
> > > >>> virtual disk and a backend SAN volume).
> > > >>>
> > > >>> 2) We want to take the current (introduced in 4.6)
> > > >>> functionality, which creates a snapshot on the SAN, and extend
> > > >>> it via a config option (or
> > > >>> something) to not only take the SAN snapshot, but to copy the
> > > >>> underlying VHD (XenServer only) to NFS.
> > > >>>
> > > >>> 3) The SAN snapshot is always taken. It's the backup to NFS
> > > >>> that is optional.
> > > >>>
> > > >>> 4) Templates can be created from the snapshot that's on the
> > > >>> SAN (already works).
> > > >>>
> > > >>> 5) CloudStack volumes can be created from the snapshot that's
> > > >>> on
> the
> > > >>> SAN (already works as long as the new CloudStack volume ends
> > > >>> up on the same primary storage).
> > > >>>
> > > >>> Would we have a need for a storage snapshot API then or would
> > > >>> that just be the standard volume snapshot without the backup to NFS?
> > > >>>
> > > >>> Thanks!
> > > >>> Mike
> > > >>>
> > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > >>> <sy...@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Is it possible to have both functionalities (snapshot on SAN
> > > >>>> & Sec
> > > >>>> Storage) coexist? Because Ideally, we would like to have both.
> > > >>>> For example, some of our customers want to implement their
> > > >>>> own backup strategies and do encryption to their backups
> > > >>>> which is a perfect use case for Storage Snapshot while our
> > > >>>> other customers will still keep using the standard volume snapshot.
> > > >>>>
> > > >>>> To keep things backward compatible, we can add a setting
> > > >>>> which
> says
> > > >>>> to not upload on secondary storage, because, after all, you
> > > >>>> would take a SAN snapshot first when doing a Volume Snapshot.
> > > >>>> You could stop the process there and not do the upload.
> > > >>>>
> > > >>>> What do you think about this approach?
> > > >>>>
> > > >>>> Thanks,
> > > >>>> -Syed
> > > >>>>
> > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > >>>> mike.tutkowski@solidfire.com> wrote:
> > > >>>>
> > > >>>>> So, this is just me thinking out load here, but if a given
> > > >>>>> CloudStack cloud doesn't actually need to provide both the
> ability
> > > >>>>> to take a SAN snapshot and export it to NFS (if just taking
> > > >>>>> a SAN snapshot is OK), then we might be able to get away
> > > >>>>> with no new
> API
> > > >>>>> calls and simply implement a new custom snapshot strategy
> > > >>>>> and
> data
> > > >>>>> motion strategy to handle the case where the CloudStack
> > > >>>>> cloud
> does
> > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > >>>>>
> > > >>>>> In other words, the "default" behavior would be to use the
> > > >>>>> snapshot strategy and data motion strategy that we already
> > > >>>>> have (the one that only takes a SAN snapshot).
> > > >>>>>
> > > >>>>> If your CloudStack cloud, however, wants to take a SAN
> > > >>>>> snapshot and have the data exported to NFS, then we could
> > > >>>>> have you manipulate a Swing config file to make use of a new
> > > >>>>> snapshot strategy and data motion strategy that performs
> > > >>>>> both of these
> > > activities.
> > > >>>>>
> > > >>>>> This way, the old behavior is still the default for users,
> > > >>>>> but CloudStack admins can change this behavior via configuration.
> > > >>>>>
> > > >>>>> Thoughts?
> > > >>>>>
> > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > >>>>> mike.tutkowski@solidfire.com> wrote:
> > > >>>>>
> > > >>>>>> Right...I think we will need to come up with a viable
> > > >>>>>> upgrade path or some reasonable way for them to move from
> > > >>>>>> the old way to the new way (and some obvious way that they
> > > >>>>>> will know they need
> to
> > > do this).
> > > >>>>>>
> > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>>> I'm not really sure about the upgrade path however,
> > > >>>>>>> customers who are using 4.6 and are on a managed storage
> > > >>>>>>> would no longer have the same functionality with Volume Snapshots.
> > > >>>>>>>
> > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > >>>>>>>
> > > >>>>>>>> So if I understand correctly, currently taking a Volume
> > > >>>>>>>> Snapshots of a volume on a managed storage keeps it on
> > > >>>>>>>> the storage array. As a part of this feature, we can make
> > > >>>>>>>> sure
> that
> > > >>>>>>>> Volume Snapshots on managed storage are uploaded to the
> > > >>>>>>>> secondary storage. This would make the Volume Snapshot
> > > >>>>>>>> feature behave the same regardless of the storage
> > > >>>>>>>> (managed or
> > > >>>>>>>> non-managed) And, for utilizing the efficient backend
> > > >>>>>>>> storage
> > > capabilities, we can use the new storage snapshots API.
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi everyone,
> > > >>>>>>>>>
> > > >>>>>>>>> Whatever we do here, we need to have a plan to deal with
> > > >>>>>>>>> the fact that we already have a feature (in 4.6 and
> > > >>>>>>>>> later) that allows you to use the existing
> > > >>>>>>>>> volume-snapshot APIs to create a volume snapshot (for
> > > >>>>>>>>> managed
> > > >>>>>>>>> storage) that resides on a backend SAN (using a custom
> > > >>>>>>>>> snapshot strategy and a custom data motion strategy).
> > > >>>>>>>>>
> > > >>>>>>>>> If these new APIs go in, then how should the original
> > > >>>>>>>>> implementation (present in 4.6 and later) be changed? If
> > > >>>>>>>>> it
> is
> > > >>>>>>>>> changed, how do we support customers who are already
> > > >>>>>>>>> using
> the
> > > >>>>>>>>> original volume-snapshot API to take snapshots on a
> > > >>>>>>>>> backend
> > SAN?
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> Mike
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > >>>>>>>>> wstevens@cloudops.com> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Will you be able to create a Template from a
> StorageSnapshot?
> > > >>>>>>>>>> If yes, will the template be stored in the secondary
> > > >>>>>>>>>> storage like normal templates or will that be handled
> > > >>>>>>>>>> somehow on the
> > > vendor side?
> > > >>>>>>>>>>
> > > >>>>>>>>>> *Will STEVENS*
> > > >>>>>>>>>> Lead Developer
> > > >>>>>>>>>>
> > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Thanks Will!!!
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira
> > > >>>>>>>>>>>> ticket because it was not clear in the 'mention'
> > > >>>>>>>>>>>> section because it shows as a page 'you do not have permission to'.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> *Will STEVENS*
> > > >>>>>>>>>>>> Lead Developer
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > >>>>>>>>>>>> > wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Design Spec:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > >>>>>>>>>>>>> rageSnapshot++API
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jira Ticket
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-927
> > > >>>>>>>>>>>>> 8
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi All,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots
> > > >>>>>>>>>>>>> on managed storage backends like SolidFire.
> > > >>>>>>>>>>>>> Snapshots on current managed storage stay on the
> > > >>>>>>>>>>>>> device which is contrary to what CloudStack calls snpshots.
> > > >>>>>>>>>>>>> But taking snapshots on storage and keeping it there
> > > >>>>>>>>>>>>> has its own advantages
> and
> > > >>>>>>>>>>>>> we would ideally like to have both ways of doing
> > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
> > > >>>>>>>>>>>>> snapshots on backend storage.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> What do you guys think of this feature? I would love
> > > >>>>>>>>>>>>> to have some feedback. I am working on making the
> > > >>>>>>>>>>>>> design
> spec
> > > >>>>>>>>>>>>> more concrete but wanted to have a high level
> > > >>>>>>>>>>>>> feedback first before starting to work on it.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>> -Syed
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> --
> > > >>>>>>>>> *Mike Tutkowski*
> > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > >>>>>>>>> o: 303.746.7302
> > > >>>>>>>>> Advancing the way the world uses the cloud
> > > >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> *Mike Tutkowski*
> > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > >>>>>> e: mike.tutkowski@solidfire.com
> > > >>>>>> o: 303.746.7302
> > > >>>>>> Advancing the way the world uses the cloud
> > > >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>> *Mike Tutkowski*
> > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > >>>>> e: mike.tutkowski@solidfire.com
> > > >>>>> o: 303.746.7302
> > > >>>>> Advancing the way the world uses the cloud
> > > >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> *Mike Tutkowski*
> > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > >>> e: mike.tutkowski@solidfire.com
> > > >>> o: 303.746.7302
> > > >>> Advancing the way the world uses the cloud
> > > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > >>>
> > > >>
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > > Find out more about ShapeBlue and our range of CloudStack related
> > services:
> > > IaaS Cloud Design & Build
> > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> > > rapid IaaS deployment framework <http://shapeblue.com/csforge/>
> > > CloudStack Consulting
> > > <http://shapeblue.com/cloudstack-consultancy/> |
> > CloudStack
> > > Software Engineering
> > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > CloudStack Infrastructure Support
> > > <http://shapeblue.com/cloudstack-infrastructure-support/> |
> > > CloudStack Bootcamp Training Courses
> > > <http://shapeblue.com/cloudstack-training/>
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

 I think the terminology confusion comes from AWS where they do EBS
snapshots backed up to S3 and CloudStack sort of followed that. And as an
end user who is oblivious to the internals of my provider, my expectation
would be something similar to what AWS as that is my biggest reference
point.

To your point Mike, I agree that a Primary Storage failure on SolidFire is
unlikely, there are other motivations for us to push data to secondary
storage. Primary storage (atleast for us) costs around 3 times as much as
secondary storage and snapshots on primary storage are rarely used
(especially for some of our customers who do daily backups).


On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
mike.tutkowski@solidfire.com> wrote:

> Some of the weirdness is around terminology.
>
> For most systems I've worked on, a snapshot and a backup are two completely
> different things (but CloudStack has traditionally used the term "volume
> snapshot" to mean backup).
>
> I will put in a SolidFire "plug" here and say, though, that if your primary
> storage is running on SolidFire that it is unlikely you'll encounter an
> issue where your primary storage goes offline (and you'll even maintain
> your performance guarantees during failure scenarios and upgrades, as
> well). That being the case, it is less useful to require a backup to Swift
> (but it's perfectly OK if that's what we want to do here).
>
> On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
> > Paul,
> >
> > I believe with the current implementation of Snapshots on managed storage
> > (SolidFire) the snapshots are never exported to the secondary storage.
> > While this solves the problem of having
> > snapshots taking forever to get to sec storage, this leaves us with a
> huge
> > liability if our primary storage goes down. We see snapshots as our
> > recovery path because we store them in Swift which is reliable and
> > resilient to failures.
> >
> > With Storage snpashots our goal is to have Volume snapshots always backed
> > up to secondary storage and Storage Snapshots stay on the primary
> storage.
> > A provider could potentially mix both
> > these and solve the problem that you mentioned where you want to meet
> > user's  expectation of a snapshot (ie backup to sec storage) while having
> > an ability  to utilize faster sanpshots (i.e. on the device)
> >
> > Hope this clarifies things.
> >
> > Thanks,
> > -Syed
> >
> > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus <pa...@shapeblue.com>
> > wrote:
> >
> > > HI guys,
> > >
> > > Could someone point me to the Jira bug of FS for the SAN-snapshot
> feature
> > > in 4.6 which is mentioned.
> > >
> > > From my discussions with users and operators around snapshots I'd make
> > the
> > > following observations:
> > > a. 'users' use snapshots as backups (both long-term and short term)
> with
> > > the expectation that they can use them for recovery if required.
> > > b. operators fall back to snapshots if something has gone wrong with
> > > primary storage.
> > > c. users sometimes want to be able to export snapshots as well as
> create
> > > new VMs from their snapshots
> > > d. snapshots are a currently a massive pain for operators, I know at
> > least
> > > one public cloud who have snapshots which take 2 days to complete.
> > > e. snapshots (as they are) can't be used for multiple LVM disks.
> > >
> > > I think the process Mike has used in the SolidFire plugin (only moving
> > the
> > > disk image to secondary storage when you absolutely have to) is a very
> > good
> > > and pragmatic solution. I wonder what problems an operator might
> > experience
> > > if they have an issue with a given primary storage pool in a cluster.
> (I
> > > know that that is REALLY unlikely in the SolidFire case Mike :) ) And
> if
> > > the transfer from primary to secondary is slow, the time to being able
> to
> > > create a template or export the volume will be slow.
> > >
> > > So for me the issue is around making sure that the end users
> expectations
> > > are met (while improving the speed/efficiency of the back end)
> > >
> > >
> > > [image: ShapeBlue] <http://www.shapeblue.com>
> > > Paul Angus
> > > VP Technology ,  ShapeBlue
> > > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > > *+44 7711 418784* <+44%207711%20418784>
> > > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > > <pa...@cloudyangus>  |  w:
> > > *www.shapeblue.com* <http://www.shapeblue.com>
> > > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> > > Services India LLP is a company incorporated in India and is operated
> > under
> > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > > company incorporated in Brasil and is operated under license from Shape
> > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic
> of
> > > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
> > is
> > > a registered trademark.
> > > This email and any attachments to it may be confidential and are
> intended
> > > solely for the use of the individual to whom it is addressed. Any views
> > or
> > > opinions expressed are solely those of the author and do not
> necessarily
> > > represent those of Shape Blue Ltd or related companies. If you are not
> > the
> > > intended recipient of this email, you must neither take any action
> based
> > > upon its contents, nor copy or show it to anyone. Please contact the
> > sender
> > > if you believe you have received this email in error.
> > >
> > >
> > > -----Original Message-----
> > > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > > Sent: Friday, February 5, 2016 12:56 PM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [Propose][New Feature] Storage Snapshots
> > >
> > > Hi Mike,
> > >
> > > The idea of introducing a new API: StorageSnapshot for managed storage
> is
> > > because the VolumeSnapshot default, or expected, behavior is to archive
> > > snapshots into the Secondary Storage. So a StorageSnapshot API would be
> > for
> > > snapshot that remain on the managed storage appliance.
> > >
> > > Quickly looking at the API doc and I don't see a strong requirement for
> > > volume snapshots to be moved into secondary storage. So, maybe
> > > StorageSnapshot API is not useful, but both use cases are required. A
> > > snapshot that remain on the managed storage, and another type of
> snapshot
> > > that end up into the secondary storage. Since you've done a lot of
> work,
> > > might easier to just add a parameter to the current snapshot API that
> > would
> > > trigger an extraction of the storage snapshot into the secondary
> storage?
> > >
> > > PL
> > >
> > >
> > >
> > > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > > mike.tutkowski@solidfire.com
> > > > wrote:
> > >
> > > > I think that all sounds reasonable then - thanks!
> > > >
> > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <
> syed1.mushtaq@gmail.com>
> > > > wrote:
> > > >
> > > >> You are correct Mike in terms of the requirements. One of our
> earlier
> > > >> iterations on this was to have an argument to the create snapshot
> API
> > > >> which decides whether to backup the volume to sec storage but we
> > > >> realized it would make management of snapshots quite messy so we
> > > >> proposed a new api instead.
> > > >>
> > > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > > >> <mi...@solidfire.com>
> > > >> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> Just to make sure I understand all the requirements here:
> > > >>>
> > > >>> 1) This relates only to managed storage (1:1 mapping between a
> > > >>> virtual disk and a backend SAN volume).
> > > >>>
> > > >>> 2) We want to take the current (introduced in 4.6) functionality,
> > > >>> which creates a snapshot on the SAN, and extend it via a config
> > > >>> option (or
> > > >>> something) to not only take the SAN snapshot, but to copy the
> > > >>> underlying VHD (XenServer only) to NFS.
> > > >>>
> > > >>> 3) The SAN snapshot is always taken. It's the backup to NFS that is
> > > >>> optional.
> > > >>>
> > > >>> 4) Templates can be created from the snapshot that's on the SAN
> > > >>> (already works).
> > > >>>
> > > >>> 5) CloudStack volumes can be created from the snapshot that's on
> the
> > > >>> SAN (already works as long as the new CloudStack volume ends up on
> > > >>> the same primary storage).
> > > >>>
> > > >>> Would we have a need for a storage snapshot API then or would that
> > > >>> just be the standard volume snapshot without the backup to NFS?
> > > >>>
> > > >>> Thanks!
> > > >>> Mike
> > > >>>
> > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > > >>> <sy...@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Is it possible to have both functionalities (snapshot on SAN & Sec
> > > >>>> Storage) coexist? Because Ideally, we would like to have both.
> > > >>>> For example, some of our customers want to implement their own
> > > >>>> backup strategies and do encryption to their backups which is a
> > > >>>> perfect use case for Storage Snapshot while our other customers
> > > >>>> will still keep using the standard volume snapshot.
> > > >>>>
> > > >>>> To keep things backward compatible, we can add a setting which
> says
> > > >>>> to not upload on secondary storage, because, after all, you would
> > > >>>> take a SAN snapshot first when doing a Volume Snapshot. You could
> > > >>>> stop the process there and not do the upload.
> > > >>>>
> > > >>>> What do you think about this approach?
> > > >>>>
> > > >>>> Thanks,
> > > >>>> -Syed
> > > >>>>
> > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > > >>>> mike.tutkowski@solidfire.com> wrote:
> > > >>>>
> > > >>>>> So, this is just me thinking out load here, but if a given
> > > >>>>> CloudStack cloud doesn't actually need to provide both the
> ability
> > > >>>>> to take a SAN snapshot and export it to NFS (if just taking a SAN
> > > >>>>> snapshot is OK), then we might be able to get away with no new
> API
> > > >>>>> calls and simply implement a new custom snapshot strategy and
> data
> > > >>>>> motion strategy to handle the case where the CloudStack cloud
> does
> > > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > > >>>>>
> > > >>>>> In other words, the "default" behavior would be to use the
> > > >>>>> snapshot strategy and data motion strategy that we already have
> > > >>>>> (the one that only takes a SAN snapshot).
> > > >>>>>
> > > >>>>> If your CloudStack cloud, however, wants to take a SAN snapshot
> > > >>>>> and have the data exported to NFS, then we could have you
> > > >>>>> manipulate a Swing config file to make use of a new snapshot
> > > >>>>> strategy and data motion strategy that performs both of these
> > > activities.
> > > >>>>>
> > > >>>>> This way, the old behavior is still the default for users, but
> > > >>>>> CloudStack admins can change this behavior via configuration.
> > > >>>>>
> > > >>>>> Thoughts?
> > > >>>>>
> > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > > >>>>> mike.tutkowski@solidfire.com> wrote:
> > > >>>>>
> > > >>>>>> Right...I think we will need to come up with a viable upgrade
> > > >>>>>> path or some reasonable way for them to move from the old way to
> > > >>>>>> the new way (and some obvious way that they will know they need
> to
> > > do this).
> > > >>>>>>
> > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>>> I'm not really sure about the upgrade path however, customers
> > > >>>>>>> who are using 4.6 and are on a managed storage would no longer
> > > >>>>>>> have the same functionality with Volume Snapshots.
> > > >>>>>>>
> > > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > >>>>>>>
> > > >>>>>>>> So if I understand correctly, currently taking a Volume
> > > >>>>>>>> Snapshots of a volume on a managed storage keeps it on the
> > > >>>>>>>> storage array. As a part of this feature, we can make sure
> that
> > > >>>>>>>> Volume Snapshots on managed storage are uploaded to the
> > > >>>>>>>> secondary storage. This would make the Volume Snapshot feature
> > > >>>>>>>> behave the same regardless of the storage (managed or
> > > >>>>>>>> non-managed) And, for utilizing the efficient backend storage
> > > capabilities, we can use the new storage snapshots API.
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi everyone,
> > > >>>>>>>>>
> > > >>>>>>>>> Whatever we do here, we need to have a plan to deal with the
> > > >>>>>>>>> fact that we already have a feature (in 4.6 and later) that
> > > >>>>>>>>> allows you to use the existing volume-snapshot APIs to create
> > > >>>>>>>>> a volume snapshot (for managed
> > > >>>>>>>>> storage) that resides on a backend SAN (using a custom
> > > >>>>>>>>> snapshot strategy and a custom data motion strategy).
> > > >>>>>>>>>
> > > >>>>>>>>> If these new APIs go in, then how should the original
> > > >>>>>>>>> implementation (present in 4.6 and later) be changed? If it
> is
> > > >>>>>>>>> changed, how do we support customers who are already using
> the
> > > >>>>>>>>> original volume-snapshot API to take snapshots on a backend
> > SAN?
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> Mike
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > > >>>>>>>>> wstevens@cloudops.com> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Will you be able to create a Template from a
> StorageSnapshot?
> > > >>>>>>>>>> If yes, will the template be stored in the secondary storage
> > > >>>>>>>>>> like normal templates or will that be handled somehow on the
> > > vendor side?
> > > >>>>>>>>>>
> > > >>>>>>>>>> *Will STEVENS*
> > > >>>>>>>>>> Lead Developer
> > > >>>>>>>>>>
> > > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Thanks Will!!!
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira ticket
> > > >>>>>>>>>>>> because it was not clear in the 'mention' section because
> > > >>>>>>>>>>>> it shows as a page 'you do not have permission to'.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> *Will STEVENS*
> > > >>>>>>>>>>>> Lead Developer
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > > >>>>>>>>>>>> <sahmed@cloudops.com
> > > >>>>>>>>>>>> > wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Design Spec:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > > >>>>>>>>>>>>> rageSnapshot++API
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jira Ticket
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi All,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots on
> > > >>>>>>>>>>>>> managed storage backends like SolidFire. Snapshots on
> > > >>>>>>>>>>>>> current managed storage stay on the device which is
> > > >>>>>>>>>>>>> contrary to what CloudStack calls snpshots.
> > > >>>>>>>>>>>>> But taking snapshots
> > > >>>>>>>>>>>>> on storage and keeping it there has its own advantages
> and
> > > >>>>>>>>>>>>> we would ideally like to have both ways of doing
> > > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
> > > >>>>>>>>>>>>> snapshots on backend storage.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> What do you guys think of this feature? I would love to
> > > >>>>>>>>>>>>> have some feedback. I am working on making the design
> spec
> > > >>>>>>>>>>>>> more concrete but wanted to have a high level feedback
> > > >>>>>>>>>>>>> first before starting to work on it.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>> -Syed
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> --
> > > >>>>>>>>> *Mike Tutkowski*
> > > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > > >>>>>>>>> o: 303.746.7302
> > > >>>>>>>>> Advancing the way the world uses the cloud
> > > >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> *Mike Tutkowski*
> > > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > >>>>>> e: mike.tutkowski@solidfire.com
> > > >>>>>> o: 303.746.7302
> > > >>>>>> Advancing the way the world uses the cloud
> > > >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>> *Mike Tutkowski*
> > > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > > >>>>> e: mike.tutkowski@solidfire.com
> > > >>>>> o: 303.746.7302
> > > >>>>> Advancing the way the world uses the cloud
> > > >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> *Mike Tutkowski*
> > > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > > >>> e: mike.tutkowski@solidfire.com
> > > >>> o: 303.746.7302
> > > >>> Advancing the way the world uses the cloud
> > > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > > >>>
> > > >>
> > > >
> > > >
> > > > --
> > > > *Mike Tutkowski*
> > > > *Senior CloudStack Developer, SolidFire Inc.*
> > > > e: mike.tutkowski@solidfire.com
> > > > o: 303.746.7302
> > > > Advancing the way the world uses the cloud
> > > > <http://solidfire.com/solution/overview/?video=play>*™*
> > > >
> > > Find out more about ShapeBlue and our range of CloudStack related
> > services:
> > > IaaS Cloud Design & Build
> > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > > IaaS deployment framework <http://shapeblue.com/csforge/>
> > > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> > CloudStack
> > > Software Engineering
> > > <http://shapeblue.com/cloudstack-software-engineering/>
> > > CloudStack Infrastructure Support
> > > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
> > >
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Some of the weirdness is around terminology.

For most systems I've worked on, a snapshot and a backup are two completely
different things (but CloudStack has traditionally used the term "volume
snapshot" to mean backup).

I will put in a SolidFire "plug" here and say, though, that if your primary
storage is running on SolidFire that it is unlikely you'll encounter an
issue where your primary storage goes offline (and you'll even maintain
your performance guarantees during failure scenarios and upgrades, as
well). That being the case, it is less useful to require a backup to Swift
(but it's perfectly OK if that's what we want to do here).

On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq <sy...@gmail.com>
wrote:

> Paul,
>
> I believe with the current implementation of Snapshots on managed storage
> (SolidFire) the snapshots are never exported to the secondary storage.
> While this solves the problem of having
> snapshots taking forever to get to sec storage, this leaves us with a huge
> liability if our primary storage goes down. We see snapshots as our
> recovery path because we store them in Swift which is reliable and
> resilient to failures.
>
> With Storage snpashots our goal is to have Volume snapshots always backed
> up to secondary storage and Storage Snapshots stay on the primary storage.
> A provider could potentially mix both
> these and solve the problem that you mentioned where you want to meet
> user's  expectation of a snapshot (ie backup to sec storage) while having
> an ability  to utilize faster sanpshots (i.e. on the device)
>
> Hope this clarifies things.
>
> Thanks,
> -Syed
>
> On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus <pa...@shapeblue.com>
> wrote:
>
> > HI guys,
> >
> > Could someone point me to the Jira bug of FS for the SAN-snapshot feature
> > in 4.6 which is mentioned.
> >
> > From my discussions with users and operators around snapshots I'd make
> the
> > following observations:
> > a. 'users' use snapshots as backups (both long-term and short term) with
> > the expectation that they can use them for recovery if required.
> > b. operators fall back to snapshots if something has gone wrong with
> > primary storage.
> > c. users sometimes want to be able to export snapshots as well as create
> > new VMs from their snapshots
> > d. snapshots are a currently a massive pain for operators, I know at
> least
> > one public cloud who have snapshots which take 2 days to complete.
> > e. snapshots (as they are) can't be used for multiple LVM disks.
> >
> > I think the process Mike has used in the SolidFire plugin (only moving
> the
> > disk image to secondary storage when you absolutely have to) is a very
> good
> > and pragmatic solution. I wonder what problems an operator might
> experience
> > if they have an issue with a given primary storage pool in a cluster. (I
> > know that that is REALLY unlikely in the SolidFire case Mike :) ) And if
> > the transfer from primary to secondary is slow, the time to being able to
> > create a template or export the volume will be slow.
> >
> > So for me the issue is around making sure that the end users expectations
> > are met (while improving the speed/efficiency of the back end)
> >
> >
> > [image: ShapeBlue] <http://www.shapeblue.com>
> > Paul Angus
> > VP Technology ,  ShapeBlue
> > d:  *+44 203 617 0528 | s: +44 203 603 0540*
> > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> > *+44 7711 418784* <+44%207711%20418784>
> > e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> > <pa...@cloudyangus>  |  w:
> > *www.shapeblue.com* <http://www.shapeblue.com>
> > a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> > Services India LLP is a company incorporated in India and is operated
> under
> > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> > company incorporated in Brasil and is operated under license from Shape
> > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue
> is
> > a registered trademark.
> > This email and any attachments to it may be confidential and are intended
> > solely for the use of the individual to whom it is addressed. Any views
> or
> > opinions expressed are solely those of the author and do not necessarily
> > represent those of Shape Blue Ltd or related companies. If you are not
> the
> > intended recipient of this email, you must neither take any action based
> > upon its contents, nor copy or show it to anyone. Please contact the
> sender
> > if you believe you have received this email in error.
> >
> >
> > -----Original Message-----
> > From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> > Sent: Friday, February 5, 2016 12:56 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [Propose][New Feature] Storage Snapshots
> >
> > Hi Mike,
> >
> > The idea of introducing a new API: StorageSnapshot for managed storage is
> > because the VolumeSnapshot default, or expected, behavior is to archive
> > snapshots into the Secondary Storage. So a StorageSnapshot API would be
> for
> > snapshot that remain on the managed storage appliance.
> >
> > Quickly looking at the API doc and I don't see a strong requirement for
> > volume snapshots to be moved into secondary storage. So, maybe
> > StorageSnapshot API is not useful, but both use cases are required. A
> > snapshot that remain on the managed storage, and another type of snapshot
> > that end up into the secondary storage. Since you've done a lot of work,
> > might easier to just add a parameter to the current snapshot API that
> would
> > trigger an extraction of the storage snapshot into the secondary storage?
> >
> > PL
> >
> >
> >
> > On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> > mike.tutkowski@solidfire.com
> > > wrote:
> >
> > > I think that all sounds reasonable then - thanks!
> > >
> > > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <sy...@gmail.com>
> > > wrote:
> > >
> > >> You are correct Mike in terms of the requirements. One of our earlier
> > >> iterations on this was to have an argument to the create snapshot API
> > >> which decides whether to backup the volume to sec storage but we
> > >> realized it would make management of snapshots quite messy so we
> > >> proposed a new api instead.
> > >>
> > >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> > >> <mi...@solidfire.com>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Just to make sure I understand all the requirements here:
> > >>>
> > >>> 1) This relates only to managed storage (1:1 mapping between a
> > >>> virtual disk and a backend SAN volume).
> > >>>
> > >>> 2) We want to take the current (introduced in 4.6) functionality,
> > >>> which creates a snapshot on the SAN, and extend it via a config
> > >>> option (or
> > >>> something) to not only take the SAN snapshot, but to copy the
> > >>> underlying VHD (XenServer only) to NFS.
> > >>>
> > >>> 3) The SAN snapshot is always taken. It's the backup to NFS that is
> > >>> optional.
> > >>>
> > >>> 4) Templates can be created from the snapshot that's on the SAN
> > >>> (already works).
> > >>>
> > >>> 5) CloudStack volumes can be created from the snapshot that's on the
> > >>> SAN (already works as long as the new CloudStack volume ends up on
> > >>> the same primary storage).
> > >>>
> > >>> Would we have a need for a storage snapshot API then or would that
> > >>> just be the standard volume snapshot without the backup to NFS?
> > >>>
> > >>> Thanks!
> > >>> Mike
> > >>>
> > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> > >>> <sy...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Is it possible to have both functionalities (snapshot on SAN & Sec
> > >>>> Storage) coexist? Because Ideally, we would like to have both.
> > >>>> For example, some of our customers want to implement their own
> > >>>> backup strategies and do encryption to their backups which is a
> > >>>> perfect use case for Storage Snapshot while our other customers
> > >>>> will still keep using the standard volume snapshot.
> > >>>>
> > >>>> To keep things backward compatible, we can add a setting which says
> > >>>> to not upload on secondary storage, because, after all, you would
> > >>>> take a SAN snapshot first when doing a Volume Snapshot. You could
> > >>>> stop the process there and not do the upload.
> > >>>>
> > >>>> What do you think about this approach?
> > >>>>
> > >>>> Thanks,
> > >>>> -Syed
> > >>>>
> > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> > >>>> mike.tutkowski@solidfire.com> wrote:
> > >>>>
> > >>>>> So, this is just me thinking out load here, but if a given
> > >>>>> CloudStack cloud doesn't actually need to provide both the ability
> > >>>>> to take a SAN snapshot and export it to NFS (if just taking a SAN
> > >>>>> snapshot is OK), then we might be able to get away with no new API
> > >>>>> calls and simply implement a new custom snapshot strategy and data
> > >>>>> motion strategy to handle the case where the CloudStack cloud does
> > >>>>> want both a SAN snapshot and exported-to-NFS backup.
> > >>>>>
> > >>>>> In other words, the "default" behavior would be to use the
> > >>>>> snapshot strategy and data motion strategy that we already have
> > >>>>> (the one that only takes a SAN snapshot).
> > >>>>>
> > >>>>> If your CloudStack cloud, however, wants to take a SAN snapshot
> > >>>>> and have the data exported to NFS, then we could have you
> > >>>>> manipulate a Swing config file to make use of a new snapshot
> > >>>>> strategy and data motion strategy that performs both of these
> > activities.
> > >>>>>
> > >>>>> This way, the old behavior is still the default for users, but
> > >>>>> CloudStack admins can change this behavior via configuration.
> > >>>>>
> > >>>>> Thoughts?
> > >>>>>
> > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> > >>>>> mike.tutkowski@solidfire.com> wrote:
> > >>>>>
> > >>>>>> Right...I think we will need to come up with a viable upgrade
> > >>>>>> path or some reasonable way for them to move from the old way to
> > >>>>>> the new way (and some obvious way that they will know they need to
> > do this).
> > >>>>>>
> > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> > >>>>>> syed1.mushtaq@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> I'm not really sure about the upgrade path however, customers
> > >>>>>>> who are using 4.6 and are on a managed storage would no longer
> > >>>>>>> have the same functionality with Volume Snapshots.
> > >>>>>>>
> > >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> > >>>>>>> syed1.mushtaq@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> So if I understand correctly, currently taking a Volume
> > >>>>>>>> Snapshots of a volume on a managed storage keeps it on the
> > >>>>>>>> storage array. As a part of this feature, we can make sure that
> > >>>>>>>> Volume Snapshots on managed storage are uploaded to the
> > >>>>>>>> secondary storage. This would make the Volume Snapshot feature
> > >>>>>>>> behave the same regardless of the storage (managed or
> > >>>>>>>> non-managed) And, for utilizing the efficient backend storage
> > capabilities, we can use the new storage snapshots API.
> > >>>>>>>>
> > >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> > >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi everyone,
> > >>>>>>>>>
> > >>>>>>>>> Whatever we do here, we need to have a plan to deal with the
> > >>>>>>>>> fact that we already have a feature (in 4.6 and later) that
> > >>>>>>>>> allows you to use the existing volume-snapshot APIs to create
> > >>>>>>>>> a volume snapshot (for managed
> > >>>>>>>>> storage) that resides on a backend SAN (using a custom
> > >>>>>>>>> snapshot strategy and a custom data motion strategy).
> > >>>>>>>>>
> > >>>>>>>>> If these new APIs go in, then how should the original
> > >>>>>>>>> implementation (present in 4.6 and later) be changed? If it is
> > >>>>>>>>> changed, how do we support customers who are already using the
> > >>>>>>>>> original volume-snapshot API to take snapshots on a backend
> SAN?
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Mike
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> > >>>>>>>>> wstevens@cloudops.com> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Will you be able to create a Template from a StorageSnapshot?
> > >>>>>>>>>> If yes, will the template be stored in the secondary storage
> > >>>>>>>>>> like normal templates or will that be handled somehow on the
> > vendor side?
> > >>>>>>>>>>
> > >>>>>>>>>> *Will STEVENS*
> > >>>>>>>>>> Lead Developer
> > >>>>>>>>>>
> > >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> > >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Thanks Will!!!
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> > >>>>>>>>>>> wstevens@cloudops.com> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira ticket
> > >>>>>>>>>>>> because it was not clear in the 'mention' section because
> > >>>>>>>>>>>> it shows as a page 'you do not have permission to'.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> *Will STEVENS*
> > >>>>>>>>>>>> Lead Developer
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> > >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> > >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> > >>>>>>>>>>>> <sahmed@cloudops.com
> > >>>>>>>>>>>> > wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Design Spec:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> > >>>>>>>>>>>>> rageSnapshot++API
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jira Ticket
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi All,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots on
> > >>>>>>>>>>>>> managed storage backends like SolidFire. Snapshots on
> > >>>>>>>>>>>>> current managed storage stay on the device which is
> > >>>>>>>>>>>>> contrary to what CloudStack calls snpshots.
> > >>>>>>>>>>>>> But taking snapshots
> > >>>>>>>>>>>>> on storage and keeping it there has its own advantages and
> > >>>>>>>>>>>>> we would ideally like to have both ways of doing
> > >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
> > >>>>>>>>>>>>> snapshots on backend storage.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> What do you guys think of this feature? I would love to
> > >>>>>>>>>>>>> have some feedback. I am working on making the design spec
> > >>>>>>>>>>>>> more concrete but wanted to have a high level feedback
> > >>>>>>>>>>>>> first before starting to work on it.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>> -Syed
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> *Mike Tutkowski*
> > >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > >>>>>>>>> e: mike.tutkowski@solidfire.com
> > >>>>>>>>> o: 303.746.7302
> > >>>>>>>>> Advancing the way the world uses the cloud
> > >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> *Mike Tutkowski*
> > >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > >>>>>> e: mike.tutkowski@solidfire.com
> > >>>>>> o: 303.746.7302
> > >>>>>> Advancing the way the world uses the cloud
> > >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> *Mike Tutkowski*
> > >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> > >>>>> e: mike.tutkowski@solidfire.com
> > >>>>> o: 303.746.7302
> > >>>>> Advancing the way the world uses the cloud
> > >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> *Mike Tutkowski*
> > >>> *Senior CloudStack Developer, SolidFire Inc.*
> > >>> e: mike.tutkowski@solidfire.com
> > >>> o: 303.746.7302
> > >>> Advancing the way the world uses the cloud
> > >>> <http://solidfire.com/solution/overview/?video=play>*™*
> > >>>
> > >>
> > >
> > >
> > > --
> > > *Mike Tutkowski*
> > > *Senior CloudStack Developer, SolidFire Inc.*
> > > e: mike.tutkowski@solidfire.com
> > > o: 303.746.7302
> > > Advancing the way the world uses the cloud
> > > <http://solidfire.com/solution/overview/?video=play>*™*
> > >
> > Find out more about ShapeBlue and our range of CloudStack related
> services:
> > IaaS Cloud Design & Build
> > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> > IaaS deployment framework <http://shapeblue.com/csforge/>
> > CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack
> > Software Engineering
> > <http://shapeblue.com/cloudstack-software-engineering/>
> > CloudStack Infrastructure Support
> > <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

Paul,

I believe with the current implementation of Snapshots on managed storage
(SolidFire) the snapshots are never exported to the secondary storage.
While this solves the problem of having
snapshots taking forever to get to sec storage, this leaves us with a huge
liability if our primary storage goes down. We see snapshots as our
recovery path because we store them in Swift which is reliable and
resilient to failures.

With Storage snpashots our goal is to have Volume snapshots always backed
up to secondary storage and Storage Snapshots stay on the primary storage.
A provider could potentially mix both
these and solve the problem that you mentioned where you want to meet
user's  expectation of a snapshot (ie backup to sec storage) while having
an ability  to utilize faster sanpshots (i.e. on the device)

Hope this clarifies things.

Thanks,
-Syed

On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus <pa...@shapeblue.com> wrote:

> HI guys,
>
> Could someone point me to the Jira bug of FS for the SAN-snapshot feature
> in 4.6 which is mentioned.
>
> From my discussions with users and operators around snapshots I'd make the
> following observations:
> a. 'users' use snapshots as backups (both long-term and short term) with
> the expectation that they can use them for recovery if required.
> b. operators fall back to snapshots if something has gone wrong with
> primary storage.
> c. users sometimes want to be able to export snapshots as well as create
> new VMs from their snapshots
> d. snapshots are a currently a massive pain for operators, I know at least
> one public cloud who have snapshots which take 2 days to complete.
> e. snapshots (as they are) can't be used for multiple LVM disks.
>
> I think the process Mike has used in the SolidFire plugin (only moving the
> disk image to secondary storage when you absolutely have to) is a very good
> and pragmatic solution. I wonder what problems an operator might experience
> if they have an issue with a given primary storage pool in a cluster. (I
> know that that is REALLY unlikely in the SolidFire case Mike :) ) And if
> the transfer from primary to secondary is slow, the time to being able to
> create a template or export the volume will be slow.
>
> So for me the issue is around making sure that the end users expectations
> are met (while improving the speed/efficiency of the back end)
>
>
> [image: ShapeBlue] <http://www.shapeblue.com>
> Paul Angus
> VP Technology ,  ShapeBlue
> d:  *+44 203 617 0528 | s: +44 203 603 0540*
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>  |  m:
> *+44 7711 418784* <+44%207711%20418784>
> e:  *paul.angus@shapeblue.com | t: @cloudyangus*
> <pa...@cloudyangus>  |  w:
> *www.shapeblue.com* <http://www.shapeblue.com>
> a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> Services India LLP is a company incorporated in India and is operated under
> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> company incorporated in Brasil and is operated under license from Shape
> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
> a registered trademark.
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error.
>
>
> -----Original Message-----
> From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
> Sent: Friday, February 5, 2016 12:56 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [Propose][New Feature] Storage Snapshots
>
> Hi Mike,
>
> The idea of introducing a new API: StorageSnapshot for managed storage is
> because the VolumeSnapshot default, or expected, behavior is to archive
> snapshots into the Secondary Storage. So a StorageSnapshot API would be for
> snapshot that remain on the managed storage appliance.
>
> Quickly looking at the API doc and I don't see a strong requirement for
> volume snapshots to be moved into secondary storage. So, maybe
> StorageSnapshot API is not useful, but both use cases are required. A
> snapshot that remain on the managed storage, and another type of snapshot
> that end up into the secondary storage. Since you've done a lot of work,
> might easier to just add a parameter to the current snapshot API that would
> trigger an extraction of the storage snapshot into the secondary storage?
>
> PL
>
>
>
> On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com
> > wrote:
>
> > I think that all sounds reasonable then - thanks!
> >
> > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <sy...@gmail.com>
> > wrote:
> >
> >> You are correct Mike in terms of the requirements. One of our earlier
> >> iterations on this was to have an argument to the create snapshot API
> >> which decides whether to backup the volume to sec storage but we
> >> realized it would make management of snapshots quite messy so we
> >> proposed a new api instead.
> >>
> >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
> >> <mi...@solidfire.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Just to make sure I understand all the requirements here:
> >>>
> >>> 1) This relates only to managed storage (1:1 mapping between a
> >>> virtual disk and a backend SAN volume).
> >>>
> >>> 2) We want to take the current (introduced in 4.6) functionality,
> >>> which creates a snapshot on the SAN, and extend it via a config
> >>> option (or
> >>> something) to not only take the SAN snapshot, but to copy the
> >>> underlying VHD (XenServer only) to NFS.
> >>>
> >>> 3) The SAN snapshot is always taken. It's the backup to NFS that is
> >>> optional.
> >>>
> >>> 4) Templates can be created from the snapshot that's on the SAN
> >>> (already works).
> >>>
> >>> 5) CloudStack volumes can be created from the snapshot that's on the
> >>> SAN (already works as long as the new CloudStack volume ends up on
> >>> the same primary storage).
> >>>
> >>> Would we have a need for a storage snapshot API then or would that
> >>> just be the standard volume snapshot without the backup to NFS?
> >>>
> >>> Thanks!
> >>> Mike
> >>>
> >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
> >>> <sy...@gmail.com>
> >>> wrote:
> >>>
> >>>> Is it possible to have both functionalities (snapshot on SAN & Sec
> >>>> Storage) coexist? Because Ideally, we would like to have both.
> >>>> For example, some of our customers want to implement their own
> >>>> backup strategies and do encryption to their backups which is a
> >>>> perfect use case for Storage Snapshot while our other customers
> >>>> will still keep using the standard volume snapshot.
> >>>>
> >>>> To keep things backward compatible, we can add a setting which says
> >>>> to not upload on secondary storage, because, after all, you would
> >>>> take a SAN snapshot first when doing a Volume Snapshot. You could
> >>>> stop the process there and not do the upload.
> >>>>
> >>>> What do you think about this approach?
> >>>>
> >>>> Thanks,
> >>>> -Syed
> >>>>
> >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> >>>> mike.tutkowski@solidfire.com> wrote:
> >>>>
> >>>>> So, this is just me thinking out load here, but if a given
> >>>>> CloudStack cloud doesn't actually need to provide both the ability
> >>>>> to take a SAN snapshot and export it to NFS (if just taking a SAN
> >>>>> snapshot is OK), then we might be able to get away with no new API
> >>>>> calls and simply implement a new custom snapshot strategy and data
> >>>>> motion strategy to handle the case where the CloudStack cloud does
> >>>>> want both a SAN snapshot and exported-to-NFS backup.
> >>>>>
> >>>>> In other words, the "default" behavior would be to use the
> >>>>> snapshot strategy and data motion strategy that we already have
> >>>>> (the one that only takes a SAN snapshot).
> >>>>>
> >>>>> If your CloudStack cloud, however, wants to take a SAN snapshot
> >>>>> and have the data exported to NFS, then we could have you
> >>>>> manipulate a Swing config file to make use of a new snapshot
> >>>>> strategy and data motion strategy that performs both of these
> activities.
> >>>>>
> >>>>> This way, the old behavior is still the default for users, but
> >>>>> CloudStack admins can change this behavior via configuration.
> >>>>>
> >>>>> Thoughts?
> >>>>>
> >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> >>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>
> >>>>>> Right...I think we will need to come up with a viable upgrade
> >>>>>> path or some reasonable way for them to move from the old way to
> >>>>>> the new way (and some obvious way that they will know they need to
> do this).
> >>>>>>
> >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> >>>>>> syed1.mushtaq@gmail.com> wrote:
> >>>>>>
> >>>>>>> I'm not really sure about the upgrade path however, customers
> >>>>>>> who are using 4.6 and are on a managed storage would no longer
> >>>>>>> have the same functionality with Volume Snapshots.
> >>>>>>>
> >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> >>>>>>> syed1.mushtaq@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> So if I understand correctly, currently taking a Volume
> >>>>>>>> Snapshots of a volume on a managed storage keeps it on the
> >>>>>>>> storage array. As a part of this feature, we can make sure that
> >>>>>>>> Volume Snapshots on managed storage are uploaded to the
> >>>>>>>> secondary storage. This would make the Volume Snapshot feature
> >>>>>>>> behave the same regardless of the storage (managed or
> >>>>>>>> non-managed) And, for utilizing the efficient backend storage
> capabilities, we can use the new storage snapshots API.
> >>>>>>>>
> >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi everyone,
> >>>>>>>>>
> >>>>>>>>> Whatever we do here, we need to have a plan to deal with the
> >>>>>>>>> fact that we already have a feature (in 4.6 and later) that
> >>>>>>>>> allows you to use the existing volume-snapshot APIs to create
> >>>>>>>>> a volume snapshot (for managed
> >>>>>>>>> storage) that resides on a backend SAN (using a custom
> >>>>>>>>> snapshot strategy and a custom data motion strategy).
> >>>>>>>>>
> >>>>>>>>> If these new APIs go in, then how should the original
> >>>>>>>>> implementation (present in 4.6 and later) be changed? If it is
> >>>>>>>>> changed, how do we support customers who are already using the
> >>>>>>>>> original volume-snapshot API to take snapshots on a backend SAN?
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Mike
> >>>>>>>>>
> >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> >>>>>>>>> wstevens@cloudops.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Will you be able to create a Template from a StorageSnapshot?
> >>>>>>>>>> If yes, will the template be stored in the secondary storage
> >>>>>>>>>> like normal templates or will that be handled somehow on the
> vendor side?
> >>>>>>>>>>
> >>>>>>>>>> *Will STEVENS*
> >>>>>>>>>> Lead Developer
> >>>>>>>>>>
> >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> >>>>>>>>>> cloudops.com *|* tw @CloudOps_
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks Will!!!
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> >>>>>>>>>>> wstevens@cloudops.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira ticket
> >>>>>>>>>>>> because it was not clear in the 'mention' section because
> >>>>>>>>>>>> it shows as a page 'you do not have permission to'.
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Will STEVENS*
> >>>>>>>>>>>> Lead Developer
> >>>>>>>>>>>>
> >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
> >>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
> >>>>>>>>>>>> <sahmed@cloudops.com
> >>>>>>>>>>>> > wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Design Spec:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
> >>>>>>>>>>>>> rageSnapshot++API
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jira Ticket
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots on
> >>>>>>>>>>>>> managed storage backends like SolidFire. Snapshots on
> >>>>>>>>>>>>> current managed storage stay on the device which is
> >>>>>>>>>>>>> contrary to what CloudStack calls snpshots.
> >>>>>>>>>>>>> But taking snapshots
> >>>>>>>>>>>>> on storage and keeping it there has its own advantages and
> >>>>>>>>>>>>> we would ideally like to have both ways of doing
> >>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
> >>>>>>>>>>>>> snapshots on backend storage.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> What do you guys think of this feature? I would love to
> >>>>>>>>>>>>> have some feedback. I am working on making the design spec
> >>>>>>>>>>>>> more concrete but wanted to have a high level feedback
> >>>>>>>>>>>>> first before starting to work on it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> -Syed
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> *Mike Tutkowski*
> >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>> o: 303.746.7302
> >>>>>>>>> Advancing the way the world uses the cloud
> >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> *Mike Tutkowski*
> >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>> e: mike.tutkowski@solidfire.com
> >>>>>> o: 303.746.7302
> >>>>>> Advancing the way the world uses the cloud
> >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> *Mike Tutkowski*
> >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>> e: mike.tutkowski@solidfire.com
> >>>>> o: 303.746.7302
> >>>>> Advancing the way the world uses the cloud
> >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> *Mike Tutkowski*
> >>> *Senior CloudStack Developer, SolidFire Inc.*
> >>> e: mike.tutkowski@solidfire.com
> >>> o: 303.746.7302
> >>> Advancing the way the world uses the cloud
> >>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>
> >>
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack
> Software Engineering
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support
> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
>

RE: [Propose][New Feature] Storage Snapshots

[Paul Angus <pa...@shapeblue.com>]

HI guys,

Could someone point me to the Jira bug of FS for the SAN-snapshot feature in 4.6 which is mentioned.

>From my discussions with users and operators around snapshots I'd make the following observations:
a. 'users' use snapshots as backups (both long-term and short term) with the expectation that they can use them for recovery if required.
b. operators fall back to snapshots if something has gone wrong with primary storage.
c. users sometimes want to be able to export snapshots as well as create new VMs from their snapshots
d. snapshots are a currently a massive pain for operators, I know at least one public cloud who have snapshots which take 2 days to complete.
e. snapshots (as they are) can't be used for multiple LVM disks.

I think the process Mike has used in the SolidFire plugin (only moving the disk image to secondary storage when you absolutely have to) is a very good and pragmatic solution. I wonder what problems an operator might experience if they have an issue with a given primary storage pool in a cluster. (I know that that is REALLY unlikely in the SolidFire case Mike :) ) And if the transfer from primary to secondary is slow, the time to being able to create a template or export the volume will be slow.

So for me the issue is around making sure that the end users expectations are met (while improving the speed/efficiency of the back end)


[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology   ,       ShapeBlue


d:      +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540>     |      m:      +44 7711 418784<tel:+44%207711%20418784>

e:      paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus>      |      w:      www.shapeblue.com<http://www.shapeblue.com>

a:      53 Chandos Place, Covent Garden London WC2N 4HS UK


[cid:imagefc0824.png@4535cfe7.4c866aec]


Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.




-----Original Message-----
From: Pierre-Luc Dion [mailto:pdion@cloudops.com]
Sent: Friday, February 5, 2016 12:56 PM
To: dev@cloudstack.apache.org
Subject: Re: [Propose][New Feature] Storage Snapshots

Hi Mike,

The idea of introducing a new API: StorageSnapshot for managed storage is because the VolumeSnapshot default, or expected, behavior is to archive snapshots into the Secondary Storage. So a StorageSnapshot API would be for snapshot that remain on the managed storage appliance.

Quickly looking at the API doc and I don't see a strong requirement for volume snapshots to be moved into secondary storage. So, maybe StorageSnapshot API is not useful, but both use cases are required. A snapshot that remain on the managed storage, and another type of snapshot that end up into the secondary storage. Since you've done a lot of work, might easier to just add a parameter to the current snapshot API that would trigger an extraction of the storage snapshot into the secondary storage?

PL



On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> I think that all sounds reasonable then - thanks!
>
> On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
>> You are correct Mike in terms of the requirements. One of our earlier
>> iterations on this was to have an argument to the create snapshot API
>> which decides whether to backup the volume to sec storage but we
>> realized it would make management of snapshots quite messy so we
>> proposed a new api instead.
>>
>> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski
>> <mi...@solidfire.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Just to make sure I understand all the requirements here:
>>>
>>> 1) This relates only to managed storage (1:1 mapping between a
>>> virtual disk and a backend SAN volume).
>>>
>>> 2) We want to take the current (introduced in 4.6) functionality,
>>> which creates a snapshot on the SAN, and extend it via a config
>>> option (or
>>> something) to not only take the SAN snapshot, but to copy the
>>> underlying VHD (XenServer only) to NFS.
>>>
>>> 3) The SAN snapshot is always taken. It's the backup to NFS that is
>>> optional.
>>>
>>> 4) Templates can be created from the snapshot that's on the SAN
>>> (already works).
>>>
>>> 5) CloudStack volumes can be created from the snapshot that's on the
>>> SAN (already works as long as the new CloudStack volume ends up on
>>> the same primary storage).
>>>
>>> Would we have a need for a storage snapshot API then or would that
>>> just be the standard volume snapshot without the backup to NFS?
>>>
>>> Thanks!
>>> Mike
>>>
>>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq
>>> <sy...@gmail.com>
>>> wrote:
>>>
>>>> Is it possible to have both functionalities (snapshot on SAN & Sec
>>>> Storage) coexist? Because Ideally, we would like to have both.
>>>> For example, some of our customers want to implement their own
>>>> backup strategies and do encryption to their backups which is a
>>>> perfect use case for Storage Snapshot while our other customers
>>>> will still keep using the standard volume snapshot.
>>>>
>>>> To keep things backward compatible, we can add a setting which says
>>>> to not upload on secondary storage, because, after all, you would
>>>> take a SAN snapshot first when doing a Volume Snapshot. You could
>>>> stop the process there and not do the upload.
>>>>
>>>> What do you think about this approach?
>>>>
>>>> Thanks,
>>>> -Syed
>>>>
>>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
>>>> mike.tutkowski@solidfire.com> wrote:
>>>>
>>>>> So, this is just me thinking out load here, but if a given
>>>>> CloudStack cloud doesn't actually need to provide both the ability
>>>>> to take a SAN snapshot and export it to NFS (if just taking a SAN
>>>>> snapshot is OK), then we might be able to get away with no new API
>>>>> calls and simply implement a new custom snapshot strategy and data
>>>>> motion strategy to handle the case where the CloudStack cloud does
>>>>> want both a SAN snapshot and exported-to-NFS backup.
>>>>>
>>>>> In other words, the "default" behavior would be to use the
>>>>> snapshot strategy and data motion strategy that we already have
>>>>> (the one that only takes a SAN snapshot).
>>>>>
>>>>> If your CloudStack cloud, however, wants to take a SAN snapshot
>>>>> and have the data exported to NFS, then we could have you
>>>>> manipulate a Swing config file to make use of a new snapshot
>>>>> strategy and data motion strategy that performs both of these activities.
>>>>>
>>>>> This way, the old behavior is still the default for users, but
>>>>> CloudStack admins can change this behavior via configuration.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>
>>>>>> Right...I think we will need to come up with a viable upgrade
>>>>>> path or some reasonable way for them to move from the old way to
>>>>>> the new way (and some obvious way that they will know they need to do this).
>>>>>>
>>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
>>>>>> syed1.mushtaq@gmail.com> wrote:
>>>>>>
>>>>>>> I'm not really sure about the upgrade path however, customers
>>>>>>> who are using 4.6 and are on a managed storage would no longer
>>>>>>> have the same functionality with Volume Snapshots.
>>>>>>>
>>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
>>>>>>> syed1.mushtaq@gmail.com> wrote:
>>>>>>>
>>>>>>>> So if I understand correctly, currently taking a Volume
>>>>>>>> Snapshots of a volume on a managed storage keeps it on the
>>>>>>>> storage array. As a part of this feature, we can make sure that
>>>>>>>> Volume Snapshots on managed storage are uploaded to the
>>>>>>>> secondary storage. This would make the Volume Snapshot feature
>>>>>>>> behave the same regardless of the storage (managed or
>>>>>>>> non-managed) And, for utilizing the efficient backend storage capabilities, we can use the new storage snapshots API.
>>>>>>>>
>>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> Whatever we do here, we need to have a plan to deal with the
>>>>>>>>> fact that we already have a feature (in 4.6 and later) that
>>>>>>>>> allows you to use the existing volume-snapshot APIs to create
>>>>>>>>> a volume snapshot (for managed
>>>>>>>>> storage) that resides on a backend SAN (using a custom
>>>>>>>>> snapshot strategy and a custom data motion strategy).
>>>>>>>>>
>>>>>>>>> If these new APIs go in, then how should the original
>>>>>>>>> implementation (present in 4.6 and later) be changed? If it is
>>>>>>>>> changed, how do we support customers who are already using the
>>>>>>>>> original volume-snapshot API to take snapshots on a backend SAN?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
>>>>>>>>> wstevens@cloudops.com> wrote:
>>>>>>>>>
>>>>>>>>>> Will you be able to create a Template from a StorageSnapshot?
>>>>>>>>>> If yes, will the template be stored in the secondary storage
>>>>>>>>>> like normal templates or will that be handled somehow on the vendor side?
>>>>>>>>>>
>>>>>>>>>> *Will STEVENS*
>>>>>>>>>> Lead Developer
>>>>>>>>>>
>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
>>>>>>>>>> cloudops.com *|* tw @CloudOps_
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
>>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Will!!!
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
>>>>>>>>>>> wstevens@cloudops.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I explicitly linked the Design Spec in the Jira ticket
>>>>>>>>>>>> because it was not clear in the 'mention' section because
>>>>>>>>>>>> it shows as a page 'you do not have permission to'.
>>>>>>>>>>>>
>>>>>>>>>>>> *Will STEVENS*
>>>>>>>>>>>> Lead Developer
>>>>>>>>>>>>
>>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w
>>>>>>>>>>>> cloudops.com *|* tw @CloudOps_
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed
>>>>>>>>>>>> <sahmed@cloudops.com
>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Design Spec:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Sto
>>>>>>>>>>>>> rageSnapshot++API
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jira Ticket
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots on
>>>>>>>>>>>>> managed storage backends like SolidFire. Snapshots on
>>>>>>>>>>>>> current managed storage stay on the device which is
>>>>>>>>>>>>> contrary to what CloudStack calls snpshots.
>>>>>>>>>>>>> But taking snapshots
>>>>>>>>>>>>> on storage and keeping it there has its own advantages and
>>>>>>>>>>>>> we would ideally like to have both ways of doing
>>>>>>>>>>>>> snapshots. This proposal adds 4 new APIs to create
>>>>>>>>>>>>> snapshots on backend storage.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you guys think of this feature? I would love to
>>>>>>>>>>>>> have some feedback. I am working on making the design spec
>>>>>>>>>>>>> more concrete but wanted to have a high level feedback
>>>>>>>>>>>>> first before starting to work on it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> -Syed
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Mike Tutkowski*
>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>> o: 303.746.7302
>>>>>>>>> Advancing the way the world uses the cloud
>>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Mike Tutkowski*
>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>> e: mike.tutkowski@solidfire.com
>>>>>> o: 303.746.7302
>>>>>> Advancing the way the world uses the cloud
>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Mike Tutkowski*
>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>> e: mike.tutkowski@solidfire.com
>>>>> o: 303.746.7302
>>>>> Advancing the way the world uses the cloud
>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkowski@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the cloud
>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>
>>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Hey Pierre-Luc,

For me, it's not a matter of how much work has already been done, but
rather maintaining backward compatibility.

Today, if using managed storage, if you elect to create a volume snapshot
(and the storage driver supports this), it is only taken on the backend
system (in my case, the SAN)...it's not backed up to NFS.

We just need to maintain that behavior (although we can extend it with a
flag or something that tells us to also back the volume snapshot up to
NFS...that would be perfectly fine).

What do you think about that?

In summary:

* Non-managed storage (traditional storage where an admin pre-creates an SR
(or datastore), hands it to CS as primary storage, and then runs multiple
virtual disks on it) implements volume snapshots as backups to NFS.

* Managed storage (1:1 mapping between an SR (or datastore) and a virtual
disk) implements volume snapshots as snapshots on the backend system (a
SAN, for me). Volume snapshots for managed storage currently is only
implemented for XenServer (first in 4.6).

Thanks,
Mike

On Fri, Feb 5, 2016 at 5:56 AM, Pierre-Luc Dion <pd...@cloudops.com> wrote:

> Hi Mike,
>
> The idea of introducing a new API: StorageSnapshot for managed storage is
> because the VolumeSnapshot default, or expected, behavior is to archive
> snapshots into the Secondary Storage. So a StorageSnapshot API would be for
> snapshot that remain on the managed storage appliance.
>
> Quickly looking at the API doc and I don't see a strong requirement for
> volume snapshots to be moved into secondary storage. So, maybe
> StorageSnapshot API is not useful, but both use cases are required. A
> snapshot that remain on the managed storage, and another type of snapshot
> that end up into the secondary storage. Since you've done a lot of work,
> might easier to just add a parameter to the current snapshot API that would
> trigger an extraction of the storage snapshot into the secondary storage?
>
> PL
>
>
>
> On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com
> > wrote:
>
> > I think that all sounds reasonable then - thanks!
> >
> > On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <sy...@gmail.com>
> > wrote:
> >
> >> You are correct Mike in terms of the requirements. One of our earlier
> >> iterations on this was to have an argument to the create snapshot API
> which
> >> decides whether to backup the volume to sec storage but we realized it
> >> would make management of snapshots quite messy so we proposed a new api
> >> instead.
> >>
> >> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski <
> mike.tutkowski@solidfire.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Just to make sure I understand all the requirements here:
> >>>
> >>> 1) This relates only to managed storage (1:1 mapping between a virtual
> >>> disk and a backend SAN volume).
> >>>
> >>> 2) We want to take the current (introduced in 4.6) functionality, which
> >>> creates a snapshot on the SAN, and extend it via a config option (or
> >>> something) to not only take the SAN snapshot, but to copy the
> underlying
> >>> VHD (XenServer only) to NFS.
> >>>
> >>> 3) The SAN snapshot is always taken. It's the backup to NFS that is
> >>> optional.
> >>>
> >>> 4) Templates can be created from the snapshot that's on the SAN
> (already
> >>> works).
> >>>
> >>> 5) CloudStack volumes can be created from the snapshot that's on the
> SAN
> >>> (already works as long as the new CloudStack volume ends up on the same
> >>> primary storage).
> >>>
> >>> Would we have a need for a storage snapshot API then or would that just
> >>> be the standard volume snapshot without the backup to NFS?
> >>>
> >>> Thanks!
> >>> Mike
> >>>
> >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq <sy...@gmail.com>
> >>> wrote:
> >>>
> >>>> Is it possible to have both functionalities (snapshot on SAN & Sec
> >>>> Storage) coexist? Because Ideally, we would like to have both.
> >>>> For example, some of our customers want to implement their own backup
> >>>> strategies and do encryption to their backups which is a perfect
> >>>> use case for Storage Snapshot while our other customers will still
> keep
> >>>> using the standard volume snapshot.
> >>>>
> >>>> To keep things backward compatible, we can add a setting which says to
> >>>> not upload on secondary storage, because, after all, you would take a
> >>>> SAN snapshot first when doing a Volume Snapshot. You could stop the
> >>>> process there and not do the upload.
> >>>>
> >>>> What do you think about this approach?
> >>>>
> >>>> Thanks,
> >>>> -Syed
> >>>>
> >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> >>>> mike.tutkowski@solidfire.com> wrote:
> >>>>
> >>>>> So, this is just me thinking out load here, but if a given CloudStack
> >>>>> cloud doesn't actually need to provide both the ability to take a SAN
> >>>>> snapshot and export it to NFS (if just taking a SAN snapshot is OK),
> then
> >>>>> we might be able to get away with no new API calls and simply
> implement a
> >>>>> new custom snapshot strategy and data motion strategy to handle the
> case
> >>>>> where the CloudStack cloud does want both a SAN snapshot and
> >>>>> exported-to-NFS backup.
> >>>>>
> >>>>> In other words, the "default" behavior would be to use the snapshot
> >>>>> strategy and data motion strategy that we already have (the one that
> only
> >>>>> takes a SAN snapshot).
> >>>>>
> >>>>> If your CloudStack cloud, however, wants to take a SAN snapshot and
> >>>>> have the data exported to NFS, then we could have you manipulate a
> Swing
> >>>>> config file to make use of a new snapshot strategy and data motion
> strategy
> >>>>> that performs both of these activities.
> >>>>>
> >>>>> This way, the old behavior is still the default for users, but
> >>>>> CloudStack admins can change this behavior via configuration.
> >>>>>
> >>>>> Thoughts?
> >>>>>
> >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> >>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>
> >>>>>> Right...I think we will need to come up with a viable upgrade path
> or
> >>>>>> some reasonable way for them to move from the old way to the new
> way (and
> >>>>>> some obvious way that they will know they need to do this).
> >>>>>>
> >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
> >>>>>> syed1.mushtaq@gmail.com> wrote:
> >>>>>>
> >>>>>>> I'm not really sure about the upgrade path however, customers who
> >>>>>>> are using 4.6 and are on a managed storage would no longer have
> the same
> >>>>>>> functionality with Volume Snapshots.
> >>>>>>>
> >>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
> >>>>>>> syed1.mushtaq@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> So if I understand correctly, currently taking a Volume Snapshots
> >>>>>>>> of a volume on a managed storage keeps it on the storage array.
> As a part
> >>>>>>>> of this feature, we can make sure that Volume Snapshots on
> managed storage
> >>>>>>>> are uploaded to the secondary storage. This would make the Volume
> Snapshot
> >>>>>>>> feature behave the same regardless of the storage (managed or
> non-managed)
> >>>>>>>> And, for utilizing the efficient backend storage capabilities, we
> can use
> >>>>>>>> the new storage snapshots API.
> >>>>>>>>
> >>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> >>>>>>>> mike.tutkowski@solidfire.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi everyone,
> >>>>>>>>>
> >>>>>>>>> Whatever we do here, we need to have a plan to deal with the fact
> >>>>>>>>> that we already have a feature (in 4.6 and later) that allows
> you to use
> >>>>>>>>> the existing volume-snapshot APIs to create a volume snapshot
> (for managed
> >>>>>>>>> storage) that resides on a backend SAN (using a custom snapshot
> strategy
> >>>>>>>>> and a custom data motion strategy).
> >>>>>>>>>
> >>>>>>>>> If these new APIs go in, then how should the original
> >>>>>>>>> implementation (present in 4.6 and later) be changed? If it is
> changed, how
> >>>>>>>>> do we support customers who are already using the original
> volume-snapshot
> >>>>>>>>> API to take snapshots on a backend SAN?
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Mike
> >>>>>>>>>
> >>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
> >>>>>>>>> wstevens@cloudops.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Will you be able to create a Template from a StorageSnapshot?
> If
> >>>>>>>>>> yes, will the template be stored in the secondary storage like
> normal
> >>>>>>>>>> templates or will that be handled somehow on the vendor side?
> >>>>>>>>>>
> >>>>>>>>>> *Will STEVENS*
> >>>>>>>>>> Lead Developer
> >>>>>>>>>>
> >>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> >>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >>>>>>>>>> w cloudops.com *|* tw @CloudOps_
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
> >>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks Will!!!
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
> >>>>>>>>>>> wstevens@cloudops.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I explicitly linked the Design Spec in the Jira ticket because
> >>>>>>>>>>>> it was not clear in the 'mention' section because it shows as
> a page 'you
> >>>>>>>>>>>> do not have permission to'.
> >>>>>>>>>>>>
> >>>>>>>>>>>> *Will STEVENS*
> >>>>>>>>>>>> Lead Developer
> >>>>>>>>>>>>
> >>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
> >>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >>>>>>>>>>>> w cloudops.com *|* tw @CloudOps_
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <
> sahmed@cloudops.com
> >>>>>>>>>>>> > wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Design Spec:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jira Ticket
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots on
> >>>>>>>>>>>>> managed storage backends like SolidFire. Snapshots on
> current managed
> >>>>>>>>>>>>> storage stay on the
> >>>>>>>>>>>>> device which is contrary to what CloudStack calls snpshots.
> >>>>>>>>>>>>> But taking snapshots
> >>>>>>>>>>>>> on storage and keeping it there has its own advantages and we
> >>>>>>>>>>>>> would ideally like
> >>>>>>>>>>>>> to have both ways of doing snapshots. This proposal adds 4
> new
> >>>>>>>>>>>>> APIs to
> >>>>>>>>>>>>> create snapshots on backend storage.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> What do you guys think of this feature? I would love to have
> >>>>>>>>>>>>> some feedback. I am working on making the design spec more
> concrete but
> >>>>>>>>>>>>> wanted to have
> >>>>>>>>>>>>> a high level feedback first before starting to work on it.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> -Syed
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> *Mike Tutkowski*
> >>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>>>>> e: mike.tutkowski@solidfire.com
> >>>>>>>>> o: 303.746.7302
> >>>>>>>>> Advancing the way the world uses the cloud
> >>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> *Mike Tutkowski*
> >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>>> e: mike.tutkowski@solidfire.com
> >>>>>> o: 303.746.7302
> >>>>>> Advancing the way the world uses the cloud
> >>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> *Mike Tutkowski*
> >>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >>>>> e: mike.tutkowski@solidfire.com
> >>>>> o: 303.746.7302
> >>>>> Advancing the way the world uses the cloud
> >>>>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> *Mike Tutkowski*
> >>> *Senior CloudStack Developer, SolidFire Inc.*
> >>> e: mike.tutkowski@solidfire.com
> >>> o: 303.746.7302
> >>> Advancing the way the world uses the cloud
> >>> <http://solidfire.com/solution/overview/?video=play>*™*
> >>>
> >>
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the cloud
> > <http://solidfire.com/solution/overview/?video=play>*™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Pierre-Luc Dion <pd...@cloudops.com>]

Hi Mike,

The idea of introducing a new API: StorageSnapshot for managed storage is
because the VolumeSnapshot default, or expected, behavior is to archive
snapshots into the Secondary Storage. So a StorageSnapshot API would be for
snapshot that remain on the managed storage appliance.

Quickly looking at the API doc and I don't see a strong requirement for
volume snapshots to be moved into secondary storage. So, maybe
StorageSnapshot API is not useful, but both use cases are required. A
snapshot that remain on the managed storage, and another type of snapshot
that end up into the secondary storage. Since you've done a lot of work,
might easier to just add a parameter to the current snapshot API that would
trigger an extraction of the storage snapshot into the secondary storage?

PL



On Thu, Feb 4, 2016 at 9:02 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> I think that all sounds reasonable then - thanks!
>
> On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
>> You are correct Mike in terms of the requirements. One of our earlier
>> iterations on this was to have an argument to the create snapshot API which
>> decides whether to backup the volume to sec storage but we realized it
>> would make management of snapshots quite messy so we proposed a new api
>> instead.
>>
>> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski <mi...@solidfire.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Just to make sure I understand all the requirements here:
>>>
>>> 1) This relates only to managed storage (1:1 mapping between a virtual
>>> disk and a backend SAN volume).
>>>
>>> 2) We want to take the current (introduced in 4.6) functionality, which
>>> creates a snapshot on the SAN, and extend it via a config option (or
>>> something) to not only take the SAN snapshot, but to copy the underlying
>>> VHD (XenServer only) to NFS.
>>>
>>> 3) The SAN snapshot is always taken. It's the backup to NFS that is
>>> optional.
>>>
>>> 4) Templates can be created from the snapshot that's on the SAN (already
>>> works).
>>>
>>> 5) CloudStack volumes can be created from the snapshot that's on the SAN
>>> (already works as long as the new CloudStack volume ends up on the same
>>> primary storage).
>>>
>>> Would we have a need for a storage snapshot API then or would that just
>>> be the standard volume snapshot without the backup to NFS?
>>>
>>> Thanks!
>>> Mike
>>>
>>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq <sy...@gmail.com>
>>> wrote:
>>>
>>>> Is it possible to have both functionalities (snapshot on SAN & Sec
>>>> Storage) coexist? Because Ideally, we would like to have both.
>>>> For example, some of our customers want to implement their own backup
>>>> strategies and do encryption to their backups which is a perfect
>>>> use case for Storage Snapshot while our other customers will still keep
>>>> using the standard volume snapshot.
>>>>
>>>> To keep things backward compatible, we can add a setting which says to
>>>> not upload on secondary storage, because, after all, you would take a
>>>> SAN snapshot first when doing a Volume Snapshot. You could stop the
>>>> process there and not do the upload.
>>>>
>>>> What do you think about this approach?
>>>>
>>>> Thanks,
>>>> -Syed
>>>>
>>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
>>>> mike.tutkowski@solidfire.com> wrote:
>>>>
>>>>> So, this is just me thinking out load here, but if a given CloudStack
>>>>> cloud doesn't actually need to provide both the ability to take a SAN
>>>>> snapshot and export it to NFS (if just taking a SAN snapshot is OK), then
>>>>> we might be able to get away with no new API calls and simply implement a
>>>>> new custom snapshot strategy and data motion strategy to handle the case
>>>>> where the CloudStack cloud does want both a SAN snapshot and
>>>>> exported-to-NFS backup.
>>>>>
>>>>> In other words, the "default" behavior would be to use the snapshot
>>>>> strategy and data motion strategy that we already have (the one that only
>>>>> takes a SAN snapshot).
>>>>>
>>>>> If your CloudStack cloud, however, wants to take a SAN snapshot and
>>>>> have the data exported to NFS, then we could have you manipulate a Swing
>>>>> config file to make use of a new snapshot strategy and data motion strategy
>>>>> that performs both of these activities.
>>>>>
>>>>> This way, the old behavior is still the default for users, but
>>>>> CloudStack admins can change this behavior via configuration.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>
>>>>>> Right...I think we will need to come up with a viable upgrade path or
>>>>>> some reasonable way for them to move from the old way to the new way (and
>>>>>> some obvious way that they will know they need to do this).
>>>>>>
>>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <
>>>>>> syed1.mushtaq@gmail.com> wrote:
>>>>>>
>>>>>>> I'm not really sure about the upgrade path however, customers who
>>>>>>> are using 4.6 and are on a managed storage would no longer have the same
>>>>>>> functionality with Volume Snapshots.
>>>>>>>
>>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <
>>>>>>> syed1.mushtaq@gmail.com> wrote:
>>>>>>>
>>>>>>>> So if I understand correctly, currently taking a Volume Snapshots
>>>>>>>> of a volume on a managed storage keeps it on the storage array. As a part
>>>>>>>> of this feature, we can make sure that Volume Snapshots on managed storage
>>>>>>>> are uploaded to the secondary storage. This would make the Volume Snapshot
>>>>>>>> feature behave the same regardless of the storage (managed or non-managed)
>>>>>>>> And, for utilizing the efficient backend storage capabilities, we can use
>>>>>>>> the new storage snapshots API.
>>>>>>>>
>>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
>>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> Whatever we do here, we need to have a plan to deal with the fact
>>>>>>>>> that we already have a feature (in 4.6 and later) that allows you to use
>>>>>>>>> the existing volume-snapshot APIs to create a volume snapshot (for managed
>>>>>>>>> storage) that resides on a backend SAN (using a custom snapshot strategy
>>>>>>>>> and a custom data motion strategy).
>>>>>>>>>
>>>>>>>>> If these new APIs go in, then how should the original
>>>>>>>>> implementation (present in 4.6 and later) be changed? If it is changed, how
>>>>>>>>> do we support customers who are already using the original volume-snapshot
>>>>>>>>> API to take snapshots on a backend SAN?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
>>>>>>>>> wstevens@cloudops.com> wrote:
>>>>>>>>>
>>>>>>>>>> Will you be able to create a Template from a StorageSnapshot?  If
>>>>>>>>>> yes, will the template be stored in the secondary storage like normal
>>>>>>>>>> templates or will that be handled somehow on the vendor side?
>>>>>>>>>>
>>>>>>>>>> *Will STEVENS*
>>>>>>>>>> Lead Developer
>>>>>>>>>>
>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
>>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Will!!!
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
>>>>>>>>>>> wstevens@cloudops.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I explicitly linked the Design Spec in the Jira ticket because
>>>>>>>>>>>> it was not clear in the 'mention' section because it shows as a page 'you
>>>>>>>>>>>> do not have permission to'.
>>>>>>>>>>>>
>>>>>>>>>>>> *Will STEVENS*
>>>>>>>>>>>> Lead Developer
>>>>>>>>>>>>
>>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sahmed@cloudops.com
>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Design Spec:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jira Ticket
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots on
>>>>>>>>>>>>> managed storage backends like SolidFire. Snapshots on current managed
>>>>>>>>>>>>> storage stay on the
>>>>>>>>>>>>> device which is contrary to what CloudStack calls snpshots.
>>>>>>>>>>>>> But taking snapshots
>>>>>>>>>>>>> on storage and keeping it there has its own advantages and we
>>>>>>>>>>>>> would ideally like
>>>>>>>>>>>>> to have both ways of doing snapshots. This proposal adds 4 new
>>>>>>>>>>>>> APIs to
>>>>>>>>>>>>> create snapshots on backend storage.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What do you guys think of this feature? I would love to have
>>>>>>>>>>>>> some feedback. I am working on making the design spec more concrete but
>>>>>>>>>>>>> wanted to have
>>>>>>>>>>>>> a high level feedback first before starting to work on it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> -Syed
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Mike Tutkowski*
>>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>>> o: 303.746.7302
>>>>>>>>> Advancing the way the world uses the cloud
>>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Mike Tutkowski*
>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>> e: mike.tutkowski@solidfire.com
>>>>>> o: 303.746.7302
>>>>>> Advancing the way the world uses the cloud
>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Mike Tutkowski*
>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>> e: mike.tutkowski@solidfire.com
>>>>> o: 303.746.7302
>>>>> Advancing the way the world uses the cloud
>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkowski@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the cloud
>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>
>>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

I think that all sounds reasonable then - thanks!

On Thu, Feb 4, 2016 at 6:52 PM, Syed Mushtaq <sy...@gmail.com>
wrote:

> You are correct Mike in terms of the requirements. One of our earlier
> iterations on this was to have an argument to the create snapshot API which
> decides whether to backup the volume to sec storage but we realized it
> would make management of snapshots quite messy so we proposed a new api
> instead.
>
> On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski <mi...@solidfire.com>
> wrote:
>
>> Hi,
>>
>> Just to make sure I understand all the requirements here:
>>
>> 1) This relates only to managed storage (1:1 mapping between a virtual
>> disk and a backend SAN volume).
>>
>> 2) We want to take the current (introduced in 4.6) functionality, which
>> creates a snapshot on the SAN, and extend it via a config option (or
>> something) to not only take the SAN snapshot, but to copy the underlying
>> VHD (XenServer only) to NFS.
>>
>> 3) The SAN snapshot is always taken. It's the backup to NFS that is
>> optional.
>>
>> 4) Templates can be created from the snapshot that's on the SAN (already
>> works).
>>
>> 5) CloudStack volumes can be created from the snapshot that's on the SAN
>> (already works as long as the new CloudStack volume ends up on the same
>> primary storage).
>>
>> Would we have a need for a storage snapshot API then or would that just
>> be the standard volume snapshot without the backup to NFS?
>>
>> Thanks!
>> Mike
>>
>> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq <sy...@gmail.com>
>> wrote:
>>
>>> Is it possible to have both functionalities (snapshot on SAN & Sec
>>> Storage) coexist? Because Ideally, we would like to have both.
>>> For example, some of our customers want to implement their own backup
>>> strategies and do encryption to their backups which is a perfect
>>> use case for Storage Snapshot while our other customers will still keep
>>> using the standard volume snapshot.
>>>
>>> To keep things backward compatible, we can add a setting which says to
>>> not upload on secondary storage, because, after all, you would take a
>>> SAN snapshot first when doing a Volume Snapshot. You could stop the
>>> process there and not do the upload.
>>>
>>> What do you think about this approach?
>>>
>>> Thanks,
>>> -Syed
>>>
>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
>>> mike.tutkowski@solidfire.com> wrote:
>>>
>>>> So, this is just me thinking out load here, but if a given CloudStack
>>>> cloud doesn't actually need to provide both the ability to take a SAN
>>>> snapshot and export it to NFS (if just taking a SAN snapshot is OK), then
>>>> we might be able to get away with no new API calls and simply implement a
>>>> new custom snapshot strategy and data motion strategy to handle the case
>>>> where the CloudStack cloud does want both a SAN snapshot and
>>>> exported-to-NFS backup.
>>>>
>>>> In other words, the "default" behavior would be to use the snapshot
>>>> strategy and data motion strategy that we already have (the one that only
>>>> takes a SAN snapshot).
>>>>
>>>> If your CloudStack cloud, however, wants to take a SAN snapshot and
>>>> have the data exported to NFS, then we could have you manipulate a Swing
>>>> config file to make use of a new snapshot strategy and data motion strategy
>>>> that performs both of these activities.
>>>>
>>>> This way, the old behavior is still the default for users, but
>>>> CloudStack admins can change this behavior via configuration.
>>>>
>>>> Thoughts?
>>>>
>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
>>>> mike.tutkowski@solidfire.com> wrote:
>>>>
>>>>> Right...I think we will need to come up with a viable upgrade path or
>>>>> some reasonable way for them to move from the old way to the new way (and
>>>>> some obvious way that they will know they need to do this).
>>>>>
>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <syed1.mushtaq@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> I'm not really sure about the upgrade path however, customers who are
>>>>>> using 4.6 and are on a managed storage would no longer have the same
>>>>>> functionality with Volume Snapshots.
>>>>>>
>>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <syed1.mushtaq@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> So if I understand correctly, currently taking a Volume Snapshots of
>>>>>>> a volume on a managed storage keeps it on the storage array. As a part of
>>>>>>> this feature, we can make sure that Volume Snapshots on managed storage are
>>>>>>> uploaded to the secondary storage. This would make the Volume Snapshot
>>>>>>> feature behave the same regardless of the storage (managed or non-managed)
>>>>>>> And, for utilizing the efficient backend storage capabilities, we can use
>>>>>>> the new storage snapshots API.
>>>>>>>
>>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
>>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> Whatever we do here, we need to have a plan to deal with the fact
>>>>>>>> that we already have a feature (in 4.6 and later) that allows you to use
>>>>>>>> the existing volume-snapshot APIs to create a volume snapshot (for managed
>>>>>>>> storage) that resides on a backend SAN (using a custom snapshot strategy
>>>>>>>> and a custom data motion strategy).
>>>>>>>>
>>>>>>>> If these new APIs go in, then how should the original
>>>>>>>> implementation (present in 4.6 and later) be changed? If it is changed, how
>>>>>>>> do we support customers who are already using the original volume-snapshot
>>>>>>>> API to take snapshots on a backend SAN?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <
>>>>>>>> wstevens@cloudops.com> wrote:
>>>>>>>>
>>>>>>>>> Will you be able to create a Template from a StorageSnapshot?  If
>>>>>>>>> yes, will the template be stored in the secondary storage like normal
>>>>>>>>> templates or will that be handled somehow on the vendor side?
>>>>>>>>>
>>>>>>>>> *Will STEVENS*
>>>>>>>>> Lead Developer
>>>>>>>>>
>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>>>
>>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
>>>>>>>>> syed1.mushtaq@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Will!!!
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
>>>>>>>>>> wstevens@cloudops.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I explicitly linked the Design Spec in the Jira ticket because
>>>>>>>>>>> it was not clear in the 'mention' section because it shows as a page 'you
>>>>>>>>>>> do not have permission to'.
>>>>>>>>>>>
>>>>>>>>>>> *Will STEVENS*
>>>>>>>>>>> Lead Developer
>>>>>>>>>>>
>>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Design Spec:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>>>>>>>>
>>>>>>>>>>>> Jira Ticket
>>>>>>>>>>>>
>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots on managed
>>>>>>>>>>>> storage backends like SolidFire. Snapshots on current managed storage stay
>>>>>>>>>>>> on the
>>>>>>>>>>>> device which is contrary to what CloudStack calls snpshots. But
>>>>>>>>>>>> taking snapshots
>>>>>>>>>>>> on storage and keeping it there has its own advantages and we
>>>>>>>>>>>> would ideally like
>>>>>>>>>>>> to have both ways of doing snapshots. This proposal adds 4 new
>>>>>>>>>>>> APIs to
>>>>>>>>>>>> create snapshots on backend storage.
>>>>>>>>>>>>
>>>>>>>>>>>> What do you guys think of this feature? I would love to have
>>>>>>>>>>>> some feedback. I am working on making the design spec more concrete but
>>>>>>>>>>>> wanted to have
>>>>>>>>>>>> a high level feedback first before starting to work on it.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> -Syed
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Mike Tutkowski*
>>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>>> o: 303.746.7302
>>>>>>>> Advancing the way the world uses the cloud
>>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Mike Tutkowski*
>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>> e: mike.tutkowski@solidfire.com
>>>>> o: 303.746.7302
>>>>> Advancing the way the world uses the cloud
>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Mike Tutkowski*
>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>> e: mike.tutkowski@solidfire.com
>>>> o: 303.746.7302
>>>> Advancing the way the world uses the cloud
>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>
>>>
>>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the cloud
>> <http://solidfire.com/solution/overview/?video=play>*™*
>>
>


-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

You are correct Mike in terms of the requirements. One of our earlier
iterations on this was to have an argument to the create snapshot API which
decides whether to backup the volume to sec storage but we realized it
would make management of snapshots quite messy so we proposed a new api
instead.

On Thu, Feb 4, 2016, 8:29 PM Mike Tutkowski <mi...@solidfire.com>
wrote:

> Hi,
>
> Just to make sure I understand all the requirements here:
>
> 1) This relates only to managed storage (1:1 mapping between a virtual
> disk and a backend SAN volume).
>
> 2) We want to take the current (introduced in 4.6) functionality, which
> creates a snapshot on the SAN, and extend it via a config option (or
> something) to not only take the SAN snapshot, but to copy the underlying
> VHD (XenServer only) to NFS.
>
> 3) The SAN snapshot is always taken. It's the backup to NFS that is
> optional.
>
> 4) Templates can be created from the snapshot that's on the SAN (already
> works).
>
> 5) CloudStack volumes can be created from the snapshot that's on the SAN
> (already works as long as the new CloudStack volume ends up on the same
> primary storage).
>
> Would we have a need for a storage snapshot API then or would that just be
> the standard volume snapshot without the backup to NFS?
>
> Thanks!
> Mike
>
> On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
>> Is it possible to have both functionalities (snapshot on SAN & Sec
>> Storage) coexist? Because Ideally, we would like to have both.
>> For example, some of our customers want to implement their own backup
>> strategies and do encryption to their backups which is a perfect
>> use case for Storage Snapshot while our other customers will still keep
>> using the standard volume snapshot.
>>
>> To keep things backward compatible, we can add a setting which says to
>> not upload on secondary storage, because, after all, you would take a
>> SAN snapshot first when doing a Volume Snapshot. You could stop the
>> process there and not do the upload.
>>
>> What do you think about this approach?
>>
>> Thanks,
>> -Syed
>>
>> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
>> mike.tutkowski@solidfire.com> wrote:
>>
>>> So, this is just me thinking out load here, but if a given CloudStack
>>> cloud doesn't actually need to provide both the ability to take a SAN
>>> snapshot and export it to NFS (if just taking a SAN snapshot is OK), then
>>> we might be able to get away with no new API calls and simply implement a
>>> new custom snapshot strategy and data motion strategy to handle the case
>>> where the CloudStack cloud does want both a SAN snapshot and
>>> exported-to-NFS backup.
>>>
>>> In other words, the "default" behavior would be to use the snapshot
>>> strategy and data motion strategy that we already have (the one that only
>>> takes a SAN snapshot).
>>>
>>> If your CloudStack cloud, however, wants to take a SAN snapshot and have
>>> the data exported to NFS, then we could have you manipulate a Swing config
>>> file to make use of a new snapshot strategy and data motion strategy that
>>> performs both of these activities.
>>>
>>> This way, the old behavior is still the default for users, but
>>> CloudStack admins can change this behavior via configuration.
>>>
>>> Thoughts?
>>>
>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
>>> mike.tutkowski@solidfire.com> wrote:
>>>
>>>> Right...I think we will need to come up with a viable upgrade path or
>>>> some reasonable way for them to move from the old way to the new way (and
>>>> some obvious way that they will know they need to do this).
>>>>
>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <sy...@gmail.com>
>>>> wrote:
>>>>
>>>>> I'm not really sure about the upgrade path however, customers who are
>>>>> using 4.6 and are on a managed storage would no longer have the same
>>>>> functionality with Volume Snapshots.
>>>>>
>>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <sy...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> So if I understand correctly, currently taking a Volume Snapshots of
>>>>>> a volume on a managed storage keeps it on the storage array. As a part of
>>>>>> this feature, we can make sure that Volume Snapshots on managed storage are
>>>>>> uploaded to the secondary storage. This would make the Volume Snapshot
>>>>>> feature behave the same regardless of the storage (managed or non-managed)
>>>>>> And, for utilizing the efficient backend storage capabilities, we can use
>>>>>> the new storage snapshots API.
>>>>>>
>>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
>>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Whatever we do here, we need to have a plan to deal with the fact
>>>>>>> that we already have a feature (in 4.6 and later) that allows you to use
>>>>>>> the existing volume-snapshot APIs to create a volume snapshot (for managed
>>>>>>> storage) that resides on a backend SAN (using a custom snapshot strategy
>>>>>>> and a custom data motion strategy).
>>>>>>>
>>>>>>> If these new APIs go in, then how should the original implementation
>>>>>>> (present in 4.6 and later) be changed? If it is changed, how do we support
>>>>>>> customers who are already using the original volume-snapshot API to take
>>>>>>> snapshots on a backend SAN?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <wstevens@cloudops.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Will you be able to create a Template from a StorageSnapshot?  If
>>>>>>>> yes, will the template be stored in the secondary storage like normal
>>>>>>>> templates or will that be handled somehow on the vendor side?
>>>>>>>>
>>>>>>>> *Will STEVENS*
>>>>>>>> Lead Developer
>>>>>>>>
>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>>
>>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
>>>>>>>> syed1.mushtaq@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks Will!!!
>>>>>>>>>
>>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <
>>>>>>>>> wstevens@cloudops.com> wrote:
>>>>>>>>>
>>>>>>>>>> I explicitly linked the Design Spec in the Jira ticket because it
>>>>>>>>>> was not clear in the 'mention' section because it shows as a page 'you do
>>>>>>>>>> not have permission to'.
>>>>>>>>>>
>>>>>>>>>> *Will STEVENS*
>>>>>>>>>> Lead Developer
>>>>>>>>>>
>>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Design Spec:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>>>>>>>
>>>>>>>>>>> Jira Ticket
>>>>>>>>>>>
>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> We plan to propose a new set of APIs to do snapshots on managed
>>>>>>>>>>> storage backends like SolidFire. Snapshots on current managed storage stay
>>>>>>>>>>> on the
>>>>>>>>>>> device which is contrary to what CloudStack calls snpshots. But
>>>>>>>>>>> taking snapshots
>>>>>>>>>>> on storage and keeping it there has its own advantages and we
>>>>>>>>>>> would ideally like
>>>>>>>>>>> to have both ways of doing snapshots. This proposal adds 4 new
>>>>>>>>>>> APIs to
>>>>>>>>>>> create snapshots on backend storage.
>>>>>>>>>>>
>>>>>>>>>>> What do you guys think of this feature? I would love to have
>>>>>>>>>>> some feedback. I am working on making the design spec more concrete but
>>>>>>>>>>> wanted to have
>>>>>>>>>>> a high level feedback first before starting to work on it.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> -Syed
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Mike Tutkowski*
>>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>>> e: mike.tutkowski@solidfire.com
>>>>>>> o: 303.746.7302
>>>>>>> Advancing the way the world uses the cloud
>>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Mike Tutkowski*
>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>> e: mike.tutkowski@solidfire.com
>>>> o: 303.746.7302
>>>> Advancing the way the world uses the cloud
>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>
>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkowski@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the cloud
>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>
>>
>>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Hi,

Just to make sure I understand all the requirements here:

1) This relates only to managed storage (1:1 mapping between a virtual disk
and a backend SAN volume).

2) We want to take the current (introduced in 4.6) functionality, which
creates a snapshot on the SAN, and extend it via a config option (or
something) to not only take the SAN snapshot, but to copy the underlying
VHD (XenServer only) to NFS.

3) The SAN snapshot is always taken. It's the backup to NFS that is
optional.

4) Templates can be created from the snapshot that's on the SAN (already
works).

5) CloudStack volumes can be created from the snapshot that's on the SAN
(already works as long as the new CloudStack volume ends up on the same
primary storage).

Would we have a need for a storage snapshot API then or would that just be
the standard volume snapshot without the backup to NFS?

Thanks!
Mike

On Thu, Feb 4, 2016 at 5:24 PM, Syed Mushtaq <sy...@gmail.com>
wrote:

> Is it possible to have both functionalities (snapshot on SAN & Sec
> Storage) coexist? Because Ideally, we would like to have both.
> For example, some of our customers want to implement their own backup
> strategies and do encryption to their backups which is a perfect
> use case for Storage Snapshot while our other customers will still keep
> using the standard volume snapshot.
>
> To keep things backward compatible, we can add a setting which says to not
> upload on secondary storage, because, after all, you would take a
> SAN snapshot first when doing a Volume Snapshot. You could stop the
> process there and not do the upload.
>
> What do you think about this approach?
>
> Thanks,
> -Syed
>
> On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
>> So, this is just me thinking out load here, but if a given CloudStack
>> cloud doesn't actually need to provide both the ability to take a SAN
>> snapshot and export it to NFS (if just taking a SAN snapshot is OK), then
>> we might be able to get away with no new API calls and simply implement a
>> new custom snapshot strategy and data motion strategy to handle the case
>> where the CloudStack cloud does want both a SAN snapshot and
>> exported-to-NFS backup.
>>
>> In other words, the "default" behavior would be to use the snapshot
>> strategy and data motion strategy that we already have (the one that only
>> takes a SAN snapshot).
>>
>> If your CloudStack cloud, however, wants to take a SAN snapshot and have
>> the data exported to NFS, then we could have you manipulate a Swing config
>> file to make use of a new snapshot strategy and data motion strategy that
>> performs both of these activities.
>>
>> This way, the old behavior is still the default for users, but CloudStack
>> admins can change this behavior via configuration.
>>
>> Thoughts?
>>
>> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
>> mike.tutkowski@solidfire.com> wrote:
>>
>>> Right...I think we will need to come up with a viable upgrade path or
>>> some reasonable way for them to move from the old way to the new way (and
>>> some obvious way that they will know they need to do this).
>>>
>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <sy...@gmail.com>
>>> wrote:
>>>
>>>> I'm not really sure about the upgrade path however, customers who are
>>>> using 4.6 and are on a managed storage would no longer have the same
>>>> functionality with Volume Snapshots.
>>>>
>>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <sy...@gmail.com>
>>>> wrote:
>>>>
>>>>> So if I understand correctly, currently taking a Volume Snapshots of a
>>>>> volume on a managed storage keeps it on the storage array. As a part of
>>>>> this feature, we can make sure that Volume Snapshots on managed storage are
>>>>> uploaded to the secondary storage. This would make the Volume Snapshot
>>>>> feature behave the same regardless of the storage (managed or non-managed)
>>>>> And, for utilizing the efficient backend storage capabilities, we can use
>>>>> the new storage snapshots API.
>>>>>
>>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
>>>>> mike.tutkowski@solidfire.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Whatever we do here, we need to have a plan to deal with the fact
>>>>>> that we already have a feature (in 4.6 and later) that allows you to use
>>>>>> the existing volume-snapshot APIs to create a volume snapshot (for managed
>>>>>> storage) that resides on a backend SAN (using a custom snapshot strategy
>>>>>> and a custom data motion strategy).
>>>>>>
>>>>>> If these new APIs go in, then how should the original implementation
>>>>>> (present in 4.6 and later) be changed? If it is changed, how do we support
>>>>>> customers who are already using the original volume-snapshot API to take
>>>>>> snapshots on a backend SAN?
>>>>>>
>>>>>> Thanks,
>>>>>> Mike
>>>>>>
>>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <ws...@cloudops.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Will you be able to create a Template from a StorageSnapshot?  If
>>>>>>> yes, will the template be stored in the secondary storage like normal
>>>>>>> templates or will that be handled somehow on the vendor side?
>>>>>>>
>>>>>>> *Will STEVENS*
>>>>>>> Lead Developer
>>>>>>>
>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>
>>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <
>>>>>>> syed1.mushtaq@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks Will!!!
>>>>>>>>
>>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <wstevens@cloudops.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> I explicitly linked the Design Spec in the Jira ticket because it
>>>>>>>>> was not clear in the 'mention' section because it shows as a page 'you do
>>>>>>>>> not have permission to'.
>>>>>>>>>
>>>>>>>>> *Will STEVENS*
>>>>>>>>> Lead Developer
>>>>>>>>>
>>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>>>
>>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Design Spec:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>>>>>>
>>>>>>>>>> Jira Ticket
>>>>>>>>>>
>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> We plan to propose a new set of APIs to do snapshots on managed
>>>>>>>>>> storage backends like SolidFire. Snapshots on current managed storage stay
>>>>>>>>>> on the
>>>>>>>>>> device which is contrary to what CloudStack calls snpshots. But
>>>>>>>>>> taking snapshots
>>>>>>>>>> on storage and keeping it there has its own advantages and we
>>>>>>>>>> would ideally like
>>>>>>>>>> to have both ways of doing snapshots. This proposal adds 4 new
>>>>>>>>>> APIs to
>>>>>>>>>> create snapshots on backend storage.
>>>>>>>>>>
>>>>>>>>>> What do you guys think of this feature? I would love to have some
>>>>>>>>>> feedback. I am working on making the design spec more concrete but wanted
>>>>>>>>>> to have
>>>>>>>>>> a high level feedback first before starting to work on it.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> -Syed
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Mike Tutkowski*
>>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>>> e: mike.tutkowski@solidfire.com
>>>>>> o: 303.746.7302
>>>>>> Advancing the way the world uses the cloud
>>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkowski@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the cloud
>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>
>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the cloud
>> <http://solidfire.com/solution/overview/?video=play>*™*
>>
>
>


-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

Is it possible to have both functionalities (snapshot on SAN & Sec Storage)
coexist? Because Ideally, we would like to have both.
For example, some of our customers want to implement their own backup
strategies and do encryption to their backups which is a perfect
use case for Storage Snapshot while our other customers will still keep
using the standard volume snapshot.

To keep things backward compatible, we can add a setting which says to not
upload on secondary storage, because, after all, you would take a
SAN snapshot first when doing a Volume Snapshot. You could stop the process
there and not do the upload.

What do you think about this approach?

Thanks,
-Syed

On Thu, Feb 4, 2016 at 4:25 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> So, this is just me thinking out load here, but if a given CloudStack
> cloud doesn't actually need to provide both the ability to take a SAN
> snapshot and export it to NFS (if just taking a SAN snapshot is OK), then
> we might be able to get away with no new API calls and simply implement a
> new custom snapshot strategy and data motion strategy to handle the case
> where the CloudStack cloud does want both a SAN snapshot and
> exported-to-NFS backup.
>
> In other words, the "default" behavior would be to use the snapshot
> strategy and data motion strategy that we already have (the one that only
> takes a SAN snapshot).
>
> If your CloudStack cloud, however, wants to take a SAN snapshot and have
> the data exported to NFS, then we could have you manipulate a Swing config
> file to make use of a new snapshot strategy and data motion strategy that
> performs both of these activities.
>
> This way, the old behavior is still the default for users, but CloudStack
> admins can change this behavior via configuration.
>
> Thoughts?
>
> On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
>> Right...I think we will need to come up with a viable upgrade path or
>> some reasonable way for them to move from the old way to the new way (and
>> some obvious way that they will know they need to do this).
>>
>> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <sy...@gmail.com>
>> wrote:
>>
>>> I'm not really sure about the upgrade path however, customers who are
>>> using 4.6 and are on a managed storage would no longer have the same
>>> functionality with Volume Snapshots.
>>>
>>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <sy...@gmail.com>
>>> wrote:
>>>
>>>> So if I understand correctly, currently taking a Volume Snapshots of a
>>>> volume on a managed storage keeps it on the storage array. As a part of
>>>> this feature, we can make sure that Volume Snapshots on managed storage are
>>>> uploaded to the secondary storage. This would make the Volume Snapshot
>>>> feature behave the same regardless of the storage (managed or non-managed)
>>>> And, for utilizing the efficient backend storage capabilities, we can use
>>>> the new storage snapshots API.
>>>>
>>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
>>>> mike.tutkowski@solidfire.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Whatever we do here, we need to have a plan to deal with the fact that
>>>>> we already have a feature (in 4.6 and later) that allows you to use the
>>>>> existing volume-snapshot APIs to create a volume snapshot (for managed
>>>>> storage) that resides on a backend SAN (using a custom snapshot strategy
>>>>> and a custom data motion strategy).
>>>>>
>>>>> If these new APIs go in, then how should the original implementation
>>>>> (present in 4.6 and later) be changed? If it is changed, how do we support
>>>>> customers who are already using the original volume-snapshot API to take
>>>>> snapshots on a backend SAN?
>>>>>
>>>>> Thanks,
>>>>> Mike
>>>>>
>>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <ws...@cloudops.com>
>>>>> wrote:
>>>>>
>>>>>> Will you be able to create a Template from a StorageSnapshot?  If
>>>>>> yes, will the template be stored in the secondary storage like normal
>>>>>> templates or will that be handled somehow on the vendor side?
>>>>>>
>>>>>> *Will STEVENS*
>>>>>> Lead Developer
>>>>>>
>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>
>>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <syed1.mushtaq@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Thanks Will!!!
>>>>>>>
>>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I explicitly linked the Design Spec in the Jira ticket because it
>>>>>>>> was not clear in the 'mention' section because it shows as a page 'you do
>>>>>>>> not have permission to'.
>>>>>>>>
>>>>>>>> *Will STEVENS*
>>>>>>>> Lead Developer
>>>>>>>>
>>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>>
>>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Design Spec:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>>>>>
>>>>>>>>> Jira Ticket
>>>>>>>>>
>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> We plan to propose a new set of APIs to do snapshots on managed
>>>>>>>>> storage backends like SolidFire. Snapshots on current managed storage stay
>>>>>>>>> on the
>>>>>>>>> device which is contrary to what CloudStack calls snpshots. But
>>>>>>>>> taking snapshots
>>>>>>>>> on storage and keeping it there has its own advantages and we
>>>>>>>>> would ideally like
>>>>>>>>> to have both ways of doing snapshots. This proposal adds 4 new
>>>>>>>>> APIs to
>>>>>>>>> create snapshots on backend storage.
>>>>>>>>>
>>>>>>>>> What do you guys think of this feature? I would love to have some
>>>>>>>>> feedback. I am working on making the design spec more concrete but wanted
>>>>>>>>> to have
>>>>>>>>> a high level feedback first before starting to work on it.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> -Syed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Mike Tutkowski*
>>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>>> e: mike.tutkowski@solidfire.com
>>>>> o: 303.746.7302
>>>>> Advancing the way the world uses the cloud
>>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the cloud
>> <http://solidfire.com/solution/overview/?video=play>*™*
>>
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

So, this is just me thinking out load here, but if a given CloudStack cloud
doesn't actually need to provide both the ability to take a SAN snapshot
and export it to NFS (if just taking a SAN snapshot is OK), then we might
be able to get away with no new API calls and simply implement a new custom
snapshot strategy and data motion strategy to handle the case where the
CloudStack cloud does want both a SAN snapshot and exported-to-NFS backup.

In other words, the "default" behavior would be to use the snapshot
strategy and data motion strategy that we already have (the one that only
takes a SAN snapshot).

If your CloudStack cloud, however, wants to take a SAN snapshot and have
the data exported to NFS, then we could have you manipulate a Swing config
file to make use of a new snapshot strategy and data motion strategy that
performs both of these activities.

This way, the old behavior is still the default for users, but CloudStack
admins can change this behavior via configuration.

Thoughts?

On Thu, Feb 4, 2016 at 11:55 AM, Mike Tutkowski <
mike.tutkowski@solidfire.com> wrote:

> Right...I think we will need to come up with a viable upgrade path or some
> reasonable way for them to move from the old way to the new way (and some
> obvious way that they will know they need to do this).
>
> On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
>> I'm not really sure about the upgrade path however, customers who are
>> using 4.6 and are on a managed storage would no longer have the same
>> functionality with Volume Snapshots.
>>
>> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <sy...@gmail.com>
>> wrote:
>>
>>> So if I understand correctly, currently taking a Volume Snapshots of a
>>> volume on a managed storage keeps it on the storage array. As a part of
>>> this feature, we can make sure that Volume Snapshots on managed storage are
>>> uploaded to the secondary storage. This would make the Volume Snapshot
>>> feature behave the same regardless of the storage (managed or non-managed)
>>> And, for utilizing the efficient backend storage capabilities, we can use
>>> the new storage snapshots API.
>>>
>>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
>>> mike.tutkowski@solidfire.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Whatever we do here, we need to have a plan to deal with the fact that
>>>> we already have a feature (in 4.6 and later) that allows you to use the
>>>> existing volume-snapshot APIs to create a volume snapshot (for managed
>>>> storage) that resides on a backend SAN (using a custom snapshot strategy
>>>> and a custom data motion strategy).
>>>>
>>>> If these new APIs go in, then how should the original implementation
>>>> (present in 4.6 and later) be changed? If it is changed, how do we support
>>>> customers who are already using the original volume-snapshot API to take
>>>> snapshots on a backend SAN?
>>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <ws...@cloudops.com>
>>>> wrote:
>>>>
>>>>> Will you be able to create a Template from a StorageSnapshot?  If yes,
>>>>> will the template be stored in the secondary storage like normal templates
>>>>> or will that be handled somehow on the vendor side?
>>>>>
>>>>> *Will STEVENS*
>>>>> Lead Developer
>>>>>
>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>
>>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <sy...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Will!!!
>>>>>>
>>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I explicitly linked the Design Spec in the Jira ticket because it
>>>>>>> was not clear in the 'mention' section because it shows as a page 'you do
>>>>>>> not have permission to'.
>>>>>>>
>>>>>>> *Will STEVENS*
>>>>>>> Lead Developer
>>>>>>>
>>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>>
>>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Design Spec:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>>>>
>>>>>>>> Jira Ticket
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> We plan to propose a new set of APIs to do snapshots on managed
>>>>>>>> storage backends like SolidFire. Snapshots on current managed storage stay
>>>>>>>> on the
>>>>>>>> device which is contrary to what CloudStack calls snpshots. But
>>>>>>>> taking snapshots
>>>>>>>> on storage and keeping it there has its own advantages and we would
>>>>>>>> ideally like
>>>>>>>> to have both ways of doing snapshots. This proposal adds 4 new APIs
>>>>>>>> to
>>>>>>>> create snapshots on backend storage.
>>>>>>>>
>>>>>>>> What do you guys think of this feature? I would love to have some
>>>>>>>> feedback. I am working on making the design spec more concrete but wanted
>>>>>>>> to have
>>>>>>>> a high level feedback first before starting to work on it.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -Syed
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Mike Tutkowski*
>>>> *Senior CloudStack Developer, SolidFire Inc.*
>>>> e: mike.tutkowski@solidfire.com
>>>> o: 303.746.7302
>>>> Advancing the way the world uses the cloud
>>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>>
>>>
>>>
>>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Right...I think we will need to come up with a viable upgrade path or some
reasonable way for them to move from the old way to the new way (and some
obvious way that they will know they need to do this).

On Thu, Feb 4, 2016 at 11:45 AM, Syed Mushtaq <sy...@gmail.com>
wrote:

> I'm not really sure about the upgrade path however, customers who are
> using 4.6 and are on a managed storage would no longer have the same
> functionality with Volume Snapshots.
>
> On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
>> So if I understand correctly, currently taking a Volume Snapshots of a
>> volume on a managed storage keeps it on the storage array. As a part of
>> this feature, we can make sure that Volume Snapshots on managed storage are
>> uploaded to the secondary storage. This would make the Volume Snapshot
>> feature behave the same regardless of the storage (managed or non-managed)
>> And, for utilizing the efficient backend storage capabilities, we can use
>> the new storage snapshots API.
>>
>> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
>> mike.tutkowski@solidfire.com> wrote:
>>
>>> Hi everyone,
>>>
>>> Whatever we do here, we need to have a plan to deal with the fact that
>>> we already have a feature (in 4.6 and later) that allows you to use the
>>> existing volume-snapshot APIs to create a volume snapshot (for managed
>>> storage) that resides on a backend SAN (using a custom snapshot strategy
>>> and a custom data motion strategy).
>>>
>>> If these new APIs go in, then how should the original implementation
>>> (present in 4.6 and later) be changed? If it is changed, how do we support
>>> customers who are already using the original volume-snapshot API to take
>>> snapshots on a backend SAN?
>>>
>>> Thanks,
>>> Mike
>>>
>>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <ws...@cloudops.com>
>>> wrote:
>>>
>>>> Will you be able to create a Template from a StorageSnapshot?  If yes,
>>>> will the template be stored in the secondary storage like normal templates
>>>> or will that be handled somehow on the vendor side?
>>>>
>>>> *Will STEVENS*
>>>> Lead Developer
>>>>
>>>> *CloudOps* *| *Cloud Solutions Experts
>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>> w cloudops.com *|* tw @CloudOps_
>>>>
>>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <sy...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Will!!!
>>>>>
>>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com>
>>>>> wrote:
>>>>>
>>>>>> I explicitly linked the Design Spec in the Jira ticket because it was
>>>>>> not clear in the 'mention' section because it shows as a page 'you do not
>>>>>> have permission to'.
>>>>>>
>>>>>> *Will STEVENS*
>>>>>> Lead Developer
>>>>>>
>>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>>
>>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Design Spec:
>>>>>>>
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>>>
>>>>>>> Jira Ticket
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>>>
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> We plan to propose a new set of APIs to do snapshots on managed
>>>>>>> storage backends like SolidFire. Snapshots on current managed storage stay
>>>>>>> on the
>>>>>>> device which is contrary to what CloudStack calls snpshots. But
>>>>>>> taking snapshots
>>>>>>> on storage and keeping it there has its own advantages and we would
>>>>>>> ideally like
>>>>>>> to have both ways of doing snapshots. This proposal adds 4 new APIs
>>>>>>> to
>>>>>>> create snapshots on backend storage.
>>>>>>>
>>>>>>> What do you guys think of this feature? I would love to have some
>>>>>>> feedback. I am working on making the design spec more concrete but wanted
>>>>>>> to have
>>>>>>> a high level feedback first before starting to work on it.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -Syed
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Mike Tutkowski*
>>> *Senior CloudStack Developer, SolidFire Inc.*
>>> e: mike.tutkowski@solidfire.com
>>> o: 303.746.7302
>>> Advancing the way the world uses the cloud
>>> <http://solidfire.com/solution/overview/?video=play>*™*
>>>
>>
>>
>


-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

I'm not really sure about the upgrade path however, customers who are using
4.6 and are on a managed storage would no longer have the same
functionality with Volume Snapshots.

On Thu, Feb 4, 2016 at 1:43 PM, Syed Mushtaq <sy...@gmail.com>
wrote:

> So if I understand correctly, currently taking a Volume Snapshots of a
> volume on a managed storage keeps it on the storage array. As a part of
> this feature, we can make sure that Volume Snapshots on managed storage are
> uploaded to the secondary storage. This would make the Volume Snapshot
> feature behave the same regardless of the storage (managed or non-managed)
> And, for utilizing the efficient backend storage capabilities, we can use
> the new storage snapshots API.
>
> On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
>> Hi everyone,
>>
>> Whatever we do here, we need to have a plan to deal with the fact that we
>> already have a feature (in 4.6 and later) that allows you to use the
>> existing volume-snapshot APIs to create a volume snapshot (for managed
>> storage) that resides on a backend SAN (using a custom snapshot strategy
>> and a custom data motion strategy).
>>
>> If these new APIs go in, then how should the original implementation
>> (present in 4.6 and later) be changed? If it is changed, how do we support
>> customers who are already using the original volume-snapshot API to take
>> snapshots on a backend SAN?
>>
>> Thanks,
>> Mike
>>
>> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <ws...@cloudops.com>
>> wrote:
>>
>>> Will you be able to create a Template from a StorageSnapshot?  If yes,
>>> will the template be stored in the secondary storage like normal templates
>>> or will that be handled somehow on the vendor side?
>>>
>>> *Will STEVENS*
>>> Lead Developer
>>>
>>> *CloudOps* *| *Cloud Solutions Experts
>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>> w cloudops.com *|* tw @CloudOps_
>>>
>>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <sy...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Will!!!
>>>>
>>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com>
>>>> wrote:
>>>>
>>>>> I explicitly linked the Design Spec in the Jira ticket because it was
>>>>> not clear in the 'mention' section because it shows as a page 'you do not
>>>>> have permission to'.
>>>>>
>>>>> *Will STEVENS*
>>>>> Lead Developer
>>>>>
>>>>> *CloudOps* *| *Cloud Solutions Experts
>>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>>> w cloudops.com *|* tw @CloudOps_
>>>>>
>>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Design Spec:
>>>>>>
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>>
>>>>>> Jira Ticket
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>>
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> We plan to propose a new set of APIs to do snapshots on managed
>>>>>> storage backends like SolidFire. Snapshots on current managed storage stay
>>>>>> on the
>>>>>> device which is contrary to what CloudStack calls snpshots. But
>>>>>> taking snapshots
>>>>>> on storage and keeping it there has its own advantages and we would
>>>>>> ideally like
>>>>>> to have both ways of doing snapshots. This proposal adds 4 new APIs
>>>>>> to
>>>>>> create snapshots on backend storage.
>>>>>>
>>>>>> What do you guys think of this feature? I would love to have some
>>>>>> feedback. I am working on making the design spec more concrete but wanted
>>>>>> to have
>>>>>> a high level feedback first before starting to work on it.
>>>>>>
>>>>>> Thanks,
>>>>>> -Syed
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> *Mike Tutkowski*
>> *Senior CloudStack Developer, SolidFire Inc.*
>> e: mike.tutkowski@solidfire.com
>> o: 303.746.7302
>> Advancing the way the world uses the cloud
>> <http://solidfire.com/solution/overview/?video=play>*™*
>>
>
>

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

So if I understand correctly, currently taking a Volume Snapshots of a
volume on a managed storage keeps it on the storage array. As a part of
this feature, we can make sure that Volume Snapshots on managed storage are
uploaded to the secondary storage. This would make the Volume Snapshot
feature behave the same regardless of the storage (managed or non-managed)
And, for utilizing the efficient backend storage capabilities, we can use
the new storage snapshots API.

On Thu, Feb 4, 2016 at 1:36 PM, Mike Tutkowski <mike.tutkowski@solidfire.com
> wrote:

> Hi everyone,
>
> Whatever we do here, we need to have a plan to deal with the fact that we
> already have a feature (in 4.6 and later) that allows you to use the
> existing volume-snapshot APIs to create a volume snapshot (for managed
> storage) that resides on a backend SAN (using a custom snapshot strategy
> and a custom data motion strategy).
>
> If these new APIs go in, then how should the original implementation
> (present in 4.6 and later) be changed? If it is changed, how do we support
> customers who are already using the original volume-snapshot API to take
> snapshots on a backend SAN?
>
> Thanks,
> Mike
>
> On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>> Will you be able to create a Template from a StorageSnapshot?  If yes,
>> will the template be stored in the secondary storage like normal templates
>> or will that be handled somehow on the vendor side?
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> *CloudOps* *| *Cloud Solutions Experts
>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> w cloudops.com *|* tw @CloudOps_
>>
>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <sy...@gmail.com>
>> wrote:
>>
>>> Thanks Will!!!
>>>
>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com>
>>> wrote:
>>>
>>>> I explicitly linked the Design Spec in the Jira ticket because it was
>>>> not clear in the 'mention' section because it shows as a page 'you do not
>>>> have permission to'.
>>>>
>>>> *Will STEVENS*
>>>> Lead Developer
>>>>
>>>> *CloudOps* *| *Cloud Solutions Experts
>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>> w cloudops.com *|* tw @CloudOps_
>>>>
>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>>>>
>>>>>
>>>>> Design Spec:
>>>>>
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>
>>>>> Jira Ticket
>>>>>
>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>
>>>>>
>>>>> Hi All,
>>>>>
>>>>> We plan to propose a new set of APIs to do snapshots on managed
>>>>> storage backends like SolidFire. Snapshots on current managed storage stay
>>>>> on the
>>>>> device which is contrary to what CloudStack calls snpshots. But taking
>>>>> snapshots
>>>>> on storage and keeping it there has its own advantages and we would
>>>>> ideally like
>>>>> to have both ways of doing snapshots. This proposal adds 4 new APIs to
>>>>> create snapshots on backend storage.
>>>>>
>>>>> What do you guys think of this feature? I would love to have some
>>>>> feedback. I am working on making the design spec more concrete but wanted
>>>>> to have
>>>>> a high level feedback first before starting to work on it.
>>>>>
>>>>> Thanks,
>>>>> -Syed
>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the cloud
> <http://solidfire.com/solution/overview/?video=play>*™*
>

Re: [Propose][New Feature] Storage Snapshots

[Mike Tutkowski <mi...@solidfire.com>]

Hi everyone,

Whatever we do here, we need to have a plan to deal with the fact that we
already have a feature (in 4.6 and later) that allows you to use the
existing volume-snapshot APIs to create a volume snapshot (for managed
storage) that resides on a backend SAN (using a custom snapshot strategy
and a custom data motion strategy).

If these new APIs go in, then how should the original implementation
(present in 4.6 and later) be changed? If it is changed, how do we support
customers who are already using the original volume-snapshot API to take
snapshots on a backend SAN?

Thanks,
Mike

On Thu, Feb 4, 2016 at 11:27 AM, Will Stevens <ws...@cloudops.com> wrote:

> Will you be able to create a Template from a StorageSnapshot?  If yes,
> will the template be stored in the secondary storage like normal templates
> or will that be handled somehow on the vendor side?
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
>> Thanks Will!!!
>>
>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com>
>> wrote:
>>
>>> I explicitly linked the Design Spec in the Jira ticket because it was
>>> not clear in the 'mention' section because it shows as a page 'you do not
>>> have permission to'.
>>>
>>> *Will STEVENS*
>>> Lead Developer
>>>
>>> *CloudOps* *| *Cloud Solutions Experts
>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>> w cloudops.com *|* tw @CloudOps_
>>>
>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>>>
>>>>
>>>> Design Spec:
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>
>>>> Jira Ticket
>>>>
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>
>>>>
>>>> Hi All,
>>>>
>>>> We plan to propose a new set of APIs to do snapshots on managed storage
>>>> backends like SolidFire. Snapshots on current managed storage stay on the
>>>> device which is contrary to what CloudStack calls snpshots. But taking
>>>> snapshots
>>>> on storage and keeping it there has its own advantages and we would
>>>> ideally like
>>>> to have both ways of doing snapshots. This proposal adds 4 new APIs to
>>>> create snapshots on backend storage.
>>>>
>>>> What do you guys think of this feature? I would love to have some
>>>> feedback. I am working on making the design spec more concrete but wanted
>>>> to have
>>>> a high level feedback first before starting to work on it.
>>>>
>>>> Thanks,
>>>> -Syed
>>>>
>>>>
>>>
>>
>


-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*

Re: [Propose][New Feature] Storage Snapshots

[Will Stevens <ws...@cloudops.com>]

Thanks Syed, that is much clearer now.  :)

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Thu, Feb 4, 2016 at 1:36 PM, Syed Mushtaq <sy...@gmail.com>
wrote:

> Right now we don't plan to create templates from this functionality mainly
> because you can achieve the same effect using the standard Volume Snapshots
> API. Since the template would always reside on the secondary storage, it
> doesn't really make sense to use the backed cloning if a copy to secondary
> storage is required anyways.
>
> To give a little more background, we have a Xenserver + SolidFire
> installation and snapshots that happen on Xenserver are very expensive
> partly because they consume dom0 resources and partly because a VDI
> snapshot on Xen can cause a significant drop in your I/O throughput during
> the snapshot process (we've seen about 30-50% drop in the number of IOPS on
> our instances). We've also seen cases where the snapshots don't get deleted
> for a long time on Xenserver and we have to deal with a degraded I/O
> performance for a long time (about an hour in our case). We've opened a bug
> with Xenserver for the same but haven't had much response on it [1]. So we
> really want to use the SolidFire's native snapshot capability and I believe
> it would be beneficial to others as well.
>
> [1]: https://bugs.xenserver.org/browse/XSO-434
>
> Thanks,
> -Syed
>
>
> On Thu, Feb 4, 2016 at 1:27 PM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>> Will you be able to create a Template from a StorageSnapshot?  If yes,
>> will the template be stored in the secondary storage like normal templates
>> or will that be handled somehow on the vendor side?
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> *CloudOps* *| *Cloud Solutions Experts
>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> w cloudops.com *|* tw @CloudOps_
>>
>> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <sy...@gmail.com>
>> wrote:
>>
>>> Thanks Will!!!
>>>
>>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com>
>>> wrote:
>>>
>>>> I explicitly linked the Design Spec in the Jira ticket because it was
>>>> not clear in the 'mention' section because it shows as a page 'you do not
>>>> have permission to'.
>>>>
>>>> *Will STEVENS*
>>>> Lead Developer
>>>>
>>>> *CloudOps* *| *Cloud Solutions Experts
>>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>>> w cloudops.com *|* tw @CloudOps_
>>>>
>>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>>>>
>>>>>
>>>>> Design Spec:
>>>>>
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>>
>>>>> Jira Ticket
>>>>>
>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>>
>>>>>
>>>>> Hi All,
>>>>>
>>>>> We plan to propose a new set of APIs to do snapshots on managed
>>>>> storage backends like SolidFire. Snapshots on current managed storage stay
>>>>> on the
>>>>> device which is contrary to what CloudStack calls snpshots. But taking
>>>>> snapshots
>>>>> on storage and keeping it there has its own advantages and we would
>>>>> ideally like
>>>>> to have both ways of doing snapshots. This proposal adds 4 new APIs to
>>>>> create snapshots on backend storage.
>>>>>
>>>>> What do you guys think of this feature? I would love to have some
>>>>> feedback. I am working on making the design spec more concrete but wanted
>>>>> to have
>>>>> a high level feedback first before starting to work on it.
>>>>>
>>>>> Thanks,
>>>>> -Syed
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

Right now we don't plan to create templates from this functionality mainly
because you can achieve the same effect using the standard Volume Snapshots
API. Since the template would always reside on the secondary storage, it
doesn't really make sense to use the backed cloning if a copy to secondary
storage is required anyways.

To give a little more background, we have a Xenserver + SolidFire
installation and snapshots that happen on Xenserver are very expensive
partly because they consume dom0 resources and partly because a VDI
snapshot on Xen can cause a significant drop in your I/O throughput during
the snapshot process (we've seen about 30-50% drop in the number of IOPS on
our instances). We've also seen cases where the snapshots don't get deleted
for a long time on Xenserver and we have to deal with a degraded I/O
performance for a long time (about an hour in our case). We've opened a bug
with Xenserver for the same but haven't had much response on it [1]. So we
really want to use the SolidFire's native snapshot capability and I believe
it would be beneficial to others as well.

[1]: https://bugs.xenserver.org/browse/XSO-434

Thanks,
-Syed


On Thu, Feb 4, 2016 at 1:27 PM, Will Stevens <ws...@cloudops.com> wrote:

> Will you be able to create a Template from a StorageSnapshot?  If yes,
> will the template be stored in the secondary storage like normal templates
> or will that be handled somehow on the vendor side?
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <sy...@gmail.com>
> wrote:
>
>> Thanks Will!!!
>>
>> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com>
>> wrote:
>>
>>> I explicitly linked the Design Spec in the Jira ticket because it was
>>> not clear in the 'mention' section because it shows as a page 'you do not
>>> have permission to'.
>>>
>>> *Will STEVENS*
>>> Lead Developer
>>>
>>> *CloudOps* *| *Cloud Solutions Experts
>>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>>> w cloudops.com *|* tw @CloudOps_
>>>
>>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>>>
>>>>
>>>> Design Spec:
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>>
>>>> Jira Ticket
>>>>
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>>
>>>>
>>>> Hi All,
>>>>
>>>> We plan to propose a new set of APIs to do snapshots on managed storage
>>>> backends like SolidFire. Snapshots on current managed storage stay on the
>>>> device which is contrary to what CloudStack calls snpshots. But taking
>>>> snapshots
>>>> on storage and keeping it there has its own advantages and we would
>>>> ideally like
>>>> to have both ways of doing snapshots. This proposal adds 4 new APIs to
>>>> create snapshots on backend storage.
>>>>
>>>> What do you guys think of this feature? I would love to have some
>>>> feedback. I am working on making the design spec more concrete but wanted
>>>> to have
>>>> a high level feedback first before starting to work on it.
>>>>
>>>> Thanks,
>>>> -Syed
>>>>
>>>>
>>>
>>
>

Re: [Propose][New Feature] Storage Snapshots

[Will Stevens <ws...@cloudops.com>]

Will you be able to create a Template from a StorageSnapshot?  If yes, will
the template be stored in the secondary storage like normal templates or
will that be handled somehow on the vendor side?

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Thu, Feb 4, 2016 at 1:22 PM, Syed Mushtaq <sy...@gmail.com>
wrote:

> Thanks Will!!!
>
> On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>> I explicitly linked the Design Spec in the Jira ticket because it was not
>> clear in the 'mention' section because it shows as a page 'you do not have
>> permission to'.
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> *CloudOps* *| *Cloud Solutions Experts
>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> w cloudops.com *|* tw @CloudOps_
>>
>> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>>
>>>
>>> Design Spec:
>>>
>>>
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>>
>>> Jira Ticket
>>>
>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>>
>>>
>>> Hi All,
>>>
>>> We plan to propose a new set of APIs to do snapshots on managed storage
>>> backends like SolidFire. Snapshots on current managed storage stay on the
>>> device which is contrary to what CloudStack calls snpshots. But taking
>>> snapshots
>>> on storage and keeping it there has its own advantages and we would
>>> ideally like
>>> to have both ways of doing snapshots. This proposal adds 4 new APIs to
>>> create snapshots on backend storage.
>>>
>>> What do you guys think of this feature? I would love to have some
>>> feedback. I am working on making the design spec more concrete but wanted
>>> to have
>>> a high level feedback first before starting to work on it.
>>>
>>> Thanks,
>>> -Syed
>>>
>>>
>>
>

Re: [Propose][New Feature] Storage Snapshots

[Syed Mushtaq <sy...@gmail.com>]

Thanks Will!!!

On Thu, Feb 4, 2016 at 1:19 PM, Will Stevens <ws...@cloudops.com> wrote:

> I explicitly linked the Design Spec in the Jira ticket because it was not
> clear in the 'mention' section because it shows as a page 'you do not have
> permission to'.
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>
>>
>> Design Spec:
>>
>>
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>>
>> Jira Ticket
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>>
>>
>> Hi All,
>>
>> We plan to propose a new set of APIs to do snapshots on managed storage
>> backends like SolidFire. Snapshots on current managed storage stay on the
>> device which is contrary to what CloudStack calls snpshots. But taking
>> snapshots
>> on storage and keeping it there has its own advantages and we would
>> ideally like
>> to have both ways of doing snapshots. This proposal adds 4 new APIs to
>> create snapshots on backend storage.
>>
>> What do you guys think of this feature? I would love to have some
>> feedback. I am working on making the design spec more concrete but wanted
>> to have
>> a high level feedback first before starting to work on it.
>>
>> Thanks,
>> -Syed
>>
>>
>

Re: [Propose][New Feature] Storage Snapshots

[Will Stevens <ws...@cloudops.com>]

I explicitly linked the Design Spec in the Jira ticket because it was not
clear in the 'mention' section because it shows as a page 'you do not have
permission to'.

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Thu, Feb 4, 2016 at 1:02 PM, Syed Ahmed <sa...@cloudops.com> wrote:

>
> Design Spec:
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/StorageSnapshot++API
>
> Jira Ticket
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-9278
>
>
> Hi All,
>
> We plan to propose a new set of APIs to do snapshots on managed storage
> backends like SolidFire. Snapshots on current managed storage stay on the
> device which is contrary to what CloudStack calls snpshots. But taking
> snapshots
> on storage and keeping it there has its own advantages and we would
> ideally like
> to have both ways of doing snapshots. This proposal adds 4 new APIs to
> create snapshots on backend storage.
>
> What do you guys think of this feature? I would love to have some
> feedback. I am working on making the design spec more concrete but wanted
> to have
> a high level feedback first before starting to work on it.
>
> Thanks,
> -Syed
>
>