You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Min Chen <mi...@citrix.com> on 2013/04/09 01:45:23 UTC
[DIscuss]Storage image store plugin framework refactoring
Hi All,
Currently CloudStack does not offer a flexible pluggable framework for users to easily integrate and configure any 3rd-party object stores for such backup services as registering templates, taking snapshots, etc. Along with Edison's recent refactored storage subsystem 2.0 that mainly refactored current CloudStack primary storage implementation, we are proposing to develop a storage backup object store plugin framework to allow CloudStack to systematically manage and configure various types of backup data stores from different vendors, like NFS, S3, Swift, etc. With this new plugin framework, we would like to achieve following functionalities:
1. Support different object store providers in a uniform and pluggable fashion.
2. Enable region wide object backup using S3-like object store.
3. Provide pluggable data motion strategies to handle data transfer from one data store to another data store.
4. Provide a scalable cache storage framework while moving data between primary storage and backup storage for certain hypervisor needs.
5. Support flexible combinations of primary storage, secondary storage and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware), (ISCSI, Swift, KVM), …., etc.
The proposed ImageStore plugin framework architecture is detailed in our FS here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+Object+Store+Plugin+Framework.
The JIRA ticket to track this feature is: https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is currently carried out in feature branch "object_store".
Please let me know your comments and suggestions.
Thanks
-min
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Min Chen <mi...@citrix.com>.
Yes. You should be able to use it to test S3 object store.
THanks
-min
On 4/22/13 4:53 AM, "Sanjeev Neelarapu" <sa...@citrix.com>
wrote:
>Hi,
>
>I have prepared RIAK CS object store in our lab environment and planning
>to use it for object store feature testing .
>As per my understanding RIAK CS supports S3 APIs. So let me know whether
>I can use it as an alternative to s3 based object store.
>
>Thanks,
>Sanjeev
>
>-----Original Message-----
>From: prasanna [mailto:srivatsav.prasanna@gmail.com] On Behalf Of
>Prasanna Santhanam
>Sent: Wednesday, April 10, 2013 12:42 PM
>To: dev@cloudstack.apache.org
>Subject: Re: [DIscuss]Storage image store plugin framework refactoring
>
>On Tue, Apr 09, 2013 at 09:19:11AM -0700, Min Chen wrote:
>> Not yet at this moment. But we are planning to provide a sample plugin
>> implementation for this effort. If you are looking at object_store
>> branch, you will see a plugin called
>> "cloud-plugin-storage-image-sample", which will be the placeholder for
>>a sample image store implementation.
>>
>
>Hmmm - more frameworks and no reference implementation? :(
>
>I'm keeping an eye on the branch and was hoping there'd be an
>implementation of an objectstore because I'm looking to span this
>solution across some qa labs. RiakCS, Swift are all opensource now.
>
>--
>Prasanna.,
RE: [DIscuss]Storage image store plugin framework refactoring
Posted by Sanjeev Neelarapu <sa...@citrix.com>.
Hi,
I have prepared RIAK CS object store in our lab environment and planning to use it for object store feature testing .
As per my understanding RIAK CS supports S3 APIs. So let me know whether I can use it as an alternative to s3 based object store.
Thanks,
Sanjeev
-----Original Message-----
From: prasanna [mailto:srivatsav.prasanna@gmail.com] On Behalf Of Prasanna Santhanam
Sent: Wednesday, April 10, 2013 12:42 PM
To: dev@cloudstack.apache.org
Subject: Re: [DIscuss]Storage image store plugin framework refactoring
On Tue, Apr 09, 2013 at 09:19:11AM -0700, Min Chen wrote:
> Not yet at this moment. But we are planning to provide a sample plugin
> implementation for this effort. If you are looking at object_store
> branch, you will see a plugin called
> "cloud-plugin-storage-image-sample", which will be the placeholder for a sample image store implementation.
>
Hmmm - more frameworks and no reference implementation? :(
I'm keeping an eye on the branch and was hoping there'd be an implementation of an objectstore because I'm looking to span this solution across some qa labs. RiakCS, Swift are all opensource now.
--
Prasanna.,
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Prasanna Santhanam <ts...@apache.org>.
On Tue, Apr 09, 2013 at 09:19:11AM -0700, Min Chen wrote:
> Not yet at this moment. But we are planning to provide a sample plugin
> implementation for this effort. If you are looking at object_store branch,
> you will see a plugin called "cloud-plugin-storage-image-sample", which
> will be the placeholder for a sample image store implementation.
>
Hmmm - more frameworks and no reference implementation? :(
I'm keeping an eye on the branch and was hoping there'd be an
implementation of an objectstore because I'm looking to span this
solution across some qa labs. RiakCS, Swift are all opensource
now.
--
Prasanna.,
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Min Chen <mi...@citrix.com>.
Not yet at this moment. But we are planning to provide a sample plugin
implementation for this effort. If you are looking at object_store branch,
you will see a plugin called "cloud-plugin-storage-image-sample", which
will be the placeholder for a sample image store implementation.
Thanks
-min
On 4/9/13 1:59 AM, "Prasanna Santhanam" <ts...@apache.org> wrote:
>On Mon, Apr 08, 2013 at 04:45:23PM -0700, Min Chen wrote:
>> Hi All,
>>
>> Currently CloudStack does not offer a flexible pluggable framework
>> for users to easily integrate and configure any 3rd-party object
>> stores for such backup services as registering templates, taking
>> snapshots, etc. Along with Edison's recent refactored storage
>> subsystem 2.0 that mainly refactored current CloudStack primary
>> storage implementation, we are proposing to develop a storage
>> backup object store plugin framework to allow CloudStack to
>> systematically manage and configure various types of backup data
>> stores from different vendors, like NFS, S3, Swift, etc. With this
>> new plugin framework, we would like to achieve following
>> functionalities:
>
>> 1. Support different object store providers in a uniform and
>> pluggable fashion.
>> 2. Enable region wide object backup using S3-like object store.
>> 3. Provide pluggable data motion strategies to handle data transfer
>> from one data store to another data store.
>> 4. Provide a scalable cache storage framework while moving data
>> between primary storage and backup storage for certain hypervisor
>> needs.
>> 5. Support flexible combinations of primary storage, secondary
>> storage and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware),
>> (ISCSI, Swift, KVM), ?., etc.
>> The proposed ImageStore plugin framework architecture is detailed in
>> our FS here:
>>
>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+Obj
>>ect+Store+Plugin+Framework.
>> The JIRA ticket to track this feature is:
>> https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
>> currently carried out in feature branch "object_store".
>> Please let me know your comments and suggestions.
>
>Perhaps it is too early to ask, will there be a reference
>implementation done for any object store solutions as part of the
>refactoring?
>
>--
>Prasanna.,
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Prasanna Santhanam <ts...@apache.org>.
On Mon, Apr 08, 2013 at 04:45:23PM -0700, Min Chen wrote:
> Hi All,
>
> Currently CloudStack does not offer a flexible pluggable framework
> for users to easily integrate and configure any 3rd-party object
> stores for such backup services as registering templates, taking
> snapshots, etc. Along with Edison's recent refactored storage
> subsystem 2.0 that mainly refactored current CloudStack primary
> storage implementation, we are proposing to develop a storage
> backup object store plugin framework to allow CloudStack to
> systematically manage and configure various types of backup data
> stores from different vendors, like NFS, S3, Swift, etc. With this
> new plugin framework, we would like to achieve following
> functionalities:
> 1. Support different object store providers in a uniform and
> pluggable fashion.
> 2. Enable region wide object backup using S3-like object store.
> 3. Provide pluggable data motion strategies to handle data transfer
> from one data store to another data store.
> 4. Provide a scalable cache storage framework while moving data
> between primary storage and backup storage for certain hypervisor
> needs.
> 5. Support flexible combinations of primary storage, secondary
> storage and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware),
> (ISCSI, Swift, KVM), ?., etc.
> The proposed ImageStore plugin framework architecture is detailed in
> our FS here:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+Object+Store+Plugin+Framework.
> The JIRA ticket to track this feature is:
> https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
> currently carried out in feature branch "object_store".
> Please let me know your comments and suggestions.
Perhaps it is too early to ask, will there be a reference
implementation done for any object store solutions as part of the
refactoring?
--
Prasanna.,
RE: [DIscuss]Storage image store plugin framework refactoring
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Min Chen
> Sent: Monday, April 22, 2013 10:15 AM
> To: dev@cloudstack.apache.org
> Cc: Edison Su
> Subject: Re: [DIscuss]Storage image store plugin framework refactoring
>
> See my comments inline.
>
> -min
>
> On 4/16/13 11:41 PM, "Sanjeev Neelarapu" <sa...@citrix.com>
> wrote:
>
> >Hi,
> >
> >Few more review comments:
> >1.In a fresh installed environment will SSVM continue to be there? If
> >yes will it be auto launched or admin should launch it?
> [Min] ssvm will still be there and auto-launched as before.
> >2.How do we seed system vm templates in case of object storage?
> [Min] We will trigger system vm template download while adding an S3
> object storage into cloudstack. The download is happening through MS host.
> >3.Is there any difference in seeding system vm templates with NFS
> >storage compared to what we do as of today?
> [Min] We are not planning to change system vm templates seeding with NFS
> storage at this point, still assume that you have pre-seeded system vm
> template on NFS.
> >4.In upgraded environment, can we have both NFS and object storage? If
> >not what is the procedure to migrate from nfs to object storage ?
> [Min] In our FS, we have made an assumption that "you can add multiple
> image stores from the same provider, but we prevent you from adding
> multiple image stores from different providers." This implied that we cannot
> have both NFS and S3 co-existing for this release. You raised a very good
> point, honestly I don't have an answer at this point. Edison, any thoughts on
> this?
The possible migration solution is that, treat existing NFS secondary storage as cache storage, then sync all the templates/snapshots on NFS to S3/swift object storage.
> >
> >Thanks,
> >Sanjeev
> >
> >-----Original Message-----
> >From: Min Chen [mailto:min.chen@citrix.com]
> >Sent: Wednesday, April 10, 2013 3:23 AM
> >To: dev@cloudstack.apache.org
> >Subject: Re: [DIscuss]Storage image store plugin framework refactoring
> >
> >Hi Sangeetha,
> >
> > See my answers inline.
> >
> > Thanks
> > -min
> >
> >On 4/9/13 2:48 PM, "Sangeetha Hariharan"
> ><Sa...@citrix.com>
> >wrote:
> >
> >>Can image stores from different providers (NFS,SWIFT,S3) coexist in
> >>the same deployment ?
> >>Is the scope for NFS always Zone-wide and SWIFT and S3 region-wide?
> >[Min] Currently this is our assumption: you can add multiple image
> >stores from the same provider, but we prevent you from adding multiple
> >image stores from different providers. For NFS image store providers,
> >it is always zone-wide. And For S3/Swift, it is always region-wide.
> >>
> >>There is an api for "enableImageStoreCmd" . Can multiple image stores
> >>be enabled at the same time?
> >>I don't see any api for disabling image store? What does it mean to
> >>disable an already enabled image store? Will all the resources like
> >>templates, iso and snapshot residing in this image store not be
> >>available to the users anymore?
> >[Min] On second thought, we decide to drop this api. addImageStore api
> >will automatically make the image store added active to be consistent
> >with current secondary storage behavior.
> >>
> >>
> >>deleteImageStoreCmd - Will we allow for deletion of an image store
> >>when it has resources like templates, iso and snapshots residing in it?
> >[Min] Deletion of an image store will check if it has resources like
> >template, snapshots, and volumes. If there are resources residing in
> >it, we will throw error.
> >>
> >>-Thanks
> >>Sangeetha
> >>
> >>-----Original Message-----
> >>From: Edison Su [mailto:Edison.su@citrix.com]
> >>Sent: Tuesday, April 09, 2013 2:01 PM
> >>To: dev@cloudstack.apache.org
> >>Subject: RE: [DIscuss]Storage image store plugin framework refactoring
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
> >>> Sent: Monday, April 08, 2013 11:33 PM
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: [DIscuss]Storage image store plugin framework
> >>> refactoring
> >>>
> >>> We can't deprecate APIs unless we are changing to 5.0 AFAIK the next
> >>> release in plan is 4.2
> >>For back compatibility, these APIs:
> >>addSecondaryStorageCmd,addS3Cmd,addSwiftCmd,listS3Cmd,listSwiftCm
> d,
> >>can still be wired to new code, so this APIs will still work, but we
> >>can mark them as deprecated in the API document.
> >>
> >>>
> >>> How does an upgrade from 4.0/4.1 to 4.2 work with this proposal?
> >>> - what happens to existing SSVMs?
> >>> - if there are multiple SSVMs?
> >>> - current cache-like deployments for S3 and Swift
> >>
> >>The existing deployment model will still work.
> >>There are following combinations currently supported by CloudStack:
> >>1. Only NFS as secondary storage. In the new framework, it's NFS as a
> >>backup storage.
> >>2. NFS + swift/S3. In the new framework, it's swift/s3 as backup
> >>storage, while, NFS as cache storage.
> >>So the upgrade code from pre-4.2 to 4.2 needs to handle above cases,
> >>migrate DB properly.
> >>
> >>>
> >>> The SSVM code also does duty for VMWare deployments to aid in data
> >>> movement.
> >>
> >>
> >>The existing SSVM will still work, it will be renamed to
> >>transportation VM in the code(maybe on the UI, it still be called
> >>SSVM). The main functionality of these VMs are to help transfer data
> >>from one place to another. As you said, Vmware deployment needs these
> >>VMs(as Vmware can't directly download template into primary storage),
> >>and old NFS based secondary storage deployment model needs
> them(data
> >>copy when cross zone).
> >>
> >>
> >>> How does this change?
> >>>
> >>> On 4/8/13 6:34 PM, "Sangeetha Hariharan"
> >>> <Sa...@citrix.com>
> >>> wrote:
> >>>
> >>> >Min,
> >>> >
> >>> >Could you also include the details of the API changes (new
> >>> >parameters) that will be proposed as part of this feature?
> >>> >Also it would be helpful if you list the request and response
> >>> >parameters for the new API calls.
> >>> >For all the API calls that are being deprecated , is there any
> >>> >specific error message that will be returned?
> >>> >
> >>> >-Thanks
> >>> >Sangeetha
> >>> >
> >>> >-----Original Message-----
> >>> >From: Min Chen [mailto:min.chen@citrix.com]
> >>> >Sent: Monday, April 08, 2013 4:45 PM
> >>> >To: dev@cloudstack.apache.org
> >>> >Subject: [DIscuss]Storage image store plugin framework refactoring
> >>> >
> >>> >Hi All,
> >>> >
> >>> >Currently CloudStack does not offer a flexible pluggable framework
> >>> >for users to easily integrate and configure any 3rd-party object
> >>> >stores for such backup services as registering templates, taking
> >>>snapshots, etc.
> >>> >Along with Edison's recent refactored storage subsystem 2.0 that
> >>> >mainly refactored current CloudStack primary storage
> >>> >implementation, we are proposing to develop a storage backup object
> >>> >store plugin framework to allow CloudStack to systematically manage
> >>> >and configure various types of backup data stores from different
> >>> >vendors, like NFS,
> >>>S3, Swift, etc.
> >>> >With this new plugin framework, we would like to achieve following
> >>> >functionalities:
> >>> >1. Support different object store providers in a uniform and
> >>> >pluggable fashion.
> >>> >2. Enable region wide object backup using S3-like object store.
> >>> >3. Provide pluggable data motion strategies to handle data transfer
> >>> >from one data store to another data store.
> >>> >4. Provide a scalable cache storage framework while moving data
> >>> >between primary storage and backup storage for certain hypervisor
> >>>needs.
> >>> >5. Support flexible combinations of primary storage, secondary
> >>> >storage and hypervisors, such as (NFS, NFS, Xen), (NF3, S3,
> >>> >Vmware), (ISCSI, Swift, KVM), ...., etc.
> >>> >The proposed ImageStore plugin framework architecture is detailed
> >>> >in our FS here:
> >>> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+B
> ack
> >>> >u
> >>> p+O
> >>> >bje
> >>> >ct+Store+Plugin+Framework.
> >>> >The JIRA ticket to track this feature is:
> >>> >https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
> >>> >currently carried out in feature branch "object_store".
> >>> >Please let me know your comments and suggestions.
> >>> >
> >>> >Thanks
> >>> >-min
> >>> >
> >>> >
> >>
> >
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Min Chen <mi...@citrix.com>.
See my comments inline.
-min
On 4/16/13 11:41 PM, "Sanjeev Neelarapu" <sa...@citrix.com>
wrote:
>Hi,
>
>Few more review comments:
>1.In a fresh installed environment will SSVM continue to be there? If yes
>will it be auto launched or admin should launch it?
[Min] ssvm will still be there and auto-launched as before.
>2.How do we seed system vm templates in case of object storage?
[Min] We will trigger system vm template download while adding an S3
object storage into cloudstack. The download is happening through MS host.
>3.Is there any difference in seeding system vm templates with NFS storage
>compared to what we do as of today?
[Min] We are not planning to change system vm templates seeding with NFS
storage at this point, still assume that you have pre-seeded system vm
template on NFS.
>4.In upgraded environment, can we have both NFS and object storage? If
>not what is the procedure to migrate from nfs to object storage ?
[Min] In our FS, we have made an assumption that "you can add multiple
image stores from the same provider, but we prevent you from adding
multiple image
stores from different providers." This implied that we cannot have both
NFS and S3 co-existing for this release. You raised a very good point,
honestly I don't have an answer at this point. Edison, any thoughts on
this?
>
>Thanks,
>Sanjeev
>
>-----Original Message-----
>From: Min Chen [mailto:min.chen@citrix.com]
>Sent: Wednesday, April 10, 2013 3:23 AM
>To: dev@cloudstack.apache.org
>Subject: Re: [DIscuss]Storage image store plugin framework refactoring
>
>Hi Sangeetha,
>
> See my answers inline.
>
> Thanks
> -min
>
>On 4/9/13 2:48 PM, "Sangeetha Hariharan" <Sa...@citrix.com>
>wrote:
>
>>Can image stores from different providers (NFS,SWIFT,S3) coexist in the
>>same deployment ?
>>Is the scope for NFS always Zone-wide and SWIFT and S3 region-wide?
>[Min] Currently this is our assumption: you can add multiple image stores
>from the same provider, but we prevent you from adding multiple image
>stores from different providers. For NFS image store providers, it is
>always zone-wide. And For S3/Swift, it is always region-wide.
>>
>>There is an api for "enableImageStoreCmd" . Can multiple image stores
>>be enabled at the same time?
>>I don't see any api for disabling image store? What does it mean to
>>disable an already enabled image store? Will all the resources like
>>templates, iso and snapshot residing in this image store not be
>>available to the users anymore?
>[Min] On second thought, we decide to drop this api. addImageStore api
>will automatically make the image store added active to be consistent
>with current secondary storage behavior.
>>
>>
>>deleteImageStoreCmd - Will we allow for deletion of an image store when
>>it has resources like templates, iso and snapshots residing in it?
>[Min] Deletion of an image store will check if it has resources like
>template, snapshots, and volumes. If there are resources residing in it,
>we will throw error.
>>
>>-Thanks
>>Sangeetha
>>
>>-----Original Message-----
>>From: Edison Su [mailto:Edison.su@citrix.com]
>>Sent: Tuesday, April 09, 2013 2:01 PM
>>To: dev@cloudstack.apache.org
>>Subject: RE: [DIscuss]Storage image store plugin framework refactoring
>>
>>
>>
>>> -----Original Message-----
>>> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>>> Sent: Monday, April 08, 2013 11:33 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: [DIscuss]Storage image store plugin framework
>>> refactoring
>>>
>>> We can't deprecate APIs unless we are changing to 5.0 AFAIK the next
>>> release in plan is 4.2
>>For back compatibility, these APIs:
>>addSecondaryStorageCmd,addS3Cmd,addSwiftCmd,listS3Cmd,listSwiftCmd, can
>>still be wired to new code, so this APIs will still work, but we can
>>mark them as deprecated in the API document.
>>
>>>
>>> How does an upgrade from 4.0/4.1 to 4.2 work with this proposal?
>>> - what happens to existing SSVMs?
>>> - if there are multiple SSVMs?
>>> - current cache-like deployments for S3 and Swift
>>
>>The existing deployment model will still work.
>>There are following combinations currently supported by CloudStack:
>>1. Only NFS as secondary storage. In the new framework, it's NFS as a
>>backup storage.
>>2. NFS + swift/S3. In the new framework, it's swift/s3 as backup
>>storage, while, NFS as cache storage.
>>So the upgrade code from pre-4.2 to 4.2 needs to handle above cases,
>>migrate DB properly.
>>
>>>
>>> The SSVM code also does duty for VMWare deployments to aid in data
>>> movement.
>>
>>
>>The existing SSVM will still work, it will be renamed to transportation
>>VM in the code(maybe on the UI, it still be called SSVM). The main
>>functionality of these VMs are to help transfer data from one place to
>>another. As you said, Vmware deployment needs these VMs(as Vmware
>>can't directly download template into primary storage), and old NFS
>>based secondary storage deployment model needs them(data copy when cross
>>zone).
>>
>>
>>> How does this change?
>>>
>>> On 4/8/13 6:34 PM, "Sangeetha Hariharan"
>>> <Sa...@citrix.com>
>>> wrote:
>>>
>>> >Min,
>>> >
>>> >Could you also include the details of the API changes (new
>>> >parameters) that will be proposed as part of this feature?
>>> >Also it would be helpful if you list the request and response
>>> >parameters for the new API calls.
>>> >For all the API calls that are being deprecated , is there any
>>> >specific error message that will be returned?
>>> >
>>> >-Thanks
>>> >Sangeetha
>>> >
>>> >-----Original Message-----
>>> >From: Min Chen [mailto:min.chen@citrix.com]
>>> >Sent: Monday, April 08, 2013 4:45 PM
>>> >To: dev@cloudstack.apache.org
>>> >Subject: [DIscuss]Storage image store plugin framework refactoring
>>> >
>>> >Hi All,
>>> >
>>> >Currently CloudStack does not offer a flexible pluggable framework
>>> >for users to easily integrate and configure any 3rd-party object
>>> >stores for such backup services as registering templates, taking
>>>snapshots, etc.
>>> >Along with Edison's recent refactored storage subsystem 2.0 that
>>> >mainly refactored current CloudStack primary storage implementation,
>>> >we are proposing to develop a storage backup object store plugin
>>> >framework to allow CloudStack to systematically manage and configure
>>> >various types of backup data stores from different vendors, like
>>> >NFS,
>>>S3, Swift, etc.
>>> >With this new plugin framework, we would like to achieve following
>>> >functionalities:
>>> >1. Support different object store providers in a uniform and
>>> >pluggable fashion.
>>> >2. Enable region wide object backup using S3-like object store.
>>> >3. Provide pluggable data motion strategies to handle data transfer
>>> >from one data store to another data store.
>>> >4. Provide a scalable cache storage framework while moving data
>>> >between primary storage and backup storage for certain hypervisor
>>>needs.
>>> >5. Support flexible combinations of primary storage, secondary
>>> >storage and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware),
>>> >(ISCSI, Swift, KVM), ...., etc.
>>> >The proposed ImageStore plugin framework architecture is detailed in
>>> >our FS here:
>>> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backu
>>> p+O
>>> >bje
>>> >ct+Store+Plugin+Framework.
>>> >The JIRA ticket to track this feature is:
>>> >https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
>>> >currently carried out in feature branch "object_store".
>>> >Please let me know your comments and suggestions.
>>> >
>>> >Thanks
>>> >-min
>>> >
>>> >
>>
>
RE: [DIscuss]Storage image store plugin framework refactoring
Posted by Sanjeev Neelarapu <sa...@citrix.com>.
Hi,
Few more review comments:
1.In a fresh installed environment will SSVM continue to be there? If yes will it be auto launched or admin should launch it?
2.How do we seed system vm templates in case of object storage?
3.Is there any difference in seeding system vm templates with NFS storage compared to what we do as of today?
4.In upgraded environment, can we have both NFS and object storage? If not what is the procedure to migrate from nfs to object storage ?
Thanks,
Sanjeev
-----Original Message-----
From: Min Chen [mailto:min.chen@citrix.com]
Sent: Wednesday, April 10, 2013 3:23 AM
To: dev@cloudstack.apache.org
Subject: Re: [DIscuss]Storage image store plugin framework refactoring
Hi Sangeetha,
See my answers inline.
Thanks
-min
On 4/9/13 2:48 PM, "Sangeetha Hariharan" <Sa...@citrix.com>
wrote:
>Can image stores from different providers (NFS,SWIFT,S3) coexist in the
>same deployment ?
>Is the scope for NFS always Zone-wide and SWIFT and S3 region-wide?
[Min] Currently this is our assumption: you can add multiple image stores from the same provider, but we prevent you from adding multiple image stores from different providers. For NFS image store providers, it is always zone-wide. And For S3/Swift, it is always region-wide.
>
>There is an api for "enableImageStoreCmd" . Can multiple image stores
>be enabled at the same time?
>I don't see any api for disabling image store? What does it mean to
>disable an already enabled image store? Will all the resources like
>templates, iso and snapshot residing in this image store not be
>available to the users anymore?
[Min] On second thought, we decide to drop this api. addImageStore api will automatically make the image store added active to be consistent with current secondary storage behavior.
>
>
>deleteImageStoreCmd - Will we allow for deletion of an image store when
>it has resources like templates, iso and snapshots residing in it?
[Min] Deletion of an image store will check if it has resources like template, snapshots, and volumes. If there are resources residing in it, we will throw error.
>
>-Thanks
>Sangeetha
>
>-----Original Message-----
>From: Edison Su [mailto:Edison.su@citrix.com]
>Sent: Tuesday, April 09, 2013 2:01 PM
>To: dev@cloudstack.apache.org
>Subject: RE: [DIscuss]Storage image store plugin framework refactoring
>
>
>
>> -----Original Message-----
>> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>> Sent: Monday, April 08, 2013 11:33 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: [DIscuss]Storage image store plugin framework
>> refactoring
>>
>> We can't deprecate APIs unless we are changing to 5.0 AFAIK the next
>> release in plan is 4.2
>For back compatibility, these APIs:
>addSecondaryStorageCmd,addS3Cmd,addSwiftCmd,listS3Cmd,listSwiftCmd, can
>still be wired to new code, so this APIs will still work, but we can
>mark them as deprecated in the API document.
>
>>
>> How does an upgrade from 4.0/4.1 to 4.2 work with this proposal?
>> - what happens to existing SSVMs?
>> - if there are multiple SSVMs?
>> - current cache-like deployments for S3 and Swift
>
>The existing deployment model will still work.
>There are following combinations currently supported by CloudStack:
>1. Only NFS as secondary storage. In the new framework, it's NFS as a
>backup storage.
>2. NFS + swift/S3. In the new framework, it's swift/s3 as backup
>storage, while, NFS as cache storage.
>So the upgrade code from pre-4.2 to 4.2 needs to handle above cases,
>migrate DB properly.
>
>>
>> The SSVM code also does duty for VMWare deployments to aid in data
>> movement.
>
>
>The existing SSVM will still work, it will be renamed to transportation
>VM in the code(maybe on the UI, it still be called SSVM). The main
>functionality of these VMs are to help transfer data from one place to
>another. As you said, Vmware deployment needs these VMs(as Vmware
>can't directly download template into primary storage), and old NFS
>based secondary storage deployment model needs them(data copy when cross zone).
>
>
>> How does this change?
>>
>> On 4/8/13 6:34 PM, "Sangeetha Hariharan"
>> <Sa...@citrix.com>
>> wrote:
>>
>> >Min,
>> >
>> >Could you also include the details of the API changes (new
>> >parameters) that will be proposed as part of this feature?
>> >Also it would be helpful if you list the request and response
>> >parameters for the new API calls.
>> >For all the API calls that are being deprecated , is there any
>> >specific error message that will be returned?
>> >
>> >-Thanks
>> >Sangeetha
>> >
>> >-----Original Message-----
>> >From: Min Chen [mailto:min.chen@citrix.com]
>> >Sent: Monday, April 08, 2013 4:45 PM
>> >To: dev@cloudstack.apache.org
>> >Subject: [DIscuss]Storage image store plugin framework refactoring
>> >
>> >Hi All,
>> >
>> >Currently CloudStack does not offer a flexible pluggable framework
>> >for users to easily integrate and configure any 3rd-party object
>> >stores for such backup services as registering templates, taking
>>snapshots, etc.
>> >Along with Edison's recent refactored storage subsystem 2.0 that
>> >mainly refactored current CloudStack primary storage implementation,
>> >we are proposing to develop a storage backup object store plugin
>> >framework to allow CloudStack to systematically manage and configure
>> >various types of backup data stores from different vendors, like
>> >NFS,
>>S3, Swift, etc.
>> >With this new plugin framework, we would like to achieve following
>> >functionalities:
>> >1. Support different object store providers in a uniform and
>> >pluggable fashion.
>> >2. Enable region wide object backup using S3-like object store.
>> >3. Provide pluggable data motion strategies to handle data transfer
>> >from one data store to another data store.
>> >4. Provide a scalable cache storage framework while moving data
>> >between primary storage and backup storage for certain hypervisor
>>needs.
>> >5. Support flexible combinations of primary storage, secondary
>> >storage and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware),
>> >(ISCSI, Swift, KVM), ...., etc.
>> >The proposed ImageStore plugin framework architecture is detailed in
>> >our FS here:
>> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backu
>> p+O
>> >bje
>> >ct+Store+Plugin+Framework.
>> >The JIRA ticket to track this feature is:
>> >https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
>> >currently carried out in feature branch "object_store".
>> >Please let me know your comments and suggestions.
>> >
>> >Thanks
>> >-min
>> >
>> >
>
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Min Chen <mi...@citrix.com>.
Hi Sangeetha,
See my answers inline.
Thanks
-min
On 4/9/13 2:48 PM, "Sangeetha Hariharan" <Sa...@citrix.com>
wrote:
>Can image stores from different providers (NFS,SWIFT,S3) coexist in the
>same deployment ?
>Is the scope for NFS always Zone-wide and SWIFT and S3 region-wide?
[Min] Currently this is our assumption: you can add multiple image stores
from the same provider, but we prevent you from adding multiple image
stores from different providers. For NFS image store providers, it is
always zone-wide. And For S3/Swift, it is always region-wide.
>
>There is an api for "enableImageStoreCmd" . Can multiple image stores be
>enabled at the same time?
>I don't see any api for disabling image store? What does it mean to
>disable an already enabled image store? Will all the resources like
>templates, iso and snapshot residing in this image store not be available
>to the users anymore?
[Min] On second thought, we decide to drop this api. addImageStore api
will automatically make the image store added active to be consistent with
current secondary storage behavior.
>
>
>deleteImageStoreCmd - Will we allow for deletion of an image store when
>it has resources like templates, iso and snapshots residing in it?
[Min] Deletion of an image store will check if it has resources like
template, snapshots, and volumes. If there are resources residing in it,
we will throw error.
>
>-Thanks
>Sangeetha
>
>-----Original Message-----
>From: Edison Su [mailto:Edison.su@citrix.com]
>Sent: Tuesday, April 09, 2013 2:01 PM
>To: dev@cloudstack.apache.org
>Subject: RE: [DIscuss]Storage image store plugin framework refactoring
>
>
>
>> -----Original Message-----
>> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>> Sent: Monday, April 08, 2013 11:33 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: [DIscuss]Storage image store plugin framework refactoring
>>
>> We can't deprecate APIs unless we are changing to 5.0 AFAIK the next
>> release in plan is 4.2
>For back compatibility, these APIs:
>addSecondaryStorageCmd,addS3Cmd,addSwiftCmd,listS3Cmd,listSwiftCmd, can
>still be wired to new code, so this APIs will still work, but we can mark
>them as deprecated in the API document.
>
>>
>> How does an upgrade from 4.0/4.1 to 4.2 work with this proposal?
>> - what happens to existing SSVMs?
>> - if there are multiple SSVMs?
>> - current cache-like deployments for S3 and Swift
>
>The existing deployment model will still work.
>There are following combinations currently supported by CloudStack:
>1. Only NFS as secondary storage. In the new framework, it's NFS as a
>backup storage.
>2. NFS + swift/S3. In the new framework, it's swift/s3 as backup storage,
>while, NFS as cache storage.
>So the upgrade code from pre-4.2 to 4.2 needs to handle above cases,
>migrate DB properly.
>
>>
>> The SSVM code also does duty for VMWare deployments to aid in data
>> movement.
>
>
>The existing SSVM will still work, it will be renamed to transportation
>VM in the code(maybe on the UI, it still be called SSVM). The main
>functionality of these VMs are to help transfer data from one place to
>another. As you said, Vmware deployment needs these VMs(as Vmware can't
>directly download template into primary storage), and old NFS based
>secondary storage deployment model needs them(data copy when cross zone).
>
>
>> How does this change?
>>
>> On 4/8/13 6:34 PM, "Sangeetha Hariharan"
>> <Sa...@citrix.com>
>> wrote:
>>
>> >Min,
>> >
>> >Could you also include the details of the API changes (new
>> >parameters) that will be proposed as part of this feature?
>> >Also it would be helpful if you list the request and response
>> >parameters for the new API calls.
>> >For all the API calls that are being deprecated , is there any
>> >specific error message that will be returned?
>> >
>> >-Thanks
>> >Sangeetha
>> >
>> >-----Original Message-----
>> >From: Min Chen [mailto:min.chen@citrix.com]
>> >Sent: Monday, April 08, 2013 4:45 PM
>> >To: dev@cloudstack.apache.org
>> >Subject: [DIscuss]Storage image store plugin framework refactoring
>> >
>> >Hi All,
>> >
>> >Currently CloudStack does not offer a flexible pluggable framework
>> >for users to easily integrate and configure any 3rd-party object
>> >stores for such backup services as registering templates, taking
>>snapshots, etc.
>> >Along with Edison's recent refactored storage subsystem 2.0 that
>> >mainly refactored current CloudStack primary storage implementation,
>> >we are proposing to develop a storage backup object store plugin
>> >framework to allow CloudStack to systematically manage and configure
>> >various types of backup data stores from different vendors, like NFS,
>>S3, Swift, etc.
>> >With this new plugin framework, we would like to achieve following
>> >functionalities:
>> >1. Support different object store providers in a uniform and
>> >pluggable fashion.
>> >2. Enable region wide object backup using S3-like object store.
>> >3. Provide pluggable data motion strategies to handle data transfer
>> >from one data store to another data store.
>> >4. Provide a scalable cache storage framework while moving data
>> >between primary storage and backup storage for certain hypervisor
>>needs.
>> >5. Support flexible combinations of primary storage, secondary
>> >storage and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware),
>> >(ISCSI, Swift, KVM), ...., etc.
>> >The proposed ImageStore plugin framework architecture is detailed in
>> >our FS here:
>> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backu
>> p+O
>> >bje
>> >ct+Store+Plugin+Framework.
>> >The JIRA ticket to track this feature is:
>> >https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
>> >currently carried out in feature branch "object_store".
>> >Please let me know your comments and suggestions.
>> >
>> >Thanks
>> >-min
>> >
>> >
>
RE: [DIscuss]Storage image store plugin framework refactoring
Posted by Sangeetha Hariharan <Sa...@citrix.com>.
Can image stores from different providers (NFS,SWIFT,S3) coexist in the same deployment ?
Is the scope for NFS always Zone-wide and SWIFT and S3 region-wide?
There is an api for "enableImageStoreCmd" . Can multiple image stores be enabled at the same time?
I don't see any api for disabling image store? What does it mean to disable an already enabled image store? Will all the resources like templates, iso and snapshot residing in this image store not be available to the users anymore?
deleteImageStoreCmd - Will we allow for deletion of an image store when it has resources like templates, iso and snapshots residing in it?
-Thanks
Sangeetha
-----Original Message-----
From: Edison Su [mailto:Edison.su@citrix.com]
Sent: Tuesday, April 09, 2013 2:01 PM
To: dev@cloudstack.apache.org
Subject: RE: [DIscuss]Storage image store plugin framework refactoring
> -----Original Message-----
> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
> Sent: Monday, April 08, 2013 11:33 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DIscuss]Storage image store plugin framework refactoring
>
> We can't deprecate APIs unless we are changing to 5.0 AFAIK the next
> release in plan is 4.2
For back compatibility, these APIs: addSecondaryStorageCmd,addS3Cmd,addSwiftCmd,listS3Cmd,listSwiftCmd, can still be wired to new code, so this APIs will still work, but we can mark them as deprecated in the API document.
>
> How does an upgrade from 4.0/4.1 to 4.2 work with this proposal?
> - what happens to existing SSVMs?
> - if there are multiple SSVMs?
> - current cache-like deployments for S3 and Swift
The existing deployment model will still work.
There are following combinations currently supported by CloudStack:
1. Only NFS as secondary storage. In the new framework, it's NFS as a backup storage.
2. NFS + swift/S3. In the new framework, it's swift/s3 as backup storage, while, NFS as cache storage.
So the upgrade code from pre-4.2 to 4.2 needs to handle above cases, migrate DB properly.
>
> The SSVM code also does duty for VMWare deployments to aid in data
> movement.
The existing SSVM will still work, it will be renamed to transportation VM in the code(maybe on the UI, it still be called SSVM). The main functionality of these VMs are to help transfer data from one place to another. As you said, Vmware deployment needs these VMs(as Vmware can't directly download template into primary storage), and old NFS based secondary storage deployment model needs them(data copy when cross zone).
> How does this change?
>
> On 4/8/13 6:34 PM, "Sangeetha Hariharan"
> <Sa...@citrix.com>
> wrote:
>
> >Min,
> >
> >Could you also include the details of the API changes (new
> >parameters) that will be proposed as part of this feature?
> >Also it would be helpful if you list the request and response
> >parameters for the new API calls.
> >For all the API calls that are being deprecated , is there any
> >specific error message that will be returned?
> >
> >-Thanks
> >Sangeetha
> >
> >-----Original Message-----
> >From: Min Chen [mailto:min.chen@citrix.com]
> >Sent: Monday, April 08, 2013 4:45 PM
> >To: dev@cloudstack.apache.org
> >Subject: [DIscuss]Storage image store plugin framework refactoring
> >
> >Hi All,
> >
> >Currently CloudStack does not offer a flexible pluggable framework
> >for users to easily integrate and configure any 3rd-party object
> >stores for such backup services as registering templates, taking snapshots, etc.
> >Along with Edison's recent refactored storage subsystem 2.0 that
> >mainly refactored current CloudStack primary storage implementation,
> >we are proposing to develop a storage backup object store plugin
> >framework to allow CloudStack to systematically manage and configure
> >various types of backup data stores from different vendors, like NFS, S3, Swift, etc.
> >With this new plugin framework, we would like to achieve following
> >functionalities:
> >1. Support different object store providers in a uniform and
> >pluggable fashion.
> >2. Enable region wide object backup using S3-like object store.
> >3. Provide pluggable data motion strategies to handle data transfer
> >from one data store to another data store.
> >4. Provide a scalable cache storage framework while moving data
> >between primary storage and backup storage for certain hypervisor needs.
> >5. Support flexible combinations of primary storage, secondary
> >storage and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware),
> >(ISCSI, Swift, KVM), ...., etc.
> >The proposed ImageStore plugin framework architecture is detailed in
> >our FS here:
> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backu
> p+O
> >bje
> >ct+Store+Plugin+Framework.
> >The JIRA ticket to track this feature is:
> >https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
> >currently carried out in feature branch "object_store".
> >Please let me know your comments and suggestions.
> >
> >Thanks
> >-min
> >
> >
RE: [DIscuss]Storage image store plugin framework refactoring
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
> Sent: Monday, April 08, 2013 11:33 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DIscuss]Storage image store plugin framework refactoring
>
> We can't deprecate APIs unless we are changing to 5.0 AFAIK the next
> release in plan is 4.2
For back compatibility, these APIs: addSecondaryStorageCmd,addS3Cmd,addSwiftCmd,listS3Cmd,listSwiftCmd, can still be wired to new code, so this APIs will still work, but we can mark them as deprecated in the API document.
>
> How does an upgrade from 4.0/4.1 to 4.2 work with this proposal?
> - what happens to existing SSVMs?
> - if there are multiple SSVMs?
> - current cache-like deployments for S3 and Swift
The existing deployment model will still work.
There are following combinations currently supported by CloudStack:
1. Only NFS as secondary storage. In the new framework, it's NFS as a backup storage.
2. NFS + swift/S3. In the new framework, it's swift/s3 as backup storage, while, NFS as cache storage.
So the upgrade code from pre-4.2 to 4.2 needs to handle above cases, migrate DB properly.
>
> The SSVM code also does duty for VMWare deployments to aid in data
> movement.
The existing SSVM will still work, it will be renamed to transportation VM in the code(maybe on the UI, it still be called SSVM). The main functionality of these VMs are to help transfer data from one place to another. As you said, Vmware deployment needs these VMs(as Vmware can't directly download template into primary storage), and old NFS based secondary storage deployment model needs them(data copy when cross zone).
> How does this change?
>
> On 4/8/13 6:34 PM, "Sangeetha Hariharan" <Sa...@citrix.com>
> wrote:
>
> >Min,
> >
> >Could you also include the details of the API changes (new parameters)
> >that will be proposed as part of this feature?
> >Also it would be helpful if you list the request and response
> >parameters for the new API calls.
> >For all the API calls that are being deprecated , is there any specific
> >error message that will be returned?
> >
> >-Thanks
> >Sangeetha
> >
> >-----Original Message-----
> >From: Min Chen [mailto:min.chen@citrix.com]
> >Sent: Monday, April 08, 2013 4:45 PM
> >To: dev@cloudstack.apache.org
> >Subject: [DIscuss]Storage image store plugin framework refactoring
> >
> >Hi All,
> >
> >Currently CloudStack does not offer a flexible pluggable framework for
> >users to easily integrate and configure any 3rd-party object stores for
> >such backup services as registering templates, taking snapshots, etc.
> >Along with Edison's recent refactored storage subsystem 2.0 that mainly
> >refactored current CloudStack primary storage implementation, we are
> >proposing to develop a storage backup object store plugin framework to
> >allow CloudStack to systematically manage and configure various types
> >of backup data stores from different vendors, like NFS, S3, Swift, etc.
> >With this new plugin framework, we would like to achieve following
> >functionalities:
> >1. Support different object store providers in a uniform and pluggable
> >fashion.
> >2. Enable region wide object backup using S3-like object store.
> >3. Provide pluggable data motion strategies to handle data transfer
> >from one data store to another data store.
> >4. Provide a scalable cache storage framework while moving data between
> >primary storage and backup storage for certain hypervisor needs.
> >5. Support flexible combinations of primary storage, secondary storage
> >and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware), (ISCSI,
> >Swift, KVM), ...., etc.
> >The proposed ImageStore plugin framework architecture is detailed in
> >our FS here:
> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backu
> p+O
> >bje
> >ct+Store+Plugin+Framework.
> >The JIRA ticket to track this feature is:
> >https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
> >currently carried out in feature branch "object_store".
> >Please let me know your comments and suggestions.
> >
> >Thanks
> >-min
> >
> >
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Chiradeep Vittal <Ch...@citrix.com>.
We can't deprecate APIs unless we are changing to 5.0
AFAIK the next release in plan is 4.2
How does an upgrade from 4.0/4.1 to 4.2 work with this proposal?
- what happens to existing SSVMs?
- if there are multiple SSVMs?
- current cache-like deployments for S3 and Swift
The SSVM code also does duty for VMWare deployments to aid in data
movement.
How does this change?
On 4/8/13 6:34 PM, "Sangeetha Hariharan" <Sa...@citrix.com>
wrote:
>Min,
>
>Could you also include the details of the API changes (new parameters)
>that will be proposed as part of this feature?
>Also it would be helpful if you list the request and response parameters
>for the new API calls.
>For all the API calls that are being deprecated , is there any specific
>error message that will be returned?
>
>-Thanks
>Sangeetha
>
>-----Original Message-----
>From: Min Chen [mailto:min.chen@citrix.com]
>Sent: Monday, April 08, 2013 4:45 PM
>To: dev@cloudstack.apache.org
>Subject: [DIscuss]Storage image store plugin framework refactoring
>
>Hi All,
>
>Currently CloudStack does not offer a flexible pluggable framework for
>users to easily integrate and configure any 3rd-party object stores for
>such backup services as registering templates, taking snapshots, etc.
>Along with Edison's recent refactored storage subsystem 2.0 that mainly
>refactored current CloudStack primary storage implementation, we are
>proposing to develop a storage backup object store plugin framework to
>allow CloudStack to systematically manage and configure various types of
>backup data stores from different vendors, like NFS, S3, Swift, etc. With
>this new plugin framework, we would like to achieve following
>functionalities:
>1. Support different object store providers in a uniform and pluggable
>fashion.
>2. Enable region wide object backup using S3-like object store.
>3. Provide pluggable data motion strategies to handle data transfer from
>one data store to another data store.
>4. Provide a scalable cache storage framework while moving data between
>primary storage and backup storage for certain hypervisor needs.
>5. Support flexible combinations of primary storage, secondary storage
>and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware), (ISCSI,
>Swift, KVM), ...., etc.
>The proposed ImageStore plugin framework architecture is detailed in our
>FS here:
>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+Obje
>ct+Store+Plugin+Framework.
>The JIRA ticket to track this feature is:
>https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
>currently carried out in feature branch "object_store".
>Please let me know your comments and suggestions.
>
>Thanks
>-min
>
>
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Nitin Mehta <Ni...@citrix.com>.
On 09/04/13 11:03 PM, "Min Chen" <mi...@citrix.com> wrote:
>It may not be that simple to just use api name alias. For example, we
>currently have listS3sCmd, with new image store plugin framework, we
>introduced a new API "listImageStoresCmd" to cover image stores from
>various providers, not only S3. We also introduced a new response
>ImageStoreResponse for this new listImageStoreCmd api. So you are
>suggesting to use the following naming alias to direct old apis to the new
>apis, right?
>
>@APICommand(name = "listImageStores, listS3s, listSwift", ..
>
>
>But there are two issues here:
>1. Previous listS3sCmd api is corresponding to new API
>listImageStores&provider=S3, so this will not be a simple redirect.
>2. Previous listS3sCmd response is S3Response, which is different from new
>ImageStoreResponse, although its information can be found in new
>ImageStoreResponse. Will this break back compatibility as well?
My $.02
Yes, it will break the backward compatibility. For now keep both the
api's, but the UI should start using the new api with
listImageStores&provider=S3.
We should use deprecated annotation to mark that the api is deprecated
just like in the Java Docs and finally remove the api in 5.0.
>
>Thanks
>-min
>
>
>On 4/9/13 10:14 AM, "Chiradeep Vittal" <Ch...@citrix.com>
>wrote:
>
>>We can have alias for an existing API. See the other ML discussion.
>>
>>On 4/9/13 9:27 AM, "Min Chen" <mi...@citrix.com> wrote:
>>
>>>Yes, I will include more api change details in FS in next few days.
>>>According to Chiradeep, it seems that we cannot simply deprecate old API
>>>in 4.2, Edison and I will discuss this and update FS with details on how
>>>to handle these old APIs.
>>>
>>>Thanks
>>>-min
>>>
>>>On 4/8/13 6:34 PM, "Sangeetha Hariharan"
>>><Sa...@citrix.com>
>>>wrote:
>>>
>>>>Min,
>>>>
>>>>Could you also include the details of the API changes (new parameters)
>>>>that will be proposed as part of this feature?
>>>>Also it would be helpful if you list the request and response
>>>>parameters
>>>>for the new API calls.
>>>>For all the API calls that are being deprecated , is there any specific
>>>>error message that will be returned?
>>>>
>>>>-Thanks
>>>>Sangeetha
>>>>
>>>>-----Original Message-----
>>>>From: Min Chen [mailto:min.chen@citrix.com]
>>>>Sent: Monday, April 08, 2013 4:45 PM
>>>>To: dev@cloudstack.apache.org
>>>>Subject: [DIscuss]Storage image store plugin framework refactoring
>>>>
>>>>Hi All,
>>>>
>>>>Currently CloudStack does not offer a flexible pluggable framework for
>>>>users to easily integrate and configure any 3rd-party object stores for
>>>>such backup services as registering templates, taking snapshots, etc.
>>>>Along with Edison's recent refactored storage subsystem 2.0 that mainly
>>>>refactored current CloudStack primary storage implementation, we are
>>>>proposing to develop a storage backup object store plugin framework to
>>>>allow CloudStack to systematically manage and configure various types
>>>>of
>>>>backup data stores from different vendors, like NFS, S3, Swift, etc.
>>>>With
>>>>this new plugin framework, we would like to achieve following
>>>>functionalities:
>>>>1. Support different object store providers in a uniform and pluggable
>>>>fashion.
>>>>2. Enable region wide object backup using S3-like object store.
>>>>3. Provide pluggable data motion strategies to handle data transfer
>>>>from
>>>>one data store to another data store.
>>>>4. Provide a scalable cache storage framework while moving data between
>>>>primary storage and backup storage for certain hypervisor needs.
>>>>5. Support flexible combinations of primary storage, secondary storage
>>>>and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware), (ISCSI,
>>>>Swift, KVM), ...., etc.
>>>>The proposed ImageStore plugin framework architecture is detailed in
>>>>our
>>>>FS here:
>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+O
>>>>b
>>>>j
>>>>e
>>>>ct+Store+Plugin+Framework.
>>>>The JIRA ticket to track this feature is:
>>>>https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
>>>>currently carried out in feature branch "object_store".
>>>>Please let me know your comments and suggestions.
>>>>
>>>>Thanks
>>>>-min
>>>>
>>>>
>>>
>>
>
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Min Chen <mi...@citrix.com>.
It may not be that simple to just use api name alias. For example, we
currently have listS3sCmd, with new image store plugin framework, we
introduced a new API "listImageStoresCmd" to cover image stores from
various providers, not only S3. We also introduced a new response
ImageStoreResponse for this new listImageStoreCmd api. So you are
suggesting to use the following naming alias to direct old apis to the new
apis, right?
@APICommand(name = "listImageStores, listS3s, listSwift", ..
But there are two issues here:
1. Previous listS3sCmd api is corresponding to new API
listImageStores&provider=S3, so this will not be a simple redirect.
2. Previous listS3sCmd response is S3Response, which is different from new
ImageStoreResponse, although its information can be found in new
ImageStoreResponse. Will this break back compatibility as well?
Thanks
-min
On 4/9/13 10:14 AM, "Chiradeep Vittal" <Ch...@citrix.com> wrote:
>We can have alias for an existing API. See the other ML discussion.
>
>On 4/9/13 9:27 AM, "Min Chen" <mi...@citrix.com> wrote:
>
>>Yes, I will include more api change details in FS in next few days.
>>According to Chiradeep, it seems that we cannot simply deprecate old API
>>in 4.2, Edison and I will discuss this and update FS with details on how
>>to handle these old APIs.
>>
>>Thanks
>>-min
>>
>>On 4/8/13 6:34 PM, "Sangeetha Hariharan" <Sa...@citrix.com>
>>wrote:
>>
>>>Min,
>>>
>>>Could you also include the details of the API changes (new parameters)
>>>that will be proposed as part of this feature?
>>>Also it would be helpful if you list the request and response parameters
>>>for the new API calls.
>>>For all the API calls that are being deprecated , is there any specific
>>>error message that will be returned?
>>>
>>>-Thanks
>>>Sangeetha
>>>
>>>-----Original Message-----
>>>From: Min Chen [mailto:min.chen@citrix.com]
>>>Sent: Monday, April 08, 2013 4:45 PM
>>>To: dev@cloudstack.apache.org
>>>Subject: [DIscuss]Storage image store plugin framework refactoring
>>>
>>>Hi All,
>>>
>>>Currently CloudStack does not offer a flexible pluggable framework for
>>>users to easily integrate and configure any 3rd-party object stores for
>>>such backup services as registering templates, taking snapshots, etc.
>>>Along with Edison's recent refactored storage subsystem 2.0 that mainly
>>>refactored current CloudStack primary storage implementation, we are
>>>proposing to develop a storage backup object store plugin framework to
>>>allow CloudStack to systematically manage and configure various types of
>>>backup data stores from different vendors, like NFS, S3, Swift, etc.
>>>With
>>>this new plugin framework, we would like to achieve following
>>>functionalities:
>>>1. Support different object store providers in a uniform and pluggable
>>>fashion.
>>>2. Enable region wide object backup using S3-like object store.
>>>3. Provide pluggable data motion strategies to handle data transfer from
>>>one data store to another data store.
>>>4. Provide a scalable cache storage framework while moving data between
>>>primary storage and backup storage for certain hypervisor needs.
>>>5. Support flexible combinations of primary storage, secondary storage
>>>and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware), (ISCSI,
>>>Swift, KVM), ...., etc.
>>>The proposed ImageStore plugin framework architecture is detailed in our
>>>FS here:
>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+Ob
>>>j
>>>e
>>>ct+Store+Plugin+Framework.
>>>The JIRA ticket to track this feature is:
>>>https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
>>>currently carried out in feature branch "object_store".
>>>Please let me know your comments and suggestions.
>>>
>>>Thanks
>>>-min
>>>
>>>
>>
>
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Chiradeep Vittal <Ch...@citrix.com>.
We can have alias for an existing API. See the other ML discussion.
On 4/9/13 9:27 AM, "Min Chen" <mi...@citrix.com> wrote:
>Yes, I will include more api change details in FS in next few days.
>According to Chiradeep, it seems that we cannot simply deprecate old API
>in 4.2, Edison and I will discuss this and update FS with details on how
>to handle these old APIs.
>
>Thanks
>-min
>
>On 4/8/13 6:34 PM, "Sangeetha Hariharan" <Sa...@citrix.com>
>wrote:
>
>>Min,
>>
>>Could you also include the details of the API changes (new parameters)
>>that will be proposed as part of this feature?
>>Also it would be helpful if you list the request and response parameters
>>for the new API calls.
>>For all the API calls that are being deprecated , is there any specific
>>error message that will be returned?
>>
>>-Thanks
>>Sangeetha
>>
>>-----Original Message-----
>>From: Min Chen [mailto:min.chen@citrix.com]
>>Sent: Monday, April 08, 2013 4:45 PM
>>To: dev@cloudstack.apache.org
>>Subject: [DIscuss]Storage image store plugin framework refactoring
>>
>>Hi All,
>>
>>Currently CloudStack does not offer a flexible pluggable framework for
>>users to easily integrate and configure any 3rd-party object stores for
>>such backup services as registering templates, taking snapshots, etc.
>>Along with Edison's recent refactored storage subsystem 2.0 that mainly
>>refactored current CloudStack primary storage implementation, we are
>>proposing to develop a storage backup object store plugin framework to
>>allow CloudStack to systematically manage and configure various types of
>>backup data stores from different vendors, like NFS, S3, Swift, etc. With
>>this new plugin framework, we would like to achieve following
>>functionalities:
>>1. Support different object store providers in a uniform and pluggable
>>fashion.
>>2. Enable region wide object backup using S3-like object store.
>>3. Provide pluggable data motion strategies to handle data transfer from
>>one data store to another data store.
>>4. Provide a scalable cache storage framework while moving data between
>>primary storage and backup storage for certain hypervisor needs.
>>5. Support flexible combinations of primary storage, secondary storage
>>and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware), (ISCSI,
>>Swift, KVM), ...., etc.
>>The proposed ImageStore plugin framework architecture is detailed in our
>>FS here:
>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+Obj
>>e
>>ct+Store+Plugin+Framework.
>>The JIRA ticket to track this feature is:
>>https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
>>currently carried out in feature branch "object_store".
>>Please let me know your comments and suggestions.
>>
>>Thanks
>>-min
>>
>>
>
Re: [DIscuss]Storage image store plugin framework refactoring
Posted by Min Chen <mi...@citrix.com>.
Yes, I will include more api change details in FS in next few days.
According to Chiradeep, it seems that we cannot simply deprecate old API
in 4.2, Edison and I will discuss this and update FS with details on how
to handle these old APIs.
Thanks
-min
On 4/8/13 6:34 PM, "Sangeetha Hariharan" <Sa...@citrix.com>
wrote:
>Min,
>
>Could you also include the details of the API changes (new parameters)
>that will be proposed as part of this feature?
>Also it would be helpful if you list the request and response parameters
>for the new API calls.
>For all the API calls that are being deprecated , is there any specific
>error message that will be returned?
>
>-Thanks
>Sangeetha
>
>-----Original Message-----
>From: Min Chen [mailto:min.chen@citrix.com]
>Sent: Monday, April 08, 2013 4:45 PM
>To: dev@cloudstack.apache.org
>Subject: [DIscuss]Storage image store plugin framework refactoring
>
>Hi All,
>
>Currently CloudStack does not offer a flexible pluggable framework for
>users to easily integrate and configure any 3rd-party object stores for
>such backup services as registering templates, taking snapshots, etc.
>Along with Edison's recent refactored storage subsystem 2.0 that mainly
>refactored current CloudStack primary storage implementation, we are
>proposing to develop a storage backup object store plugin framework to
>allow CloudStack to systematically manage and configure various types of
>backup data stores from different vendors, like NFS, S3, Swift, etc. With
>this new plugin framework, we would like to achieve following
>functionalities:
>1. Support different object store providers in a uniform and pluggable
>fashion.
>2. Enable region wide object backup using S3-like object store.
>3. Provide pluggable data motion strategies to handle data transfer from
>one data store to another data store.
>4. Provide a scalable cache storage framework while moving data between
>primary storage and backup storage for certain hypervisor needs.
>5. Support flexible combinations of primary storage, secondary storage
>and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware), (ISCSI,
>Swift, KVM), ...., etc.
>The proposed ImageStore plugin framework architecture is detailed in our
>FS here:
>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+Obje
>ct+Store+Plugin+Framework.
>The JIRA ticket to track this feature is:
>https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is
>currently carried out in feature branch "object_store".
>Please let me know your comments and suggestions.
>
>Thanks
>-min
>
>
RE: [DIscuss]Storage image store plugin framework refactoring
Posted by Sangeetha Hariharan <Sa...@citrix.com>.
Min,
Could you also include the details of the API changes (new parameters) that will be proposed as part of this feature?
Also it would be helpful if you list the request and response parameters for the new API calls.
For all the API calls that are being deprecated , is there any specific error message that will be returned?
-Thanks
Sangeetha
-----Original Message-----
From: Min Chen [mailto:min.chen@citrix.com]
Sent: Monday, April 08, 2013 4:45 PM
To: dev@cloudstack.apache.org
Subject: [DIscuss]Storage image store plugin framework refactoring
Hi All,
Currently CloudStack does not offer a flexible pluggable framework for users to easily integrate and configure any 3rd-party object stores for such backup services as registering templates, taking snapshots, etc. Along with Edison's recent refactored storage subsystem 2.0 that mainly refactored current CloudStack primary storage implementation, we are proposing to develop a storage backup object store plugin framework to allow CloudStack to systematically manage and configure various types of backup data stores from different vendors, like NFS, S3, Swift, etc. With this new plugin framework, we would like to achieve following functionalities:
1. Support different object store providers in a uniform and pluggable fashion.
2. Enable region wide object backup using S3-like object store.
3. Provide pluggable data motion strategies to handle data transfer from one data store to another data store.
4. Provide a scalable cache storage framework while moving data between primary storage and backup storage for certain hypervisor needs.
5. Support flexible combinations of primary storage, secondary storage and hypervisors, such as (NFS, NFS, Xen), (NF3, S3, Vmware), (ISCSI, Swift, KVM), ...., etc.
The proposed ImageStore plugin framework architecture is detailed in our FS here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+Object+Store+Plugin+Framework.
The JIRA ticket to track this feature is: https://issues.apache.org/jira/browse/CLOUDSTACK-1975. The work is currently carried out in feature branch "object_store".
Please let me know your comments and suggestions.
Thanks
-min