You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Edison Su <Ed...@citrix.com> on 2013/06/15 05:07:39 UTC
[MERGE]object_store branch into master [Second round]
Hi all,
The second round, call for merge object_store branch into master, is coming!
The issues fixed:
1. All the major issues addressed by John are fixed:
1.1 A cache storage replacement algorithm is added: StorageCacheReplacementAlgorithmLRU, based on the reference count and least recently used.
1.2 A new S3 transport is added, can upload > 5G template into S3 directly.
1.3 Retry, if S3 upload failed.
1.4 some comments from https://reviews.apache.org/r/11277/, mostly, the coding style, are addressed and clean up some unused code.
2. DB upgrade path from 4.1 to 4.2
3. Bug fix
The size of the patch is even bigger now, around 100000 LOC, you can find about the patch from https://reviews.apache.org/r/11277/diff/2/.
Comments/feedback are welcome. Thanks.
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: Friday, May 17, 2013 1:11 AM
> To: dev@cloudstack.apache.org
> Subject: [MERGE]object_store branch into master
>
> Hi all,
> Min and I worked on object_store branch during the last one and half
> month. We made a lot of refactor on the storage code, mostly related to
> secondary storage, but also on the general storage framework. The following
> goals are made:
>
> 1. An unified storage framework. Both secondary storages(nfs/s3/swift
> etc) and primary storages will share the same plugin model, the same
> interface. Add any other new storages into cloudstack will much easier and
> straightforward.
>
> 2. The storage interface between mgt server and resource is unified,
> currently there are only 5 commands send out by mgt server:
> copycommand/createobjectcommand/deletecommand/attachcommand/de
> ttachcommand, and each storage vendor can decode/encode all the
> entities(volume/snapshot/storage pool/ template etc) by its own.
>
> 3. NFS secondary storage is not explicitly depended on by other
> components. For example, when registering template into S3, template will
> be write into S3 directly, instead of storing into nfs secondary storage, then
> push to S3. If s3 is used as secondary storage, then nfs storage will be used as
> cache storage, but from other components point of view, cache storage is
> invisible. So, it's possible to make nfs storage as optional if s3 is used for
> certain hypervisors.
> The detailed FS is at
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup
> +Object+Store+Plugin+Framework
> The test we did:
>
> 1. We modified marvin to use new storage api
>
> 2. Test_volume and test_vm_life_cycle, test_template under smoke test
> folder are executed against xenserver/kvm/vmware and devcloud, some of
> them are failed, it's partly due to bugs introduced by our code, partly master
> branch itself has issue(e.g. resizevolume doesn't work). We want to fix these
> issues after merging into master.
>
> The basic follow does work: create user vm, attach/detach volume, register
> template, create template from volume/snapshot, take snapshot, create
> volume from snapshot.
> It's a huge change, around 60k LOC patch, to review the code, you can try:
> git diff master..object_store, will show all the diff.
> Comments/feedback are welcome. Thanks.
>
Re: [MERGE]object_store branch into master [Second round]
Posted by Prasanna Santhanam <ts...@apache.org>.
On Sat, Jun 15, 2013 at 03:07:39AM +0000, Edison Su wrote:
> Hi all,
> The second round, call for merge object_store branch into master, is coming!
> The issues fixed:
> 1. All the major issues addressed by John are fixed:
> 1.1 A cache storage replacement algorithm is added: StorageCacheReplacementAlgorithmLRU, based on the reference count and least recently used.
> 1.2 A new S3 transport is added, can upload > 5G template into S3 directly.
> 1.3 Retry, if S3 upload failed.
> 1.4 some comments from https://reviews.apache.org/r/11277/, mostly, the coding style, are addressed and clean up some unused code.
> 2. DB upgrade path from 4.1 to 4.2
> 3. Bug fix
>
> The size of the patch is even bigger now, around 100000 LOC, you
> can find about the patch from
> https://reviews.apache.org/r/11277/diff/2/.
> Comments/feedback are welcome. Thanks.
Awesome work - looking forward to this. Great discussions in the past
few weeks. Some of the better topic threads we've had in the recent
past!
--
Prasanna.,
------------------------
Powered by BigRock.com
Re: [MERGE]object_store branch into master [Second round]
Posted by Chip Childers <ch...@sungard.com>.
On Sat, Jun 15, 2013 at 03:07:39AM +0000, Edison Su wrote:
> Hi all,
> The second round, call for merge object_store branch into master, is coming!
> The issues fixed:
> 1. All the major issues addressed by John are fixed:
> 1.1 A cache storage replacement algorithm is added: StorageCacheReplacementAlgorithmLRU, based on the reference count and least recently used.
> 1.2 A new S3 transport is added, can upload > 5G template into S3 directly.
> 1.3 Retry, if S3 upload failed.
> 1.4 some comments from https://reviews.apache.org/r/11277/, mostly, the coding style, are addressed and clean up some unused code.
> 2. DB upgrade path from 4.1 to 4.2
> 3. Bug fix
>
> The size of the patch is even bigger now, around 100000 LOC, you can find about the patch from https://reviews.apache.org/r/11277/diff/2/.
> Comments/feedback are welcome. Thanks.
+1 to this. Great work Edison, Min and John (for the detailed reviews).
I'm really glad we spent the time to get to a consensus on the
implementation.
-chip