template copy to primary

[Marcus Sorensen <sh...@gmail.com>]

Again, forgive my ignorance if I don't understand how this works for all
storage types, but it seems that when a template is first used in a cluster
we copy the template to primary storage first and use that for all
subsequent. Should we add an enhancement to be able to tag templates that
should be cached on primary vs ones that should be copied direct from
secondary storage whenever used?

 I'm thinking about scale, if I've got 20 templates that are used a lot by
everyone, then I want those cached, but if my users are making 1000
templates and only use each once or twice, I don't want them to go to my
primary storage. Especially with CLVM or another raw style storage where we
actually use the whole physical size of the template.

Re: template copy to primary

[Wido den Hollander <wi...@widodh.nl>]

On 07-11-12 19:04, Marcus Sorensen wrote:
> I agree about using plugins in general.
>
> As far as local caching, I'm not really asking about adding a feature so
> much as modifying the behavior of an existing one. It seems cloudstack
> already does this (at least from what I've seen) download template to
> primary storage.
>
> I now see where the issue is. It's downloading the template and using it as
> a backing file, but this only works for certain storage types and is not
> accounted for in some places. I'll file it as a bug and work on it, then
> the local caching thing can be added as a feature.
>

Keep in mind though that the complete storage code needs a refactor. 
This should go on the list of things to do for that.

Wido

>
> On Wed, Nov 7, 2012 at 10:37 AM, Alex Huang <Al...@citrix.com> wrote:
>
>> Marcus,
>>
>> You can add a plugin for something like that.  I'm sure you'll need to
>> make some adjustments in cloudstack code to handle this (particularly
>> specifying the storage pool the vm should be on) but we should largely keep
>> the logic outside of cloudstack code and put it in a template.
>>
>> - The plugin should implement PluggableService so it can expose APIs
>> allowing the system administrator to identify which template should be
>> prepopulate.  It might also want to take in to which storage pool.  I'll
>> leave it up to you to design that api.
>> - When called the plugin uses the deploy call to create a VM owned by the
>> system on every storage pool.  (This prevents template gc to delete the
>> template)
>> - When it is determined that the template is no longer useful in the
>> system.  A call into the API to delete the VM.  Once the VM is deleted and
>> all other VMs based on the template is deleted from the system, template gc
>> will delete the template.
>> - Add this plugin to components.xml.
>>
>> In general we should think about adding to CloudStack by utilizing
>> plugins.  There are many benefits.
>>
>> - Code will be well separated because plugins compile by themselves.
>> - Plugins can be disabled in case it has some bugs that was found in
>> production.
>> - Plugins can be separately unit tested by mocking up cloudstack
>> functionalities that it needs.
>>
>> There's been too much of this type of automation functionality pushed into
>> cloudstack orchestration.  CloudStack Orchestration needs to be kept small
>> and tight.
>>
>> --Alex
>>
>>
>> -----Original Message-----
>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> Sent: Wednesday, November 07, 2012 9:17 AM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: template copy to primary
>>
>> Again, forgive my ignorance if I don't understand how this works for all
>> storage types, but it seems that when a template is first used in a cluster
>> we copy the template to primary storage first and use that for all
>> subsequent. Should we add an enhancement to be able to tag templates that
>> should be cached on primary vs ones that should be copied direct from
>> secondary storage whenever used?
>>
>>   I'm thinking about scale, if I've got 20 templates that are used a lot by
>> everyone, then I want those cached, but if my users are making 1000
>> templates and only use each once or twice, I don't want them to go to my
>> primary storage. Especially with CLVM or another raw style storage where we
>> actually use the whole physical size of the template.
>>
>

Re: template copy to primary

[Marcus Sorensen <sh...@gmail.com>]

I agree about using plugins in general.

As far as local caching, I'm not really asking about adding a feature so
much as modifying the behavior of an existing one. It seems cloudstack
already does this (at least from what I've seen) download template to
primary storage.

I now see where the issue is. It's downloading the template and using it as
a backing file, but this only works for certain storage types and is not
accounted for in some places. I'll file it as a bug and work on it, then
the local caching thing can be added as a feature.


On Wed, Nov 7, 2012 at 10:37 AM, Alex Huang <Al...@citrix.com> wrote:

> Marcus,
>
> You can add a plugin for something like that.  I'm sure you'll need to
> make some adjustments in cloudstack code to handle this (particularly
> specifying the storage pool the vm should be on) but we should largely keep
> the logic outside of cloudstack code and put it in a template.
>
> - The plugin should implement PluggableService so it can expose APIs
> allowing the system administrator to identify which template should be
> prepopulate.  It might also want to take in to which storage pool.  I'll
> leave it up to you to design that api.
> - When called the plugin uses the deploy call to create a VM owned by the
> system on every storage pool.  (This prevents template gc to delete the
> template)
> - When it is determined that the template is no longer useful in the
> system.  A call into the API to delete the VM.  Once the VM is deleted and
> all other VMs based on the template is deleted from the system, template gc
> will delete the template.
> - Add this plugin to components.xml.
>
> In general we should think about adding to CloudStack by utilizing
> plugins.  There are many benefits.
>
> - Code will be well separated because plugins compile by themselves.
> - Plugins can be disabled in case it has some bugs that was found in
> production.
> - Plugins can be separately unit tested by mocking up cloudstack
> functionalities that it needs.
>
> There's been too much of this type of automation functionality pushed into
> cloudstack orchestration.  CloudStack Orchestration needs to be kept small
> and tight.
>
> --Alex
>
>
> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Wednesday, November 07, 2012 9:17 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: template copy to primary
>
> Again, forgive my ignorance if I don't understand how this works for all
> storage types, but it seems that when a template is first used in a cluster
> we copy the template to primary storage first and use that for all
> subsequent. Should we add an enhancement to be able to tag templates that
> should be cached on primary vs ones that should be copied direct from
> secondary storage whenever used?
>
>  I'm thinking about scale, if I've got 20 templates that are used a lot by
> everyone, then I want those cached, but if my users are making 1000
> templates and only use each once or twice, I don't want them to go to my
> primary storage. Especially with CLVM or another raw style storage where we
> actually use the whole physical size of the template.
>

RE: template copy to primary

[Alex Huang <Al...@citrix.com>]

Marcus,

You can add a plugin for something like that.  I'm sure you'll need to make some adjustments in cloudstack code to handle this (particularly specifying the storage pool the vm should be on) but we should largely keep the logic outside of cloudstack code and put it in a template.

- The plugin should implement PluggableService so it can expose APIs allowing the system administrator to identify which template should be prepopulate.  It might also want to take in to which storage pool.  I'll leave it up to you to design that api.
- When called the plugin uses the deploy call to create a VM owned by the system on every storage pool.  (This prevents template gc to delete the template)
- When it is determined that the template is no longer useful in the system.  A call into the API to delete the VM.  Once the VM is deleted and all other VMs based on the template is deleted from the system, template gc will delete the template.
- Add this plugin to components.xml.

In general we should think about adding to CloudStack by utilizing plugins.  There are many benefits.

- Code will be well separated because plugins compile by themselves.
- Plugins can be disabled in case it has some bugs that was found in production.
- Plugins can be separately unit tested by mocking up cloudstack functionalities that it needs.

There's been too much of this type of automation functionality pushed into cloudstack orchestration.  CloudStack Orchestration needs to be kept small and tight.

--Alex


-----Original Message-----
From: Marcus Sorensen [mailto:shadowsor@gmail.com] 
Sent: Wednesday, November 07, 2012 9:17 AM
To: cloudstack-dev@incubator.apache.org
Subject: template copy to primary

Again, forgive my ignorance if I don't understand how this works for all storage types, but it seems that when a template is first used in a cluster we copy the template to primary storage first and use that for all subsequent. Should we add an enhancement to be able to tag templates that should be cached on primary vs ones that should be copied direct from secondary storage whenever used?

 I'm thinking about scale, if I've got 20 templates that are used a lot by everyone, then I want those cached, but if my users are making 1000 templates and only use each once or twice, I don't want them to go to my primary storage. Especially with CLVM or another raw style storage where we actually use the whole physical size of the template.