You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jclouds.apache.org by Ignasi Barrera <na...@apache.org> on 2014/06/10 18:00:58 UTC
Image cache
Hi!
TL;DR: Populating images in the image cache.
I'm working on a fix for JCLOUDS-588 [1] and I have a first
implementation [2]. However, before submitting a pull request I want
to validate it here.
Basically the issue is the population of the image cache. When using
the ComputeService, jclouds caches the list of images (as it can be an
expensive operation) in a memoized supplier. That cache is used by the
TemplateBuilder, the default ComputeServiceAdapter and by most of the
classes in the compute abstraction, to avoid making unnecessary
expensive calls to the provider.
There are, at least, two cases in which the use of this cache is
leading to images not being found:
* When images are created with the ImageExtension (see JCLOUDS-512 [3]).
* When providers don't return some images when listing them but a
"getById" call works (see JCLOUDS-570 [1]).
IMO, the safest approach to fix this is to populate the missing images
in the cache as soon as we find them. The challenging thing is that
the image cache is currently implemented as a memoized supplier, which
is basically a singleton immutable set, so we can't write to it or
copy it to a new instance, as there are many classes that will have it
already injected.
In the current patch I'm working on, I've created an
ImageCacheSupplier [5] that acts as a "view" of that memoized
supplier. It provides a "registerImage" method to register the
discovered images, and it returns the composition of both, the cached
and discovered ones when someone calls its "get()" method. That
ImageCacheSupplier replaces the current memoized supplier, so it is by
default injected in all providers.
And my questions are:
* We have to manually populate the images in the cache as soon as we
know they're missing. This is obvious in the template builder and the
image extension. Do you have in mind any other case or place in the
code where we should take care of populating images? (Doing this by
default can be overkill as noted in JCLOUDS-588 comments, so we'd
better identify where we might find unknown images and register them
there).
* Do you have in mind an alternate/better approach to this one? :)
Thanks!
Ignasi
[1] https://issues.apache.org/jira/browse/JCLOUDS-588
[2] https://github.com/nacx/jclouds/tree/588-imagecache
[3] https://issues.apache.org/jira/browse/JCLOUDS-512
[4] https://issues.apache.org/jira/browse/JCLOUDS-570
[5] https://github.com/nacx/jclouds/blob/588-imagecache/compute/src/main/java/org/jclouds/compute/suppliers/ImageCacheSupplier.java
Re: Image cache
Posted by Ignasi Barrera <na...@apache.org>.
No in the current versions. Please open a feature request in Jira for that
purpose so we can track it and/or open a specific thread in the mailing
list, but let's keep the focus of this thread on how to fix the mentioned
issues.
El 10/06/2014 18:49, "Inbar Stolberg" <in...@gmail.com> escribió:
> Is there any way of not using or even not reserving the memory of this
> cache?
> I am tring to reduce memory consumption to bare minimum and i dont use this
> cache at all ...
> On Jun 10, 2014 7:01 PM, "Ignasi Barrera" <na...@apache.org> wrote:
>
> > Hi!
> >
> > TL;DR: Populating images in the image cache.
> >
> > I'm working on a fix for JCLOUDS-588 [1] and I have a first
> > implementation [2]. However, before submitting a pull request I want
> > to validate it here.
> >
> > Basically the issue is the population of the image cache. When using
> > the ComputeService, jclouds caches the list of images (as it can be an
> > expensive operation) in a memoized supplier. That cache is used by the
> > TemplateBuilder, the default ComputeServiceAdapter and by most of the
> > classes in the compute abstraction, to avoid making unnecessary
> > expensive calls to the provider.
> >
> > There are, at least, two cases in which the use of this cache is
> > leading to images not being found:
> >
> > * When images are created with the ImageExtension (see JCLOUDS-512 [3]).
> > * When providers don't return some images when listing them but a
> > "getById" call works (see JCLOUDS-570 [1]).
> >
> >
> > IMO, the safest approach to fix this is to populate the missing images
> > in the cache as soon as we find them. The challenging thing is that
> > the image cache is currently implemented as a memoized supplier, which
> > is basically a singleton immutable set, so we can't write to it or
> > copy it to a new instance, as there are many classes that will have it
> > already injected.
> >
> > In the current patch I'm working on, I've created an
> > ImageCacheSupplier [5] that acts as a "view" of that memoized
> > supplier. It provides a "registerImage" method to register the
> > discovered images, and it returns the composition of both, the cached
> > and discovered ones when someone calls its "get()" method. That
> > ImageCacheSupplier replaces the current memoized supplier, so it is by
> > default injected in all providers.
> >
> >
> > And my questions are:
> >
> > * We have to manually populate the images in the cache as soon as we
> > know they're missing. This is obvious in the template builder and the
> > image extension. Do you have in mind any other case or place in the
> > code where we should take care of populating images? (Doing this by
> > default can be overkill as noted in JCLOUDS-588 comments, so we'd
> > better identify where we might find unknown images and register them
> > there).
> >
> > * Do you have in mind an alternate/better approach to this one? :)
> >
> >
> > Thanks!
> >
> > Ignasi
> >
> >
> > [1] https://issues.apache.org/jira/browse/JCLOUDS-588
> > [2] https://github.com/nacx/jclouds/tree/588-imagecache
> > [3] https://issues.apache.org/jira/browse/JCLOUDS-512
> > [4] https://issues.apache.org/jira/browse/JCLOUDS-570
> > [5]
> >
> https://github.com/nacx/jclouds/blob/588-imagecache/compute/src/main/java/org/jclouds/compute/suppliers/ImageCacheSupplier.java
> >
>
Re: Image cache
Posted by Inbar Stolberg <in...@gmail.com>.
Is there any way of not using or even not reserving the memory of this
cache?
I am tring to reduce memory consumption to bare minimum and i dont use this
cache at all ...
On Jun 10, 2014 7:01 PM, "Ignasi Barrera" <na...@apache.org> wrote:
> Hi!
>
> TL;DR: Populating images in the image cache.
>
> I'm working on a fix for JCLOUDS-588 [1] and I have a first
> implementation [2]. However, before submitting a pull request I want
> to validate it here.
>
> Basically the issue is the population of the image cache. When using
> the ComputeService, jclouds caches the list of images (as it can be an
> expensive operation) in a memoized supplier. That cache is used by the
> TemplateBuilder, the default ComputeServiceAdapter and by most of the
> classes in the compute abstraction, to avoid making unnecessary
> expensive calls to the provider.
>
> There are, at least, two cases in which the use of this cache is
> leading to images not being found:
>
> * When images are created with the ImageExtension (see JCLOUDS-512 [3]).
> * When providers don't return some images when listing them but a
> "getById" call works (see JCLOUDS-570 [1]).
>
>
> IMO, the safest approach to fix this is to populate the missing images
> in the cache as soon as we find them. The challenging thing is that
> the image cache is currently implemented as a memoized supplier, which
> is basically a singleton immutable set, so we can't write to it or
> copy it to a new instance, as there are many classes that will have it
> already injected.
>
> In the current patch I'm working on, I've created an
> ImageCacheSupplier [5] that acts as a "view" of that memoized
> supplier. It provides a "registerImage" method to register the
> discovered images, and it returns the composition of both, the cached
> and discovered ones when someone calls its "get()" method. That
> ImageCacheSupplier replaces the current memoized supplier, so it is by
> default injected in all providers.
>
>
> And my questions are:
>
> * We have to manually populate the images in the cache as soon as we
> know they're missing. This is obvious in the template builder and the
> image extension. Do you have in mind any other case or place in the
> code where we should take care of populating images? (Doing this by
> default can be overkill as noted in JCLOUDS-588 comments, so we'd
> better identify where we might find unknown images and register them
> there).
>
> * Do you have in mind an alternate/better approach to this one? :)
>
>
> Thanks!
>
> Ignasi
>
>
> [1] https://issues.apache.org/jira/browse/JCLOUDS-588
> [2] https://github.com/nacx/jclouds/tree/588-imagecache
> [3] https://issues.apache.org/jira/browse/JCLOUDS-512
> [4] https://issues.apache.org/jira/browse/JCLOUDS-570
> [5]
> https://github.com/nacx/jclouds/blob/588-imagecache/compute/src/main/java/org/jclouds/compute/suppliers/ImageCacheSupplier.java
>
Re: Image cache
Posted by Everett Toews <ev...@RACKSPACE.COM>.
On Jun 10, 2014, at 11:00 AM, Ignasi Barrera <na...@apache.org> wrote:
> * We have to manually populate the images in the cache as soon as we
> know they're missing. This is obvious in the template builder and the
> image extension. Do you have in mind any other case or place in the
> code where we should take care of populating images?
The only place that comes to mind is openstack-glance but that doesn’t interact with the cache right now and I’m not even sure we would want it to.
> (Doing this by
> default can be overkill as noted in JCLOUDS-588 comments, so we'd
> better identify where we might find unknown images and register them
> there).
>
> * Do you have in mind an alternate/better approach to this one? :)
Nope. This seems like a good approach to me.
+1!
Everett