custom ordering for TemplateBuilder images

[Alex Heneveld <al...@cloudsoftcorp.com>]

Hi folks,

I'd like to tie in to jclouds's TemplateBuilder so that I can use its 
filtering capabilities but then use custom logic to determine which of 
the matching images is "best" (for my custom value of best).

The driving use case is that I want a portable way to say "any recent 
ubuntu or centos".  Normally this is the default of course but it 
doesn't always do the right thing when other options are specified -- 
broken ubuntu "alpha" images in AWS being the worst offender (asking for 
16gb RAM in us-west-1 gives back an awful ubuntu 8.04 alpha, for 
instance!).  It would also handle use cases where someone says "Ubuntu 
11 or 12, or CentOS 6.x, is best. failing that, Ubuntu 10 or CentOS 5.x. 
(and never any alpha images!)".

I'm happy to add an issue and work on this but I wanted to check: does 
this look like a good idea?

In terms of the solution, I see two options:  TemplateBuilder could take 
an imageOrdering predicate where the builder applies my ordering 
(currently the Impl just uses a hard-coded DEFAULT_IMAGE_ORDERING); or 
we could make the existing Impl buildImagePredicate method public and 
let the user do the filtering then sorting.  Any preference?

Thanks
Alex

Re: custom ordering for TemplateBuilder images

[Andrew Phillips <ap...@qrmedia.com>]

> I'd like to tie in to jclouds's TemplateBuilder so that I can use its
> filtering capabilities but then use custom logic to determine which of
> the matching images is "best" (for my custom value of best).

Rather than adding ordering semantics, how about allowing you to add a  
"filterBy(...)" or similar predicate that would be applied to each  
image that matches the template? Or do you need the *whole list* of  
matching images in order to be able to make your decision..?

ap

Re: custom ordering for TemplateBuilder images

[Alex Heneveld <al...@cloudsoftcorp.com>]

Thanks Ignasi, Andrew.

I've opened a pull request for this at [1].  Pretty straightforward and 
hopefully others will find it useful too!

--A

[1]  https://github.com/jclouds/jclouds/pull/166



On 02/10/2013 06:05, Ignasi wrote:
> The TemplateBuilder currently allows you to customize the filters you
> want to apply to the templates, but does not allow to customize the
> "reduce" operation to get the template that is best, so I think this
> would be a nice addition.
> Personally, I like the idea of exposing a method in the builder to
> provide a custom Ordering, to keep things simple and respect the
> custom behavior.
>
> On 2 October 2013 14:59, Alex Heneveld <al...@cloudsoftcorp.com> wrote:
>> Hey Andrew
>>
>>
>>>> I'd like to tie in to jclouds's TemplateBuilder so that I can use its
>>>> filtering capabilities but then use custom logic to determine which of
>>>> the matching images is "best" (for my custom value of best).
>>
>>> Rather than adding ordering semantics, how about allowing you to add a
>>> "filterBy(...)" or similar predicate that would be applied to each
>>> image that matches the template? Or do you need the *whole list* of
>>> matching images in order to be able to make your decision..?
>>
>> There is already `TemplateBuilder.imageMatches(Predicate)`. However I think
>> we do need to whole list in order to decide which is best, i.e. what is the
>> latest version of Ubuntu or CentOS.
>>
>> I'm thinking allowing to set an `imageSorter(Ordering)`. This fits with how
>> the Impl currently works (it already uses an Ordering, you just can't change
>> it; and it stays in line with the naming convention of `Sorter` as in
>> `Ordering hardwareSorter()`.)
>>
>> Note that this doesn't change runtime from $O(N)$ to $O(N \log N)$ (or
>> worse)
>> (if that is your concern Andrew, I had to convince myself!) because we
>> aren't
>> sorting the whole list, we are just finding the max.
>>
>> Best
>> Alex
>>

Re: custom ordering for TemplateBuilder images

[Ignasi <ig...@gmail.com>]

The TemplateBuilder currently allows you to customize the filters you
want to apply to the templates, but does not allow to customize the
"reduce" operation to get the template that is best, so I think this
would be a nice addition.
Personally, I like the idea of exposing a method in the builder to
provide a custom Ordering, to keep things simple and respect the
custom behavior.

On 2 October 2013 14:59, Alex Heneveld <al...@cloudsoftcorp.com> wrote:
>
> Hey Andrew
>
>
>> > I'd like to tie in to jclouds's TemplateBuilder so that I can use its
>> > filtering capabilities but then use custom logic to determine which of
>> > the matching images is "best" (for my custom value of best).
>
>
>> Rather than adding ordering semantics, how about allowing you to add a
>> "filterBy(...)" or similar predicate that would be applied to each
>> image that matches the template? Or do you need the *whole list* of
>> matching images in order to be able to make your decision..?
>
>
> There is already `TemplateBuilder.imageMatches(Predicate)`. However I think
> we do need to whole list in order to decide which is best, i.e. what is the
> latest version of Ubuntu or CentOS.
>
> I'm thinking allowing to set an `imageSorter(Ordering)`. This fits with how
> the Impl currently works (it already uses an Ordering, you just can't change
> it; and it stays in line with the naming convention of `Sorter` as in
> `Ordering hardwareSorter()`.)
>
> Note that this doesn't change runtime from $O(N)$ to $O(N \log N)$ (or
> worse)
> (if that is your concern Andrew, I had to convince myself!) because we
> aren't
> sorting the whole list, we are just finding the max.
>
> Best
> Alex
>

Re: custom ordering for TemplateBuilder images

[Andrew Phillips <ap...@qrmedia.com>]

> Note that this doesn't change runtime from $O(N)$ to $O(N \log N)$ (or worse)
> (if that is your concern Andrew, I had to convince myself!) because we aren't
> sorting the whole list, we are just finding the max.

No, I was indeed thinking about more about the need to introduce new  
concepts. From your description and a more detailed look at the code  
[1], it looks like this is mainly a matter of exposing an existing  
implementation choice. Good to know it won't change the runtime, too.

Since ordering seems very much an internal implementation detail right  
now, do we think it makes sense to expose that and then basically  
force the implementation to always choose it's result value through a  
sorting/ordering process in future?

Or would it make sense to allow users to control the choice process  
*explicitly* by providing a Function<Iterable<? extends Image>,  
Image>. This would seems to be a little closer to the functional  
interface we're looking for here..?

ap

[1]  
https://github.com/jclouds/jclouds/blob/master/compute/src/main/java/org/jclouds/compute/domain/internal/TemplateBuilderImpl.java#L802

Re: custom ordering for TemplateBuilder images

[Alex Heneveld <al...@cloudsoftcorp.com>]

Hey Andrew

> > I'd like to tie in to jclouds's TemplateBuilder so that I can use its
> > filtering capabilities but then use custom logic to determine which of
> > the matching images is "best" (for my custom value of best).

> Rather than adding ordering semantics, how about allowing you to add a
> "filterBy(...)" or similar predicate that would be applied to each
> image that matches the template? Or do you need the *whole list* of
> matching images in order to be able to make your decision..?

There is already `TemplateBuilder.imageMatches(Predicate)`. However I think
we do need to whole list in order to decide which is best, i.e. what is the
latest version of Ubuntu or CentOS.

I'm thinking allowing to set an `imageSorter(Ordering)`. This fits with how
the Impl currently works (it already uses an Ordering, you just can't change
it; and it stays in line with the naming convention of `Sorter` as in
`Ordering hardwareSorter()`.)

Note that this doesn't change runtime from $O(N)$ to $O(N \log N)$ (or worse)
(if that is your concern Andrew, I had to convince myself!) because we aren't
sorting the whole list, we are just finding the max.

Best
Alex