Kill the half-baked extensions

[Adrian Cole <ad...@gmail.com>]

Hi, team.

The following extensions absorb a lot of code and have questionable value.

http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/extensions/SecurityGroupExtension.html

http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/extensions/ImageExtension.html

The image extension launches a complex task to construct and image
given an instance. Back when I recall testing this, it could easily
take 15 minutes, before failing with a massive stack trace.

The security groups extension is based heavily on the EC2 classic
concept of security groups, which are not related to modern status
quo, where more sophisticated firewall-type rules are present,
complicated by multiple networks.

Rather than carry the burden and reputation risk of continuing to
review and/or try to make these extensions pass tests, I recommend
calling bankruptcy and starting over.

I have no idea why @Beta is not on these classes, so we should
deprecate them straight away, and leave the extensions absent from any
labs providers we promote.

If you know of customer use-cases which depend on these classes,
please make them known, because at least in the case of
ImageExtension, I have serious doubts something implemented in this
way isn't just a timebomb waiting for the caller.

We should focus more on portability and less on shotgun magic. The
next Image extension should make it easy to represent what an image
api can do, not attempt to perform a multi-step workflow that is
almost guaranteed to fail for a measurable percentage of callers.

-A

Re: Kill the half-baked extensions

[Adrian Cole <ad...@gmail.com>]

Ok, Chris.

So thanks again for the feedback. ImageExtension has moved from kill,
then do better, to do better, then deprecate :)

Are you using the SecurityGroupExtension, too?

I'll start a discussion on the user list shortly, gathering feedback
on what aspects are most important, what aren't, and what's missing.
Hope you don't mind summarizing when that happens!

Till then, thanks and good night!
-A



On Wed, Nov 5, 2014 at 6:40 PM, Dancy, Chris <Ch...@pega.com> wrote:
> Though I will say the length of time it takes to create snapshots through
> jclouds, which indeed can be 15 minutes or so, does also happen when I
> initiate a create-snapshot through Horizon. So while the code may be
> overly complex and unnecessary the wall-clock time for taking snapshots
> seems to be an OpenStack issue and not indicative of something going
> sideways in jclouds.
>
> On 11/5/14, 4:59 PM, "Adrian Cole" <ad...@gmail.com> wrote:
>
>>Sounds like the essential use case you need is to create/delete
>>snapshots. Image (similar to security groups) sometimes conflates
>>image with snapshot, and for example what part of the node are you
>>persisting (I suppose the boot disk). If you can list out the use
>>cases you need, we can make sure that the next swing of the axe does
>>at least as good, if not better at that!
>>
>>-A
>>
>>On Wed, Nov 5, 2014 at 1:49 PM, Dancy, Chris <Ch...@pega.com> wrote:
>>> I hear you on the confusion as it was not immediately intuitive as to
>>> which Api I had to use in order to create/delete snapshots. Thankfully
>>>the
>>> devs on IRC, notably @ccustine, took what I was doing on nova command
>>>line
>>> and was able to map it to this Api for me to use.
>>>
>>> I’m all for performance improvements and as the maintainer of our ‘cloud
>>> utils’ library here in-house I can make whatever changes you guys
>>>propose
>>> and/or would have coming down the pipe at any point in the future w/out
>>> too much fuss.
>>>
>>> On 11/5/14, 4:26 PM, "Adrian Cole" <ad...@gmail.com> wrote:
>>>
>>>>Thanks for piping up, Chris.
>>>>
>>>>Besides the design being very indirect (ex ImageTemplate is an
>>>>interface with just name with "interesting" subtypes), the biggest
>>>>flaw of that api is that it returns a future as opposed to a status.
>>>>This makes it very fragile and keeps jclouds responsible for tracking
>>>>it for a long time.
>>>>
>>>>A better design would be to help invoke the process and keep status as
>>>>a conversion routine. That way if jclouds crashed, you'd not be
>>>>15minutes in etc.
>>>>
>>>>Make sense?
>>>>-A
>>>>
>>>>On Wed, Nov 5, 2014 at 1:21 PM, Dancy, Chris <Ch...@pega.com>
>>>>wrote:
>>>>> User/occasional-code-contributer here. We¹ve just recently started
>>>>>using
>>>>> the ImageExtension API, via
>>>>> computeServiceContext.getComputeService().getImageExtension(), to
>>>>>create
>>>>> and delete snapshots. Not sure if there is a better or more official
>>>>>way
>>>>> of doing this but if so we¹d love to hear it. Yes it can take some
>>>>>time
>>>>>to
>>>>> complete but has yet to fail for us (we¹re using RedHat OpenStack)
>>>>>though
>>>>> admittedly our workload in this area is quite small.
>>>>>
>>>>> On 11/5/14, 3:56 PM, "Adrian Cole" <ad...@gmail.com> wrote:
>>>>>
>>>>>>Hi, team.
>>>>>>
>>>>>>The following extensions absorb a lot of code and have questionable
>>>>>>value.
>>>>>>
>>>>>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/
>>>>>>ex
>>>>>>te
>>>>>>nsions/SecurityGroupExtension.html
>>>>>>
>>>>>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/
>>>>>>ex
>>>>>>te
>>>>>>nsions/ImageExtension.html
>>>>>>
>>>>>>The image extension launches a complex task to construct and image
>>>>>>given an instance. Back when I recall testing this, it could easily
>>>>>>take 15 minutes, before failing with a massive stack trace.
>>>>>>
>>>>>>The security groups extension is based heavily on the EC2 classic
>>>>>>concept of security groups, which are not related to modern status
>>>>>>quo, where more sophisticated firewall-type rules are present,
>>>>>>complicated by multiple networks.
>>>>>>
>>>>>>Rather than carry the burden and reputation risk of continuing to
>>>>>>review and/or try to make these extensions pass tests, I recommend
>>>>>>calling bankruptcy and starting over.
>>>>>>
>>>>>>I have no idea why @Beta is not on these classes, so we should
>>>>>>deprecate them straight away, and leave the extensions absent from any
>>>>>>labs providers we promote.
>>>>>>
>>>>>>If you know of customer use-cases which depend on these classes,
>>>>>>please make them known, because at least in the case of
>>>>>>ImageExtension, I have serious doubts something implemented in this
>>>>>>way isn't just a timebomb waiting for the caller.
>>>>>>
>>>>>>We should focus more on portability and less on shotgun magic. The
>>>>>>next Image extension should make it easy to represent what an image
>>>>>>api can do, not attempt to perform a multi-step workflow that is
>>>>>>almost guaranteed to fail for a measurable percentage of callers.
>>>>>>
>>>>>>-A
>>>>>
>>>
>

Re: Kill the half-baked extensions

["Dancy, Chris" <Ch...@pega.com>]

Though I will say the length of time it takes to create snapshots through
jclouds, which indeed can be 15 minutes or so, does also happen when I
initiate a create-snapshot through Horizon. So while the code may be
overly complex and unnecessary the wall-clock time for taking snapshots
seems to be an OpenStack issue and not indicative of something going
sideways in jclouds.

On 11/5/14, 4:59 PM, "Adrian Cole" <ad...@gmail.com> wrote:

>Sounds like the essential use case you need is to create/delete
>snapshots. Image (similar to security groups) sometimes conflates
>image with snapshot, and for example what part of the node are you
>persisting (I suppose the boot disk). If you can list out the use
>cases you need, we can make sure that the next swing of the axe does
>at least as good, if not better at that!
>
>-A
>
>On Wed, Nov 5, 2014 at 1:49 PM, Dancy, Chris <Ch...@pega.com> wrote:
>> I hear you on the confusion as it was not immediately intuitive as to
>> which Api I had to use in order to create/delete snapshots. Thankfully
>>the
>> devs on IRC, notably @ccustine, took what I was doing on nova command
>>line
>> and was able to map it to this Api for me to use.
>>
>> I’m all for performance improvements and as the maintainer of our ‘cloud
>> utils’ library here in-house I can make whatever changes you guys
>>propose
>> and/or would have coming down the pipe at any point in the future w/out
>> too much fuss.
>>
>> On 11/5/14, 4:26 PM, "Adrian Cole" <ad...@gmail.com> wrote:
>>
>>>Thanks for piping up, Chris.
>>>
>>>Besides the design being very indirect (ex ImageTemplate is an
>>>interface with just name with "interesting" subtypes), the biggest
>>>flaw of that api is that it returns a future as opposed to a status.
>>>This makes it very fragile and keeps jclouds responsible for tracking
>>>it for a long time.
>>>
>>>A better design would be to help invoke the process and keep status as
>>>a conversion routine. That way if jclouds crashed, you'd not be
>>>15minutes in etc.
>>>
>>>Make sense?
>>>-A
>>>
>>>On Wed, Nov 5, 2014 at 1:21 PM, Dancy, Chris <Ch...@pega.com>
>>>wrote:
>>>> User/occasional-code-contributer here. We¹ve just recently started
>>>>using
>>>> the ImageExtension API, via
>>>> computeServiceContext.getComputeService().getImageExtension(), to
>>>>create
>>>> and delete snapshots. Not sure if there is a better or more official
>>>>way
>>>> of doing this but if so we¹d love to hear it. Yes it can take some
>>>>time
>>>>to
>>>> complete but has yet to fail for us (we¹re using RedHat OpenStack)
>>>>though
>>>> admittedly our workload in this area is quite small.
>>>>
>>>> On 11/5/14, 3:56 PM, "Adrian Cole" <ad...@gmail.com> wrote:
>>>>
>>>>>Hi, team.
>>>>>
>>>>>The following extensions absorb a lot of code and have questionable
>>>>>value.
>>>>>
>>>>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/
>>>>>ex
>>>>>te
>>>>>nsions/SecurityGroupExtension.html
>>>>>
>>>>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/
>>>>>ex
>>>>>te
>>>>>nsions/ImageExtension.html
>>>>>
>>>>>The image extension launches a complex task to construct and image
>>>>>given an instance. Back when I recall testing this, it could easily
>>>>>take 15 minutes, before failing with a massive stack trace.
>>>>>
>>>>>The security groups extension is based heavily on the EC2 classic
>>>>>concept of security groups, which are not related to modern status
>>>>>quo, where more sophisticated firewall-type rules are present,
>>>>>complicated by multiple networks.
>>>>>
>>>>>Rather than carry the burden and reputation risk of continuing to
>>>>>review and/or try to make these extensions pass tests, I recommend
>>>>>calling bankruptcy and starting over.
>>>>>
>>>>>I have no idea why @Beta is not on these classes, so we should
>>>>>deprecate them straight away, and leave the extensions absent from any
>>>>>labs providers we promote.
>>>>>
>>>>>If you know of customer use-cases which depend on these classes,
>>>>>please make them known, because at least in the case of
>>>>>ImageExtension, I have serious doubts something implemented in this
>>>>>way isn't just a timebomb waiting for the caller.
>>>>>
>>>>>We should focus more on portability and less on shotgun magic. The
>>>>>next Image extension should make it easy to represent what an image
>>>>>api can do, not attempt to perform a multi-step workflow that is
>>>>>almost guaranteed to fail for a measurable percentage of callers.
>>>>>
>>>>>-A
>>>>
>>

Re: Kill the half-baked extensions

[Adrian Cole <ad...@gmail.com>]

Sounds like the essential use case you need is to create/delete
snapshots. Image (similar to security groups) sometimes conflates
image with snapshot, and for example what part of the node are you
persisting (I suppose the boot disk). If you can list out the use
cases you need, we can make sure that the next swing of the axe does
at least as good, if not better at that!

-A

On Wed, Nov 5, 2014 at 1:49 PM, Dancy, Chris <Ch...@pega.com> wrote:
> I hear you on the confusion as it was not immediately intuitive as to
> which Api I had to use in order to create/delete snapshots. Thankfully the
> devs on IRC, notably @ccustine, took what I was doing on nova command line
> and was able to map it to this Api for me to use.
>
> I’m all for performance improvements and as the maintainer of our ‘cloud
> utils’ library here in-house I can make whatever changes you guys propose
> and/or would have coming down the pipe at any point in the future w/out
> too much fuss.
>
> On 11/5/14, 4:26 PM, "Adrian Cole" <ad...@gmail.com> wrote:
>
>>Thanks for piping up, Chris.
>>
>>Besides the design being very indirect (ex ImageTemplate is an
>>interface with just name with "interesting" subtypes), the biggest
>>flaw of that api is that it returns a future as opposed to a status.
>>This makes it very fragile and keeps jclouds responsible for tracking
>>it for a long time.
>>
>>A better design would be to help invoke the process and keep status as
>>a conversion routine. That way if jclouds crashed, you'd not be
>>15minutes in etc.
>>
>>Make sense?
>>-A
>>
>>On Wed, Nov 5, 2014 at 1:21 PM, Dancy, Chris <Ch...@pega.com> wrote:
>>> User/occasional-code-contributer here. We¹ve just recently started using
>>> the ImageExtension API, via
>>> computeServiceContext.getComputeService().getImageExtension(), to create
>>> and delete snapshots. Not sure if there is a better or more official way
>>> of doing this but if so we¹d love to hear it. Yes it can take some time
>>>to
>>> complete but has yet to fail for us (we¹re using RedHat OpenStack)
>>>though
>>> admittedly our workload in this area is quite small.
>>>
>>> On 11/5/14, 3:56 PM, "Adrian Cole" <ad...@gmail.com> wrote:
>>>
>>>>Hi, team.
>>>>
>>>>The following extensions absorb a lot of code and have questionable
>>>>value.
>>>>
>>>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/ex
>>>>te
>>>>nsions/SecurityGroupExtension.html
>>>>
>>>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/ex
>>>>te
>>>>nsions/ImageExtension.html
>>>>
>>>>The image extension launches a complex task to construct and image
>>>>given an instance. Back when I recall testing this, it could easily
>>>>take 15 minutes, before failing with a massive stack trace.
>>>>
>>>>The security groups extension is based heavily on the EC2 classic
>>>>concept of security groups, which are not related to modern status
>>>>quo, where more sophisticated firewall-type rules are present,
>>>>complicated by multiple networks.
>>>>
>>>>Rather than carry the burden and reputation risk of continuing to
>>>>review and/or try to make these extensions pass tests, I recommend
>>>>calling bankruptcy and starting over.
>>>>
>>>>I have no idea why @Beta is not on these classes, so we should
>>>>deprecate them straight away, and leave the extensions absent from any
>>>>labs providers we promote.
>>>>
>>>>If you know of customer use-cases which depend on these classes,
>>>>please make them known, because at least in the case of
>>>>ImageExtension, I have serious doubts something implemented in this
>>>>way isn't just a timebomb waiting for the caller.
>>>>
>>>>We should focus more on portability and less on shotgun magic. The
>>>>next Image extension should make it easy to represent what an image
>>>>api can do, not attempt to perform a multi-step workflow that is
>>>>almost guaranteed to fail for a measurable percentage of callers.
>>>>
>>>>-A
>>>
>

Re: Kill the half-baked extensions

["Dancy, Chris" <Ch...@pega.com>]

I hear you on the confusion as it was not immediately intuitive as to
which Api I had to use in order to create/delete snapshots. Thankfully the
devs on IRC, notably @ccustine, took what I was doing on nova command line
and was able to map it to this Api for me to use.

I’m all for performance improvements and as the maintainer of our ‘cloud
utils’ library here in-house I can make whatever changes you guys propose
and/or would have coming down the pipe at any point in the future w/out
too much fuss.

On 11/5/14, 4:26 PM, "Adrian Cole" <ad...@gmail.com> wrote:

>Thanks for piping up, Chris.
>
>Besides the design being very indirect (ex ImageTemplate is an
>interface with just name with "interesting" subtypes), the biggest
>flaw of that api is that it returns a future as opposed to a status.
>This makes it very fragile and keeps jclouds responsible for tracking
>it for a long time.
>
>A better design would be to help invoke the process and keep status as
>a conversion routine. That way if jclouds crashed, you'd not be
>15minutes in etc.
>
>Make sense?
>-A
>
>On Wed, Nov 5, 2014 at 1:21 PM, Dancy, Chris <Ch...@pega.com> wrote:
>> User/occasional-code-contributer here. We¹ve just recently started using
>> the ImageExtension API, via
>> computeServiceContext.getComputeService().getImageExtension(), to create
>> and delete snapshots. Not sure if there is a better or more official way
>> of doing this but if so we¹d love to hear it. Yes it can take some time
>>to
>> complete but has yet to fail for us (we¹re using RedHat OpenStack)
>>though
>> admittedly our workload in this area is quite small.
>>
>> On 11/5/14, 3:56 PM, "Adrian Cole" <ad...@gmail.com> wrote:
>>
>>>Hi, team.
>>>
>>>The following extensions absorb a lot of code and have questionable
>>>value.
>>>
>>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/ex
>>>te
>>>nsions/SecurityGroupExtension.html
>>>
>>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/ex
>>>te
>>>nsions/ImageExtension.html
>>>
>>>The image extension launches a complex task to construct and image
>>>given an instance. Back when I recall testing this, it could easily
>>>take 15 minutes, before failing with a massive stack trace.
>>>
>>>The security groups extension is based heavily on the EC2 classic
>>>concept of security groups, which are not related to modern status
>>>quo, where more sophisticated firewall-type rules are present,
>>>complicated by multiple networks.
>>>
>>>Rather than carry the burden and reputation risk of continuing to
>>>review and/or try to make these extensions pass tests, I recommend
>>>calling bankruptcy and starting over.
>>>
>>>I have no idea why @Beta is not on these classes, so we should
>>>deprecate them straight away, and leave the extensions absent from any
>>>labs providers we promote.
>>>
>>>If you know of customer use-cases which depend on these classes,
>>>please make them known, because at least in the case of
>>>ImageExtension, I have serious doubts something implemented in this
>>>way isn't just a timebomb waiting for the caller.
>>>
>>>We should focus more on portability and less on shotgun magic. The
>>>next Image extension should make it easy to represent what an image
>>>api can do, not attempt to perform a multi-step workflow that is
>>>almost guaranteed to fail for a measurable percentage of callers.
>>>
>>>-A
>>

Re: Kill the half-baked extensions

[Adrian Cole <ad...@gmail.com>]

Thanks for piping up, Chris.

Besides the design being very indirect (ex ImageTemplate is an
interface with just name with "interesting" subtypes), the biggest
flaw of that api is that it returns a future as opposed to a status.
This makes it very fragile and keeps jclouds responsible for tracking
it for a long time.

A better design would be to help invoke the process and keep status as
a conversion routine. That way if jclouds crashed, you'd not be
15minutes in etc.

Make sense?
-A

On Wed, Nov 5, 2014 at 1:21 PM, Dancy, Chris <Ch...@pega.com> wrote:
> User/occasional-code-contributer here. We¹ve just recently started using
> the ImageExtension API, via
> computeServiceContext.getComputeService().getImageExtension(), to create
> and delete snapshots. Not sure if there is a better or more official way
> of doing this but if so we¹d love to hear it. Yes it can take some time to
> complete but has yet to fail for us (we¹re using RedHat OpenStack) though
> admittedly our workload in this area is quite small.
>
> On 11/5/14, 3:56 PM, "Adrian Cole" <ad...@gmail.com> wrote:
>
>>Hi, team.
>>
>>The following extensions absorb a lot of code and have questionable value.
>>
>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/exte
>>nsions/SecurityGroupExtension.html
>>
>>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/exte
>>nsions/ImageExtension.html
>>
>>The image extension launches a complex task to construct and image
>>given an instance. Back when I recall testing this, it could easily
>>take 15 minutes, before failing with a massive stack trace.
>>
>>The security groups extension is based heavily on the EC2 classic
>>concept of security groups, which are not related to modern status
>>quo, where more sophisticated firewall-type rules are present,
>>complicated by multiple networks.
>>
>>Rather than carry the burden and reputation risk of continuing to
>>review and/or try to make these extensions pass tests, I recommend
>>calling bankruptcy and starting over.
>>
>>I have no idea why @Beta is not on these classes, so we should
>>deprecate them straight away, and leave the extensions absent from any
>>labs providers we promote.
>>
>>If you know of customer use-cases which depend on these classes,
>>please make them known, because at least in the case of
>>ImageExtension, I have serious doubts something implemented in this
>>way isn't just a timebomb waiting for the caller.
>>
>>We should focus more on portability and less on shotgun magic. The
>>next Image extension should make it easy to represent what an image
>>api can do, not attempt to perform a multi-step workflow that is
>>almost guaranteed to fail for a measurable percentage of callers.
>>
>>-A
>

Re: Kill the half-baked extensions

["Dancy, Chris" <Ch...@pega.com>]

User/occasional-code-contributer here. We¹ve just recently started using
the ImageExtension API, via
computeServiceContext.getComputeService().getImageExtension(), to create
and delete snapshots. Not sure if there is a better or more official way
of doing this but if so we¹d love to hear it. Yes it can take some time to
complete but has yet to fail for us (we¹re using RedHat OpenStack) though
admittedly our workload in this area is quite small.

On 11/5/14, 3:56 PM, "Adrian Cole" <ad...@gmail.com> wrote:

>Hi, team.
>
>The following extensions absorb a lot of code and have questionable value.
>
>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/exte
>nsions/SecurityGroupExtension.html
>
>http://jclouds.apache.org/reference/javadoc/1.8.x/org/jclouds/compute/exte
>nsions/ImageExtension.html
>
>The image extension launches a complex task to construct and image
>given an instance. Back when I recall testing this, it could easily
>take 15 minutes, before failing with a massive stack trace.
>
>The security groups extension is based heavily on the EC2 classic
>concept of security groups, which are not related to modern status
>quo, where more sophisticated firewall-type rules are present,
>complicated by multiple networks.
>
>Rather than carry the burden and reputation risk of continuing to
>review and/or try to make these extensions pass tests, I recommend
>calling bankruptcy and starting over.
>
>I have no idea why @Beta is not on these classes, so we should
>deprecate them straight away, and leave the extensions absent from any
>labs providers we promote.
>
>If you know of customer use-cases which depend on these classes,
>please make them known, because at least in the case of
>ImageExtension, I have serious doubts something implemented in this
>way isn't just a timebomb waiting for the caller.
>
>We should focus more on portability and less on shotgun magic. The
>next Image extension should make it easy to represent what an image
>api can do, not attempt to perform a multi-step workflow that is
>almost guaranteed to fail for a measurable percentage of callers.
>
>-A