[libcloud] We need a release (libcloud in Debian and Ubuntu apt repos)

[Alex Polvi <po...@cloudkick.com>]

Hello,

What things do we need to get fixed before we cut an initial release?

Some of the things on my list are:

* Consistent exceptions across the drivers. There should be a base
ProviderException that gets inherited by each provider implementation.
We also need a standard set of exceptions to handle common errors
(InvalidCredentials comes to mind).
* NodeSize should probably be renamed to NodeOptions. Size is a bit to
strict, if we start looking at things like the vCloud API -- which
allow you to choose CPUs and bandwidth.
* Node/NodeImage/NodeOption should all have UUIDs that are globally
unique across the project. This is, for example, so you can get a list
of images across multiple providers and not have the id field collide.
* Docs! We do not really have any docs right now. It has proven to be
a pain to generate them because of the zope interfaces...
* Supporting 10 distinct providers would be a nice round number to
shoot for. I know GoGrid is in flight and Eucalyptus should be easy.
That's 9, what should be the 10th?

This question is mainly motivated by libcloud appearing in Debian Sid
and Ubuntu Lucid (Thank you Soren!):

http://packages.debian.org/sid/python-libcloud
http://packages.ubuntu.com/lucid/python-libcloud

Any thoughts appreciated! Looking forward to seeing many of you this evening,

-Alex

-- 
co-founder, cloudkick.com
twitter.com/cloudkick
541 231 0624

Re: [libcloud] We need a release (libcloud in Debian and Ubuntu apt repos)

[Ian Bicking <ia...@gmail.com>]

On Thu, Dec 10, 2009 at 1:51 PM, Tom Davis <to...@dislocatedday.com> wrote:
>>
>> I'd like to see better debugging support.
>
>
> Could you elaborate on that?

When anything goes wrong, I want to see exactly what was sent and what
was received (including headers).  Sometimes even when things go
right; for instance, I learned Rackspace returns 203 Non-Authoritative
Information when you get cached responses.  Had I seen that somehow, I
would have known not to trust those responses.  (That it gives
nonauthoritative information is itself kind of stupid, and is
something they need to be pushed on, but for now it's out there.)

For successful responses, I'm not really sure.  Maybe the logging
module?  For errors, there's an exception object that can have
everything attached to it, though it may not know the entire history
of requests and responses, only the last request/response that led to
the error.  Still, that would be useful.

Something like .parse_error is a bit suspicious to me, as it can throw
away information.  In the current code the RackspaceResponse object
throws away the actual error message.  I submitted a pull request with
a fix... but that just saves the text, which is nice because it is
readable, but I want the XML to be saved too.  When something goes
wrong you can't really trust anything.

> Another concern I have is that many methods take objects, and can't take
>> something like IDs.  E.g., if you want to destroy a node, you need the node
>> object.
>
>
> I'll stop the quote there to make this point: you don't need *the* node, you
> just need something that *conforms to the interface of a node*. Implementing
> *INode* gives you this, but in your example all you need is some object with
> some attribute "id". If you want to store the IDs to save on network i/o
> then by all means, store those IDs and stick them as an *id* attribute on
> some object.

That's true.  I wish I could serialize objects to a string and parse
them from a string, without having to inspect the code to see what the
essential piece is.  Like you say, it's .id for some services, and
.id+.slug for another, etc.

I gettable/settable full_id or something like it would be nice; it
might be derived from multiple pieces.  E.g., for RimuHosting, maybe
you'd do:

def _get_full_id(self):
    return self.id + ':' + self.slug
def _set_full_id(self, value):
    self.id, self.slug = value.split(':', 1)
full_id = property(_get_full_id, _set_full_id)

I just really like strings because you can put them in nice text
configuration files.

> ID's aren't accepted because it isn't safe to assume the only necessary
> information will be an ID (e.g. RimuHosting uses a "slug" attribute). You
> can look at the code to find out what your provider(s) need and optimize
> your own usage that way. You can also feel free to instantiate your own
> NodeImage and Node objects as found in base.py; they're little more than
> dictionaries with some convenience methods.
>
> I feel it would be more confusing to provide additional ways to
> create/destroy nodes than it is for industrious developers to sidestep the
> issue by making their own barebones conforming objects.

[...]

> My own stuff:
>
>   - What about the interfaces interferes with proper documentation?
>   - Glancing through the code just now I've noticed a lot of clobbering of
>   reserved words (the global "id", "object", etc.) This sort of stuff should
>   have been cleaned up before it got into master...

I think .object doesn't really mean anything, so it's not a very good
use of a reserved word, but the id attribute is sensible, and doesn't
really clobber anything.  Plus the id() builtin isn't particularly
important.

-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://topplabs.org/civichacker

Re: [libcloud] We need a release (libcloud in Debian and Ubuntu apt repos)

[Tom Davis <to...@dislocatedday.com>]

>
> I'd like to see better debugging support.


Could you elaborate on that?

Another concern I have is that many methods take objects, and can't take
> something like IDs.  E.g., if you want to destroy a node, you need the node
> object.


I'll stop the quote there to make this point: you don't need *the* node, you
just need something that *conforms to the interface of a node*. Implementing
*INode* gives you this, but in your example all you need is some object with
some attribute "id". If you want to store the IDs to save on network i/o
then by all means, store those IDs and stick them as an *id* attribute on
some object.

ID's aren't accepted because it isn't safe to assume the only necessary
information will be an ID (e.g. RimuHosting uses a "slug" attribute). You
can look at the code to find out what your provider(s) need and optimize
your own usage that way. You can also feel free to instantiate your own
NodeImage and Node objects as found in base.py; they're little more than
dictionaries with some convenience methods.

I feel it would be more confusing to provide additional ways to
create/destroy nodes than it is for industrious developers to sidestep the
issue by making their own barebones conforming objects.

Do providers need their own exceptions at all?  I can imagine they might
> want to specialize exceptions a little, if they really have provider-unique
> errors, but generally I'd expect a common set of exceptions would work for
> everyone.


Probably not. But exceptions should be improved a lot and should certainly
include more information about the underlying request / response for those
who may need more information.

 Having a .uuid attribute on objects (which might be a property, generating
> the uuid from the id) would be less confusing.


I thought this was essentially a solved issue long ago (it was in the
initial base class as-merged in August), implementing basically what you've
suggested here (minus using a property, which I like):
http://github.com/cloudkick/libcloud/blob/master/libcloud/base.py#L33

My own stuff:

   - What about the interfaces interferes with proper documentation?
   - Glancing through the code just now I've noticed a lot of clobbering of
   reserved words (the global "id", "object", etc.) This sort of stuff should
   have been cleaned up before it got into master...
   - Proper docstrings would help a lot when it comes to showing how
   provider methods differ and would aid Sphinx (or whatever) in creating
   proper documentation. (
   http://github.com/cloudkick/libcloud/blob/master/libcloud/drivers/rimuhosting.py#L169
   )

At some point I will go through the code and find stuff to refactor. Just
haven't had the time recently :(

Thanks for all the input, Ian!


On Thu, Dec 10, 2009 at 1:56 PM, Ian Bicking <ia...@gmail.com> wrote:

> On Thu, Dec 10, 2009 at 2:28 AM, Alex Polvi <po...@cloudkick.com> wrote:
> > What things do we need to get fixed before we cut an initial release?
>
> I'd like to see better debugging support.  Maybe I'm biased because
> having just started to use libcloud I'm debugging a bunch of stuff (a
> little in libcloud, more in my own code or the upstream API).
>
> Another concern I have is that many methods take objects, and can't
> take something like IDs.  E.g., if you want to destroy a node, you
> need the node object.  To get that, you do a query.  But given an
> identifier, the underlying API doesn't need an "object" just that
> identifier.
>
> A more concrete example, actually; I want to create servers mostly
> from the same image.  I'd like to just save that image ID in a
> configuration file, and create it from there.  With the current API I
> save the image ID, do driver.list_images(), match those results
> against that ID, and then do driver.create_node().  (I could probably
> pickle the image object, but I'm just using a text configuration file
> so a pickle would be awkward.)
>
> > Some of the things on my list are:
> >
> > * Consistent exceptions across the drivers. There should be a base
> > ProviderException that gets inherited by each provider implementation.
> > We also need a standard set of exceptions to handle common errors
> > (InvalidCredentials comes to mind).
>
> Do providers need their own exceptions at all?  I can imagine they
> might want to specialize exceptions a little, if they really have
> provider-unique errors, but generally I'd expect a common set of
> exceptions would work for everyone.
>
> I'd like to see the exceptions tied more closely to the HTTP
> request/response that generated them.  For instance, for exceptions to
> have a .request and .response attribute that would contain some object
> related to the request and response.  And maybe a method like
> .debug_info() that would return a full report that could be submitted
> upstream.
>
> > * NodeSize should probably be renamed to NodeOptions. Size is a bit to
> > strict, if we start looking at things like the vCloud API -- which
> > allow you to choose CPUs and bandwidth.
>
> Size confused me a bit, but Options would be an even more confusing
> name to me.  "Plan" is the word I think of when I think of
> size/bandwidth/etc packaged up with a price, but NodePlan isn't a very
> good name at all.  That Size is a slight misnomer doesn't seem so bad,
> since at least it's fairly clear.  Options could mean all sorts of
> things, e.g., some flavoring of an Image, or that you could select
> multiple Options and together they'd form a plan, or... well, it just
> seems like an overused word.
>
> > * Node/NodeImage/NodeOption should all have UUIDs that are globally
> > unique across the project. This is, for example, so you can get a list
> > of images across multiple providers and not have the id field collide.
>
> Would this mean some kind of mangling, like an id of "1234" would
> become "rackspace:1234", and then that would get split off again when
> sending the id over the wire?  That would confuse me.  Having a .uuid
> attribute on objects (which might be a property, generating the uuid
> from the id) would be less confusing.  Also, sometimes I crosscheck
> information from libcloud against information found in a web interface
> or elsewhere, so a disconnect there might also be confusing.
>
> > * Docs! We do not really have any docs right now. It has proven to be
> > a pain to generate them because of the zope interfaces...
>
> What have you tried?  I've been using Sphinx for everything of my own,
> but I don't know if it barfs on zope.interface stuff; Sphinx mostly
> requires explicitly listing what you want documented, so if you can
> just leave out the interfaces it seems like it should work.
>
> (I'm not sure what the interfaces really add anyway, but I guess
> that's a different discussion.)
>
> --
> Ian Bicking  |  http://blog.ianbicking.org  |
> http://topplabs.org/civichacker
>

Re: [libcloud] We need a release (libcloud in Debian and Ubuntu apt repos)

[Ian Bicking <ia...@gmail.com>]

On Thu, Dec 10, 2009 at 2:28 AM, Alex Polvi <po...@cloudkick.com> wrote:
> What things do we need to get fixed before we cut an initial release?

I'd like to see better debugging support.  Maybe I'm biased because
having just started to use libcloud I'm debugging a bunch of stuff (a
little in libcloud, more in my own code or the upstream API).

Another concern I have is that many methods take objects, and can't
take something like IDs.  E.g., if you want to destroy a node, you
need the node object.  To get that, you do a query.  But given an
identifier, the underlying API doesn't need an "object" just that
identifier.

A more concrete example, actually; I want to create servers mostly
from the same image.  I'd like to just save that image ID in a
configuration file, and create it from there.  With the current API I
save the image ID, do driver.list_images(), match those results
against that ID, and then do driver.create_node().  (I could probably
pickle the image object, but I'm just using a text configuration file
so a pickle would be awkward.)

> Some of the things on my list are:
>
> * Consistent exceptions across the drivers. There should be a base
> ProviderException that gets inherited by each provider implementation.
> We also need a standard set of exceptions to handle common errors
> (InvalidCredentials comes to mind).

Do providers need their own exceptions at all?  I can imagine they
might want to specialize exceptions a little, if they really have
provider-unique errors, but generally I'd expect a common set of
exceptions would work for everyone.

I'd like to see the exceptions tied more closely to the HTTP
request/response that generated them.  For instance, for exceptions to
have a .request and .response attribute that would contain some object
related to the request and response.  And maybe a method like
.debug_info() that would return a full report that could be submitted
upstream.

> * NodeSize should probably be renamed to NodeOptions. Size is a bit to
> strict, if we start looking at things like the vCloud API -- which
> allow you to choose CPUs and bandwidth.

Size confused me a bit, but Options would be an even more confusing
name to me.  "Plan" is the word I think of when I think of
size/bandwidth/etc packaged up with a price, but NodePlan isn't a very
good name at all.  That Size is a slight misnomer doesn't seem so bad,
since at least it's fairly clear.  Options could mean all sorts of
things, e.g., some flavoring of an Image, or that you could select
multiple Options and together they'd form a plan, or... well, it just
seems like an overused word.

> * Node/NodeImage/NodeOption should all have UUIDs that are globally
> unique across the project. This is, for example, so you can get a list
> of images across multiple providers and not have the id field collide.

Would this mean some kind of mangling, like an id of "1234" would
become "rackspace:1234", and then that would get split off again when
sending the id over the wire?  That would confuse me.  Having a .uuid
attribute on objects (which might be a property, generating the uuid
from the id) would be less confusing.  Also, sometimes I crosscheck
information from libcloud against information found in a web interface
or elsewhere, so a disconnect there might also be confusing.

> * Docs! We do not really have any docs right now. It has proven to be
> a pain to generate them because of the zope interfaces...

What have you tried?  I've been using Sphinx for everything of my own,
but I don't know if it barfs on zope.interface stuff; Sphinx mostly
requires explicitly listing what you want documented, so if you can
just leave out the interfaces it seems like it should work.

(I'm not sure what the interfaces really add anyway, but I guess
that's a different discussion.)

-- 
Ian Bicking  |  http://blog.ianbicking.org  |  http://topplabs.org/civichacker

Re: [libcloud] We need a release (libcloud in Debian and Ubuntu apt repos)

[Soren Hansen <so...@ubuntu.com>]

On Thu, Dec 10, 2009 at 12:28:40AM -0800, Alex Polvi wrote:
> This question is mainly motivated by libcloud appearing in Debian Sid
> and Ubuntu Lucid (Thank you Soren!):

You're very welcome :)

While I would certainly like a release of libcloud to happen, please
don't feel pressured in any way to rush this because of my putting it in
Debian/Ubuntu. It's ready when it's ready. :)

-- 
Soren Hansen                 | 
Lead virtualisation engineer | Ubuntu Server Team
Canonical Ltd.               | http://www.ubuntu.com/