[libcloud] [RFC] libvirt support

[Jeremy Whitlock <jc...@gmail.com>]

Hello all,
	I have a need for managing "local clouds" and I was hoping to add support to libcloud for such a thing, since I'm already using it for remote/private cloud management.  When I brought this up in IRC, someone mentioned the idea of creating a libvirt (http://libvirt.org/) driver for libcloud.  That being said, I've created a GitHub fork for that purpose (https://github.com/jcscoobyrs/libcloud).  I have committed only one revision at this time but it does add partial libvirt support such that you can connect to any of the supported libvirt hypervisors, although I've only tested VirtualBox, and list nodes with node information.  As the effort stands right now, once you successfully create a LibvirtNodeDriver (LND), you can perform the following:

* LND.list_nodes()
* LND.destroy_node(Node)
* LND.reboot_node(Node)

For an example in action:

>>> from libcloud.compute.drivers import virtlib
>>> d = virtlib.LibvirtNodeDriver('vbox')
>>> d.list_nodes()
[<Node: uuid=c11d579a204f0c41102bd2fde52ea4364856ff30, name=Ubuntu, state=2, public_ip=None, provider=Libvirt Node Provider (VirtualBox) ...>, <Node: uuid=a79ecb055026d19949dee544e532e5ca87a8f75c, name=Fedora, state=2, public_ip=None, 
provider=Libvirt Node Provider (VirtualBox) ...>]
>>> nodes = d.list_nodes()
>>> nodes[0].extra
{'os_type': 'hvm', 'ram': 1024L, 'cpus': 1, 'snapshot_names': ['Initial Working State']}
>>> nodes[0].name
'Ubuntu'
>>> nodes[0].id
'387ea2e2-997c-46a0-9d07-e183d6a1dda1'

So given the list of functions implemented above, there are a few others that need implementing to finish libvirt's initial support:

* LND.create_node()
* LND.deploy_node() (I'm not sure this one is something that I plan on doing initially.)
* Constants for making the API easier to use and more intuitive
* Helper functions for getting Libvirt/driver information
* Helper functions for easing the interaction with libvirt

That being said, when I started working on the create_node function, I ran into a problem where function signatures that libcloud publishes/expects no longer make sense in the libvirt world.  In libvirt, there is no notion of images, locations and sizes.  In libvirt, you create an XML document and that is what describes the new guest/node and its "pieces" (http://libvirt.org/format.html).  (I do have an idea about locations where it could list the currently available hypervisors that libvirt was compiled with support for but the other things do not really lend themselves to be done.)

The purpose of this email is really two fold:

1) Show my current work so others can critique/test
2) Explain the problems I'm facing with finishing this up

All of this being said, let's talk about the feasibility of libvirt support being in libcloud and if it's feasible, how might I be able to get past the initial problem of describing guests/nodes in a way that libvirt needs without completely diverting from the libcloud API.

Take care,

Jeremy Whitlock <jc...@gmail.com>
Twitter: jcscoobyrs
Website: http://www.thoughtspark.org

Re: [libcloud] [RFC] libvirt support

[Joe Miller <jo...@joeym.net>]

Hi Jeremy,

I'm coming at this with very little context since I've never used libvirt,
but your problem sounds similar to other situations that I have seen come
up.

You could implement list_images(), list_sizes(), and list_locations() as
functions that simply return some statically defined NodeImage, NodeSize,
and Location objects.   Those could then be passed into create_node().

for example, maybe you pick a few sizes and statically define them, such as:
 small, medium, large.   Or, maybe you could copy the amazon sizes.  This
came up on irc recently.  In the future maybe users could specify their own
sizes with yaml files or similar.

I imagine creating static NodeImage objects would be a little trickier since
each user may have different images available on their libvirt-managed
hypervisor.

Here are some examples of what I mean.

The IBM driver fakes list_nodes():
-
https://github.com/joemiller/libcloud/blob/opsource-driver/libcloud/compute/drivers/ibm_sbc.py#L149-158

The linode driver fakes list_nodes() as well, and it looks like it uses
drive sizes to construct 'images'.
-
https://github.com/joemiller/libcloud/blob/opsource-driver/libcloud/compute/drivers/opennebula.py

On Mon, Apr 18, 2011 at 12:15 PM, Jeremy Whitlock <jc...@gmail.com>wrote:

> Hello all,
>        I have a need for managing "local clouds" and I was hoping to add
> support to libcloud for such a thing, since I'm already using it for
> remote/private cloud management.  When I brought this up in IRC, someone
> mentioned the idea of creating a libvirt (http://libvirt.org/) driver for
> libcloud.  That being said, I've created a GitHub fork for that purpose (
> https://github.com/jcscoobyrs/libcloud).  I have committed only one
> revision at this time but it does add partial libvirt support such that you
> can connect to any of the supported libvirt hypervisors, although I've only
> tested VirtualBox, and list nodes with node information.  As the effort
> stands right now, once you successfully create a LibvirtNodeDriver (LND),
> you can perform the following:
>
> * LND.list_nodes()
> * LND.destroy_node(Node)
> * LND.reboot_node(Node)
>
> For an example in action:
>
> >>> from libcloud.compute.drivers import virtlib
> >>> d = virtlib.LibvirtNodeDriver('vbox')
> >>> d.list_nodes()
> [<Node: uuid=c11d579a204f0c41102bd2fde52ea4364856ff30, name=Ubuntu,
> state=2, public_ip=None, provider=Libvirt Node Provider (VirtualBox) ...>,
> <Node: uuid=a79ecb055026d19949dee544e532e5ca87a8f75c, name=Fedora, state=2,
> public_ip=None,
> provider=Libvirt Node Provider (VirtualBox) ...>]
> >>> nodes = d.list_nodes()
> >>> nodes[0].extra
> {'os_type': 'hvm', 'ram': 1024L, 'cpus': 1, 'snapshot_names': ['Initial
> Working State']}
> >>> nodes[0].name
> 'Ubuntu'
> >>> nodes[0].id
> '387ea2e2-997c-46a0-9d07-e183d6a1dda1'
>
> So given the list of functions implemented above, there are a few others
> that need implementing to finish libvirt's initial support:
>
> * LND.create_node()
> * LND.deploy_node() (I'm not sure this one is something that I plan on
> doing initially.)
> * Constants for making the API easier to use and more intuitive
> * Helper functions for getting Libvirt/driver information
> * Helper functions for easing the interaction with libvirt
>
> That being said, when I started working on the create_node function, I ran
> into a problem where function signatures that libcloud publishes/expects no
> longer make sense in the libvirt world.  In libvirt, there is no notion of
> images, locations and sizes.  In libvirt, you create an XML document and
> that is what describes the new guest/node and its "pieces" (
> http://libvirt.org/format.html).  (I do have an idea about locations where
> it could list the currently available hypervisors that libvirt was compiled
> with support for but the other things do not really lend themselves to be
> done.)
>
> The purpose of this email is really two fold:
>
> 1) Show my current work so others can critique/test
> 2) Explain the problems I'm facing with finishing this up
>
> All of this being said, let's talk about the feasibility of libvirt support
> being in libcloud and if it's feasible, how might I be able to get past the
> initial problem of describing guests/nodes in a way that libvirt needs
> without completely diverting from the libcloud API.
>
> Take care,
>
> Jeremy Whitlock <jc...@gmail.com>
> Twitter: jcscoobyrs
> Website: http://www.thoughtspark.org
>
>