[libcloud] SoftLayer driver

[Henri Kivelä <he...@iki.fi>]

Hello,

I've been working on my own branch at 

http://github.com/hkivela/libcloud

and exactly with

http://github.com/hkivela/libcloud/blob/trunk/libcloud/drivers/softlayer.py

First some notes on SL API with Python.

SoftLayer has SOAP and XML-RPC interfaces. I went trough the pain to try ZSI, soappy and SUDS. None of them works for creating virtual guests (ordering part of the API). ZSI and soappy - don't remember the exact problems. SUDS has problems with performance (SL returns WSDL/XSD which can be close to 1MB, processing takes seconds), and then with multi-refs when parsing/building order templates.

XML-RPC API works, but there was no support for the complex data types required for ordering. SL fixed this quickly when requested, however.

The API not only covers virtual guests, but pretty much everything. It's quite powerful. Creating "instances" has to be done by constructing a complex datatype including varying product items. There are no ready "instance types" to use. Existing examples were getting a template out from a running node, modifying it and then using it for ordering. That template call returns a lot of data, but I was able to get just the required parts out of there and that's now in the code as an example template. Building new templates takes time, should make a tool for it. Available packages and items can be digged out using the API.

With CloudLayer, which is interesting when it comes to libcloud, there are two different parts of API (at least): for hardware and virtual guests. Hardware here is the Bare Metal servers which are bare metal, not virtualized at all like the name says. Those can be ordered on hourly basis, so I decided to include it in the implementation, there is need for it. Does someone disagree with libcloud having this support...

Because of the data structures for hardware and virtual guests have differences, destroy_node and other functions have to take this into account. The implementation is there, could be done neater in proper OO way but works now.

What is missing:

- Support for ordering bare metal instances. I've been working on this with SL support, hope to get it done soon. It's a bit more complicated since there is no function to get a ready template out of the API.
- create_node function parameters may need to be changed after this bare metal thing works ... need to think how to do it so that it's easy to understand and use. Order quantity is missing from there.
- Testing cases. The MockHttp way cannot be used, since xmlrpclib is used. Maybe something that uses dicts as fixtures.

Some notes:

After ordering, some data may be missing or not final. This is because of the provisioning process. For example, IP addresses may be missing. Bare metal instances hourly billing flag stays False until certain point in the provisioning process, even if you ordered with that flag on. Cannot do much about it, just to take it into account with the implementation. Also the nodes will not show there instantly just after ordering.

After all, it's quite close. Please have a look and tell if this could be merged to apache/libcloud, what needs to be done and changed. At least you can take some code/ideas from there if nothing else ;)

Re: [libcloud] SoftLayer driver

[Jeremiah Orem <or...@oremj.com>]

On Mar 18, 2010, at 1:18 PM, Henri Kivelä wrote:
> 
> SoftLayer has SOAP and XML-RPC interfaces. I went trough the pain to try ZSI, soappy and SUDS. None of them works for creating virtual guests (ordering part of the API). ZSI and soappy - don't remember the exact problems. SUDS has problems with performance (SL returns WSDL/XSD which can be close to 1MB, processing takes seconds), and then with multi-refs when parsing/building order templates.
> 
> XML-RPC API works, but there was no support for the complex data types required for ordering. SL fixed this quickly when requested, however.
I think xmlrpclib is the way to go, since it won't add any third party dependencies.

> 
> The API not only covers virtual guests, but pretty much everything. It's quite powerful. Creating "instances" has to be done by constructing a complex datatype including varying product items. There are no ready "instance types" to use. Existing examples were getting a template out from a running node, modifying it and then using it for ordering. That template call returns a lot of data, but I was able to get just the required parts out of there and that's now in the code as an example template. Building new templates takes time, should make a tool for it. Available packages and items can be digged out using the API.
> 
> With CloudLayer, which is interesting when it comes to libcloud, there are two different parts of API (at least): for hardware and virtual guests. Hardware here is the Bare Metal servers which are bare metal, not virtualized at all like the name says. Those can be ordered on hourly basis, so I decided to include it in the implementation, there is need for it. Does someone disagree with libcloud having this support...
> 
> Because of the data structures for hardware and virtual guests have differences, destroy_node and other functions have to take this into account. The implementation is there, could be done neater in proper OO way but works now.
We should probably subclass SoftLayerDriver as SoftLayerBareMetalDriver?

> 
> What is missing:
> 
> - Support for ordering bare metal instances. I've been working on this with SL support, hope to get it done soon. It's a bit more complicated since there is no function to get a ready template out of the API.
> - create_node function parameters may need to be changed after this bare metal thing works ... need to think how to do it so that it's easy to understand and use. Order quantity is missing from there.
> - Testing cases. The MockHttp way cannot be used, since xmlrpclib is used. Maybe something that uses dicts as fixtures.
Last night I checked some test cases in, take a look when you have a chance.

> 
> After all, it's quite close. Please have a look and tell if this could be merged to apache/libcloud, what needs to be done and changed. At least you can take some code/ideas from there if nothing else ;)

I'll poke at your code and merge some of it in tonight or tomorrow. 

Thanks!

Jeremy