You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@libcloud.apache.org by Paul Querna <pa...@querna.org> on 2010/03/01 02:15:13 UTC
[libcloud] need feedback on deployment API in trunk!
Hello Everyone,
I've taken a stab at a basic deployment API, and its in libcloud trunk now.
An example of using it:
from libcloud.providers import get_driver
from libcloud.deployment import MultiStepDeployment, ScriptDeployment,
SSHKeyDeployment
driver = get_driver(Provider.RACKSPACE)('libcloud', 'you secret here')
loc = driver.list_locations()
sizes = driver.list_sizes()
images = driver.list_images()
key = open(os.path.expanduser("~/.ssh/my_rsa.pub")).read()
sd = SSHKeyDeployment(key)
script = ScriptDeployment("apt-get install -y puppet")
msd = MultiStepDeployment([sd, script])
node = driver.deploy_node(name="lc-test.k1k.me", location=loc[0],
image=images[0],
size=sizes[0], deploy=msd)
So, basically there is a new deploy_node() method on the base driver
class. If a driver supports either giving us the nodes password, or
setting one, it logs in using Paramiko, and can do basic setup of the
node. Supporting SSH Keys for places like amazon shouldn't be too
hard to extend in here quickly either!
Thoughts on the API? Any major flaws or suggestions on how to make it
easier and more robust?
Thanks,
Paul
Re: [libcloud] need feedback on deployment API in trunk!
Posted by Paul Querna <pa...@querna.org>.
On Sun, Feb 28, 2010 at 7:04 PM, Tom White <to...@cloudera.com> wrote:
> Hi Paul,
>
> This looks great! One thing I noticed is that paramiko is LGPL'd which
> may not be used with Apache products
> (http://www.apache.org/legal/resolved.html#category-x), so we'll need
> to use something else, unfortunately.
I've converted it in trunk to be an optional plugin with a generic
wrapper class, that could be powered by just about any SSH/SFTP
implementation:
<https://svn.apache.org/repos/asf/incubator/libcloud/trunk/libcloud/ssh.py>
If anyone has time, it would be great to flush out the
ShellOutSSHClient (or rename it), so that you can just use the command
line openssh if that is useful to you.
I think it might also be good if we could find or make a python
wrapper around libssh2, because libssh2 is under the BSD license.
Thanks,
Paul
Re: [libcloud] need feedback on deployment API in trunk!
Posted by Adrian Cole <fe...@gmail.com>.
I agree with the implementation as described. It reflects our cloudhackers
discussion a week or so ago: basically, be able to bootstrap something else
;)
WRT error handling,
+1 Optionally destroy a node who fails to successfully execute bootstrap
commands.
By default, don't, which implies the user must check to see if the node
state is correct.
Future work could allow custom error handlers to be specified by the user.
For example, one could simply "re-run" the script or whatever.
Anyway, good progress: it looks clean.
-Adrian
founder jclouds/cloudhackers
http://www.meetup.com/cloudhackers-SF
On Sun, Feb 28, 2010 at 9:58 PM, Paul Querna <pa...@querna.org> wrote:
> On Sun, Feb 28, 2010 at 9:54 PM, Ian Bicking <ia...@gmail.com> wrote:
> > On Sun, Feb 28, 2010 at 11:37 PM, Paul Querna <pa...@querna.org> wrote:
> >
> >> > I'm a little wary of paramiko, mostly because of the pycrypto
> >> > requirement, which itself require compiling stuff. It's not terrible,
> >> > but it introduces a potential installation problem that I was hoping
> >> > to avoid (by using pure Python libraries). OTOH, I'm on the fence
> >> > about using Fabric for the same reason, and if libcloud uses paramiko
> >> > then I'd probably not be on the fence and just start using Fabric too
> >> > ;) (Currently I'm using ssh calls via subprocess.)
> >>
> >> I am not aware of any pure python SSH clients, other than Twisted
> >> Conch, which would mean... porting to twisted (and depending on
> >> twisted!)
> >>
> >
> > Certainly not what I'm asking for ;) ssh via subprocess works fine on
> any
> > Posixy system, but of course is more tricky on Windows. (try/except
> around
> > the paramiko import would work for me though)
> >
> >
> >> > Does deploy_node do both a create, and then run the deployment? What
> >> > kind of error checking does it have? E.g., if that apt-get fails,
> >> > what then? Does Fabric already accomplish the same things with
> >> > greater depth?
> >>
> >> It both creates the node, and does the initial connection. It has
> >> very little error handling right now. I definitely would like to
> >> improve this though -- this is just a first draft :-)
> >>
> >
> > Upon further thought, and given the use cases for my tool (
> > http://toppcloud.colorstudy.com -- soon to be renamed Silver Lining once
> I
> > finish up the text substitution), I can't really envision using this code
> as
> > an underlying layer. Once the system is setup and ssh is accessible I
> think
> > it should be outside libcloud's scope. At that point you aren't
> abstracting
> > the cloud, you are handling concrete systems, and other abstraction
> layers
> > (like Puppet, Fabric, etc) are more appropriate. What *would* be helpful
> is
> > getting that initial ssh connection working, or some kind of abstraction
> > around that. Right now on Rackspace Cloud I'm using their files
> extension
> > to upload authorized_keys, but I'd have to switch things around some to
> get
> > the same process working on EC2. So... abstracting that difference would
> be
> > nice. But after that, eh.
>
> Yes, exactly, that is my only goal with this API.
>
> Convert various password generated, password assigned, ssh key via
> create_node, ssh key via metadata script, or backed in ssh key -> you
> install an ssh key of your choice.
>
> Once there, its no longer libcloud's problem imo, and going beyond
> just the tools to get a consistent way to get access to the machine is
> not something I am interested in doing, so I think we are on the same
> page.
>
Re: [libcloud] need feedback on deployment API in trunk!
Posted by Paul Querna <pa...@querna.org>.
On Sun, Feb 28, 2010 at 9:54 PM, Ian Bicking <ia...@gmail.com> wrote:
> On Sun, Feb 28, 2010 at 11:37 PM, Paul Querna <pa...@querna.org> wrote:
>
>> > I'm a little wary of paramiko, mostly because of the pycrypto
>> > requirement, which itself require compiling stuff. It's not terrible,
>> > but it introduces a potential installation problem that I was hoping
>> > to avoid (by using pure Python libraries). OTOH, I'm on the fence
>> > about using Fabric for the same reason, and if libcloud uses paramiko
>> > then I'd probably not be on the fence and just start using Fabric too
>> > ;) (Currently I'm using ssh calls via subprocess.)
>>
>> I am not aware of any pure python SSH clients, other than Twisted
>> Conch, which would mean... porting to twisted (and depending on
>> twisted!)
>>
>
> Certainly not what I'm asking for ;) ssh via subprocess works fine on any
> Posixy system, but of course is more tricky on Windows. (try/except around
> the paramiko import would work for me though)
>
>
>> > Does deploy_node do both a create, and then run the deployment? What
>> > kind of error checking does it have? E.g., if that apt-get fails,
>> > what then? Does Fabric already accomplish the same things with
>> > greater depth?
>>
>> It both creates the node, and does the initial connection. It has
>> very little error handling right now. I definitely would like to
>> improve this though -- this is just a first draft :-)
>>
>
> Upon further thought, and given the use cases for my tool (
> http://toppcloud.colorstudy.com -- soon to be renamed Silver Lining once I
> finish up the text substitution), I can't really envision using this code as
> an underlying layer. Once the system is setup and ssh is accessible I think
> it should be outside libcloud's scope. At that point you aren't abstracting
> the cloud, you are handling concrete systems, and other abstraction layers
> (like Puppet, Fabric, etc) are more appropriate. What *would* be helpful is
> getting that initial ssh connection working, or some kind of abstraction
> around that. Right now on Rackspace Cloud I'm using their files extension
> to upload authorized_keys, but I'd have to switch things around some to get
> the same process working on EC2. So... abstracting that difference would be
> nice. But after that, eh.
Yes, exactly, that is my only goal with this API.
Convert various password generated, password assigned, ssh key via
create_node, ssh key via metadata script, or backed in ssh key -> you
install an ssh key of your choice.
Once there, its no longer libcloud's problem imo, and going beyond
just the tools to get a consistent way to get access to the machine is
not something I am interested in doing, so I think we are on the same
page.
Re: [libcloud] need feedback on deployment API in trunk!
Posted by Ian Bicking <ia...@gmail.com>.
On Sun, Feb 28, 2010 at 11:37 PM, Paul Querna <pa...@querna.org> wrote:
> > I'm a little wary of paramiko, mostly because of the pycrypto
> > requirement, which itself require compiling stuff. It's not terrible,
> > but it introduces a potential installation problem that I was hoping
> > to avoid (by using pure Python libraries). OTOH, I'm on the fence
> > about using Fabric for the same reason, and if libcloud uses paramiko
> > then I'd probably not be on the fence and just start using Fabric too
> > ;) (Currently I'm using ssh calls via subprocess.)
>
> I am not aware of any pure python SSH clients, other than Twisted
> Conch, which would mean... porting to twisted (and depending on
> twisted!)
>
Certainly not what I'm asking for ;) ssh via subprocess works fine on any
Posixy system, but of course is more tricky on Windows. (try/except around
the paramiko import would work for me though)
> > Does deploy_node do both a create, and then run the deployment? What
> > kind of error checking does it have? E.g., if that apt-get fails,
> > what then? Does Fabric already accomplish the same things with
> > greater depth?
>
> It both creates the node, and does the initial connection. It has
> very little error handling right now. I definitely would like to
> improve this though -- this is just a first draft :-)
>
Upon further thought, and given the use cases for my tool (
http://toppcloud.colorstudy.com -- soon to be renamed Silver Lining once I
finish up the text substitution), I can't really envision using this code as
an underlying layer. Once the system is setup and ssh is accessible I think
it should be outside libcloud's scope. At that point you aren't abstracting
the cloud, you are handling concrete systems, and other abstraction layers
(like Puppet, Fabric, etc) are more appropriate. What *would* be helpful is
getting that initial ssh connection working, or some kind of abstraction
around that. Right now on Rackspace Cloud I'm using their files extension
to upload authorized_keys, but I'd have to switch things around some to get
the same process working on EC2. So... abstracting that difference would be
nice. But after that, eh.
--
Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking
Re: [libcloud] need feedback on deployment API in trunk!
Posted by Paul Querna <pa...@querna.org>.
On Sun, Feb 28, 2010 at 7:12 PM, Ian Bicking <ia...@gmail.com> wrote:
> On Sun, Feb 28, 2010 at 9:04 PM, Tom White <to...@cloudera.com> wrote:
>>> So, basically there is a new deploy_node() method on the base driver
>>> class. If a driver supports either giving us the nodes password, or
>>> setting one, it logs in using Paramiko, and can do basic setup of the
>>> node. Supporting SSH Keys for places like amazon shouldn't be too
>>> hard to extend in here quickly either!
>
> I'm a little wary of paramiko, mostly because of the pycrypto
> requirement, which itself require compiling stuff. It's not terrible,
> but it introduces a potential installation problem that I was hoping
> to avoid (by using pure Python libraries). OTOH, I'm on the fence
> about using Fabric for the same reason, and if libcloud uses paramiko
> then I'd probably not be on the fence and just start using Fabric too
> ;) (Currently I'm using ssh calls via subprocess.)
I am not aware of any pure python SSH clients, other than Twisted
Conch, which would mean... porting to twisted (and depending on
twisted!)
If there are any others out there, I'd be happy to investigate switching.
> Does deploy_node do both a create, and then run the deployment? What
> kind of error checking does it have? E.g., if that apt-get fails,
> what then? Does Fabric already accomplish the same things with
> greater depth?
It both creates the node, and does the initial connection. It has
very little error handling right now. I definitely would like to
improve this though -- this is just a first draft :-)
I think there are a few options if we have a problem with the SSH connection:
- We could leave the node alone (clean it up yourself later somehow)
- We could destroy the node
- We could provide an optional argument to pick one of the above behaviors.
Re: [libcloud] need feedback on deployment API in trunk!
Posted by Ian Bicking <ia...@gmail.com>.
On Sun, Feb 28, 2010 at 9:04 PM, Tom White <to...@cloudera.com> wrote:
>> So, basically there is a new deploy_node() method on the base driver
>> class. If a driver supports either giving us the nodes password, or
>> setting one, it logs in using Paramiko, and can do basic setup of the
>> node. Supporting SSH Keys for places like amazon shouldn't be too
>> hard to extend in here quickly either!
I'm a little wary of paramiko, mostly because of the pycrypto
requirement, which itself require compiling stuff. It's not terrible,
but it introduces a potential installation problem that I was hoping
to avoid (by using pure Python libraries). OTOH, I'm on the fence
about using Fabric for the same reason, and if libcloud uses paramiko
then I'd probably not be on the fence and just start using Fabric too
;) (Currently I'm using ssh calls via subprocess.)
Does deploy_node do both a create, and then run the deployment? What
kind of error checking does it have? E.g., if that apt-get fails,
what then? Does Fabric already accomplish the same things with
greater depth?
--
Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking
Re: [libcloud] need feedback on deployment API in trunk!
Posted by Tom White <to...@cloudera.com>.
Hi Paul,
This looks great! One thing I noticed is that paramiko is LGPL'd which
may not be used with Apache products
(http://www.apache.org/legal/resolved.html#category-x), so we'll need
to use something else, unfortunately.
Cheers,
Tom
On Sun, Feb 28, 2010 at 5:15 PM, Paul Querna <pa...@querna.org> wrote:
> Hello Everyone,
>
> I've taken a stab at a basic deployment API, and its in libcloud trunk now.
>
> An example of using it:
>
> from libcloud.providers import get_driver
> from libcloud.deployment import MultiStepDeployment, ScriptDeployment,
> SSHKeyDeployment
>
> driver = get_driver(Provider.RACKSPACE)('libcloud', 'you secret here')
> loc = driver.list_locations()
> sizes = driver.list_sizes()
> images = driver.list_images()
>
> key = open(os.path.expanduser("~/.ssh/my_rsa.pub")).read()
>
> sd = SSHKeyDeployment(key)
> script = ScriptDeployment("apt-get install -y puppet")
> msd = MultiStepDeployment([sd, script])
>
> node = driver.deploy_node(name="lc-test.k1k.me", location=loc[0],
> image=images[0],
> size=sizes[0], deploy=msd)
>
> So, basically there is a new deploy_node() method on the base driver
> class. If a driver supports either giving us the nodes password, or
> setting one, it logs in using Paramiko, and can do basic setup of the
> node. Supporting SSH Keys for places like amazon shouldn't be too
> hard to extend in here quickly either!
>
> Thoughts on the API? Any major flaws or suggestions on how to make it
> easier and more robust?
>
> Thanks,
>
> Paul
>
Re: [libcloud] need feedback on deployment API in trunk!
Posted by Paul Querna <pa...@querna.org>.
On Tue, Mar 2, 2010 at 7:09 AM, Jed Smith <je...@jedsmith.org> wrote:
> On Feb 28, 2010, at 8:15 PM, Paul Querna wrote:
>
>> from libcloud.providers import get_driver
>> from libcloud.deployment import MultiStepDeployment, ScriptDeployment,
>> SSHKeyDeployment
>
>> key = open(os.path.expanduser("~/.ssh/my_rsa.pub")).read()
>
>> sd = SSHKeyDeployment(key)
>> script = ScriptDeployment("apt-get install -y puppet")
>> msd = MultiStepDeployment([sd, script])
>
> This might be interesting for Linode, and not require SSH at all. We just
> rolled out Stackscripts, which are shebang'd scripts that are written to the
> filesystem before first boot and run once booted.
>
> SSHKeyDeployment, in this scenario, is supported in the create API via the same
> philosophy (and will write /root/.ssh/authorized_keys2 for you).
> ScriptDeployment I could see being tied to Stackscripts.
>
> I haven't looked at the code at all, but this would be the deployment path for
> what you've described in our API:
>
> linode.create
>
> stackscript.create
> script = "#!/bin/bash \r\n apt-get update \r\n apt-get install -y puppet"
>
> linode.disk.createfromstackscript
> # Might not support an SSH key yet, I need to check on that.
> # linode.disk.createfromdistribution does.
>
> At this point, the Linode is armed with the script, but is not booted. The
> StackScript is also reusable.
>
> Does this make sense, though, given the implementation? It sounds like the
> implementation as it stands is tied to SSH being required, and it isn't for us.
> Is it up to the driver? (Again, haven't looked -- just sat down for the day.)
Right now all the code is in the base class' deploy_node function, but
it can be overridden in the child class drivers, like Linode, if they
provided a better way to get started.
Re: [libcloud] need feedback on deployment API in trunk!
Posted by Jed Smith <je...@jedsmith.org>.
On Feb 28, 2010, at 8:15 PM, Paul Querna wrote:
> from libcloud.providers import get_driver
> from libcloud.deployment import MultiStepDeployment, ScriptDeployment,
> SSHKeyDeployment
> key = open(os.path.expanduser("~/.ssh/my_rsa.pub")).read()
> sd = SSHKeyDeployment(key)
> script = ScriptDeployment("apt-get install -y puppet")
> msd = MultiStepDeployment([sd, script])
This might be interesting for Linode, and not require SSH at all. We just
rolled out Stackscripts, which are shebang'd scripts that are written to the
filesystem before first boot and run once booted.
SSHKeyDeployment, in this scenario, is supported in the create API via the same
philosophy (and will write /root/.ssh/authorized_keys2 for you).
ScriptDeployment I could see being tied to Stackscripts.
I haven't looked at the code at all, but this would be the deployment path for
what you've described in our API:
linode.create
stackscript.create
script = "#!/bin/bash \r\n apt-get update \r\n apt-get install -y puppet"
linode.disk.createfromstackscript
# Might not support an SSH key yet, I need to check on that.
# linode.disk.createfromdistribution does.
At this point, the Linode is armed with the script, but is not booted. The
StackScript is also reusable.
Does this make sense, though, given the implementation? It sounds like the
implementation as it stands is tied to SSH being required, and it isn't for us.
Is it up to the driver? (Again, haven't looked -- just sat down for the day.)
JS