You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@libcloud.apache.org by Tomaz Muraus <to...@cloudkick.com> on 2011/02/05 21:31:09 UTC
[libcloud] libcloud roadmap and plans for the future
I know that the development of the storage API has started just recently,
but I would still love talk about the libcloud roadmap and plans for the
future.
I would eventually like to see the following features:
- non-blocking / asynchronous version of libcloud - I know that this is a
major feature and requires a lot of work, but it would be very cool if, in
the future you could do something like this:
from libcloud.compute.drivers.sync import EC2Driver - imports blocking
version of the driver
from libcloud.compute.drivers.async import EC2Driver - imports not-blocking
version of the driver
Since Twisted is pretty much a de-facto standard for creating non-blocking
Python applications, I think we should also use it for a non-blocking
version of out library.
- "Resources" concept - this feature is also be a pretty major one and
requires a lot of thought.
A "Resource" would be a generic concept which would represent a some kind of
"Cloud resource" - a resource could be an IP address, load balancer, etc.
This would basically allow us to implement a lot of (currently) provider
specific feature, but in a generic way so it would later be easy to adapt it
to work with a different provider.
- Python 3.x support - Jed has already suggested this on IRC. I actually
haven't tested libcloud with Python 3.x yet, so I don't know how many things
needs to be changed to make it work (without using 2to3 or a similar tool).
Thoughts, opinions?
Thanks,
Tomaz
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Tom Davis <to...@dislocatedday.com>.
On Feb 6, 2011, at 10:58 PM, Jerry Chen <je...@apache.org> wrote:
>
> On Feb 6, 2011, at 9:57 PM, Jed Smith wrote:
>
>> At the risk of sounding dense, how does asynchronous benefit the
>> library as a whole? It seems like a (potential) benefit for storage,
>> but not compute.
>>
>> Almost every use case on the compute side is, by nature, blocking.
>> Create a node, delete a node, list nodes - the calling application or
>> library is going to wait on these. On the other hand, there is a
>> benefit to the storage side in that the calling app can put a file
>> asynchronously. Am I reading this correctly, or is there something I'm
>> missing?
>>
>> I think in this case implementing asynchronous behavior is going to be
>> important to a subset of users. Right now, storage isn't finished, so
>> it poses no benefit to current use cases, does it? Are there a bunch
>> of cases where a transition to a callback model would benefit the
>> calling app? Would it be more effective for the calling application to
>> wrap libcloud in an asynchronous framework if it so desires
>>
>> I'm just not seeing the benefit, that's all.
>
> I don't think we need async capabilities in Libcloud either.
>
I was more trying to argue *against* reimplementing libcloud in node.js than *for* a port of the library. For most of the node-related stuff it's not enough to simply callback on a response anyway; in many cases you'd need to keep checking the node status until it is up, ensure consistency, etc. To actually be useful it'd require far more than a simple asyncore http client.
Anyway, I'm rather out of my element having not taken in the full state of the project recently; my spidey sense started tingling with talk of brand new networking engines just for libcloud, language ports, and so on—I had to throw my hat into the "overkill" ring on that.
Best,
Tom
> Cheers,
> Jerry
>
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Jerry Chen <je...@apache.org>.
On Feb 6, 2011, at 9:57 PM, Jed Smith wrote:
> At the risk of sounding dense, how does asynchronous benefit the
> library as a whole? It seems like a (potential) benefit for storage,
> but not compute.
>
> Almost every use case on the compute side is, by nature, blocking.
> Create a node, delete a node, list nodes - the calling application or
> library is going to wait on these. On the other hand, there is a
> benefit to the storage side in that the calling app can put a file
> asynchronously. Am I reading this correctly, or is there something I'm
> missing?
>
> I think in this case implementing asynchronous behavior is going to be
> important to a subset of users. Right now, storage isn't finished, so
> it poses no benefit to current use cases, does it? Are there a bunch
> of cases where a transition to a callback model would benefit the
> calling app? Would it be more effective for the calling application to
> wrap libcloud in an asynchronous framework if it so desires
>
> I'm just not seeing the benefit, that's all.
I don't think we need async capabilities in Libcloud either.
Cheers,
Jerry
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Jed Smith <je...@jedsmith.org>.
On Mon, Feb 7, 2011 at 9:48 AM, Tomaz Muraus <to...@cloudkick.com> wrote:
>
> Currently, if you want to launch multiple number of nodes or, for example,
> you are constantly polling for node status on multiple accounts, you need to
> wait for a previous call to return or use something like threading which is
> in most cases less than ideal.
I don't think the majority of libcloud users are using libcloud in a
manner that requires such gymnastics.
I also don't think threading is easily dismissable as "less than
ideal". I mean yes, we can do the GIL dance and talk about the
pitfalls of concurrency; that is not an argument with a shortcoming of
evidence for both sides. However, given Python's natives for things
like a multiprocessing pool, threadsafe Queue, and the other rich APIs
in that department I don't see how libcloud reinventing that wheel is
going to benefit the end user.
If I'm using a library like libcloud, and I know it's a blocker and I
need to execute 100 node creations, I'm always fine with queueing up
the requests and firing off worker threads. It's a common pattern and
I've done it before. I understand that it's necessary because of the
way libcloud is designed. If, on the other hand, I'm writing a
highly-available TCP server in Python, I'd hope the libraries I use
are designed asynchronously.
The difference, I think, is whether an application is reaching out to
get something or responding to incoming requests. Asynchronous tends
to benefit response models, and I don't think it will help us a lot
with reaching out - libcloud users most likely understand HTTP and
waiting and are willing to work around it if they need to.
Asynchronous, to me, seems like a significant investment in the
library for not much benefit. Certainly it will benefit the Cloudkicks
of the world, but I think if we roll it out it will be very unused,
and we will have significantly refactored the library for little gain.
As Jerry mentioned, I'm willing to be proven wrong by the users -
please, speak up.
--
Jed Smith
jed@jedsmith.org
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Jed Smith <je...@jedsmith.org>.
On Mon, Feb 7, 2011 at 10:04 AM, Jerry Chen <je...@apache.org> wrote:
>
> As long as the abstraction doesn't create weird race conditions, nor forces us to modify each driver for async stuff to work, I would be okay with this capability. I just can't wrap my head around how one would test for correctness and stability.
I believe it would. You now need to track what requests you have out
and call the appropriate callback, a very difficult (and tedious
task). A supposedly stateless library just inherited a /lot/ of state.
Anecdote: I solved this problem for the Linode iPhone app. If you're
not familiar with UIKit/Foundation development for iOS, it's all
asynchronous. I spent very many hours debugging the code that did the
linkage between APIRequest instances and the callback event on them.
It got so difficult to do properly with Apple's APIs that I eventually
stumbled upon a hopelessly inelegant solution: every request the
iPhone app makes is a batch, with a test.echo action containing an ID
which is a key into the pending-requests dictionary. Seriously.
That's what asynchronous ends up becoming in a request-receive
environment. It's a rabbit hole of inelegant solutions and a lot of
tedious state tracking in the library.
--
Jed Smith
jed@jedsmith.org
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Jerry Chen <je...@apache.org>.
On Feb 8, 2011, at 8:58 AM, Tomaz Muraus wrote:
> On Tue, Feb 8, 2011 at 2:57 PM, Schwartz, Philip Marc (LNG-BCT) <
> Philip.Schwartz@lexisnexis.com> wrote:
>
>> I would also vote -1 to twisted. But 0 to async. I am for async to a point
>> as long as it is setup as a per instance option for the user.
>>
>
> Yes, that is how I have envisioned it. I probably should have explained it
> better in my original post.
>
> The change should be non-disruptive and the Twisted dependency should be
> optional (basically it would be totally up to the user if he wants to use
> async API or not).
>
> And like Paul has pointed out above, refactoring a library might not be a
> bad idea, since we will need to split and refactor some of the existing
> methods and this will make testing (among other things) easier.
All this being said -- Paul, where is your node.js port of Libcloud? ;)
Cheers,
Jerry
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Tomaz Muraus <to...@cloudkick.com>.
On Tue, Feb 8, 2011 at 2:57 PM, Schwartz, Philip Marc (LNG-BCT) <
Philip.Schwartz@lexisnexis.com> wrote:
> I would also vote -1 to twisted. But 0 to async. I am for async to a point
> as long as it is setup as a per instance option for the user.
>
Yes, that is how I have envisioned it. I probably should have explained it
better in my original post.
The change should be non-disruptive and the Twisted dependency should be
optional (basically it would be totally up to the user if he wants to use
async API or not).
And like Paul has pointed out above, refactoring a library might not be a
bad idea, since we will need to split and refactor some of the existing
methods and this will make testing (among other things) easier.
Thanks,
Tomaz
>
> In other words making a secondary async http client library to allow usage
> if desired with no lasting effect on the core of libcloud. The moment we
> make a change as drastic as making the http client async will be the moment
> that we make people using the library a little angry as most would not want
> an async http client for compute work.
>
> Thank You,
> Philip Schwartz
> Software Engineering
> LexisNexis RS
> O - 561 999 4472
> C - 954 290 4024
>
>
> -----Original Message-----
> From: Upayavira [mailto:uv@odoko.co.uk]
> Sent: Monday, February 07, 2011 3:43 PM
> To: libcloud@incubator.apache.org
> Subject: Re: [libcloud] libcloud roadmap and plans for the future
>
> Personally, I'd be concerned if Libcloud took on Twisted. I (as
> something of an outsider) see Libcloud as a lightweight library for
> interfacing with cloud providers. If you add Twisted, you add a
> heavyweight dependency to a lightweight library, and suddenly deploying
> your Libcloud based apps becomes a whole lot harder, especially if
> you've never used Twisted.
>
> Upayavira
>
> On Mon, 07 Feb 2011 19:03 +0100, "Tomaz Muraus" <to...@cloudkick.com>
> wrote:
> > On Mon, Feb 7, 2011 at 4:04 PM, Jerry Chen <je...@apache.org> wrote:
> >
> > >
> > > On Feb 7, 2011, at 8:48 AM, Tomaz Muraus wrote:
> > >
> > > > On Mon, Feb 7, 2011 at 4:57 AM, Jed Smith <je...@jedsmith.org> wrote:
> > > I agree. Speak up, people!
> > >
> > > BTW, I started up a wish list on the wiki a while back:
> > > http://wiki.apache.org/incubator/LibcloudWishlist
> > >
> > > I'm not convinced it has the best voting mechanism (it acts like a IRC
> > > topic vote!) but at least we can get an idea of the weighting.
> > >
> >
> > Cool, I have updated the wiki with other ideas.
> >
> > On Mon, Feb 7, 2011 at 4:24 PM, Jed Smith <je...@jedsmith.org> wrote:
> >
> > > On Mon, Feb 7, 2011 at 10:04 AM, Jerry Chen <je...@apache.org> wrote:
> > > >
> > > > As long as the abstraction doesn't create weird race conditions, nor
> > > forces us to modify each driver for async stuff to work, I would be
> okay
> > > with this capability. I just can't wrap my head around how one would
> test
> > > for correctness and stability.
> > >
> > > I believe it would. You now need to track what requests you have out
> > > and call the appropriate callback, a very difficult (and tedious
> > > task). A supposedly stateless library just inherited a /lot/ of state.
> > >
> >
> > In Twisted and other event-driven system everything runs inside a single
> > thread (reactor) and as long you are only using asynchronous API, you
> > don't
> > need to worry about race conditions.
> >
> > In any case, adapting the library for async stuff would definitely
> > require
> > some refactoring, but I could argue that this might not be such a bad
> > idea.
> >
> > Anecdote: I solved this problem for the Linode iPhone app. If you're
> > > not familiar with UIKit/Foundation development for iOS, it's all
> > > asynchronous. I spent very many hours debugging the code that did the
> > > linkage between APIRequest instances and the callback event on them.
> > > It got so difficult to do properly with Apple's APIs that I eventually
> > > stumbled upon a hopelessly inelegant solution: every request the
> > > iPhone app makes is a batch, with a test.echo action containing an ID
> > > which is a key into the pending-requests dictionary. Seriously.
> > >
> > > That's what asynchronous ends up becoming in a request-receive
> > > environment. It's a rabbit hole of inelegant solutions and a lot of
> > > tedious state tracking in the library.
> >
> >
> > Yes, I agree, debugging in a lot of callback oriented frameworks can be a
> > huge pain. One of the good examples of this is NodeJS - with a lot of
> > nested
> > callbacks and no long stack traces finding a root cause can be pretty
> > painful (Twisted partially solves this with defer.SetDebugging(True)).
> >
> > So, yes, in general debugging and writing asynchronous code can be
> > painful,
> > but Twited has enough of syntactic sugar which makes it a lot less
> > painful
> > (deferreds, inlineCallbacks, ...).
> >
> >
> > > --
> > > Jed Smith
> > > jed@jedsmith.org
> > >
> >
>
> This message (including any attachments) contains confidential information
> intended for a specific individual and purpose, and is protected by law. If
> you are not the intended recipient, you should delete this message. Any
> disclosure, copying, or distribution of this message, or the taking of any
> action based on it, is strictly prohibited.
>
RE: [libcloud] libcloud roadmap and plans for the future
Posted by "Schwartz, Philip Marc (LNG-BCT)" <Ph...@lexisnexis.com>.
I would also vote -1 to twisted. But 0 to async. I am for async to a point as long as it is setup as a per instance option for the user.
In other words making a secondary async http client library to allow usage if desired with no lasting effect on the core of libcloud. The moment we make a change as drastic as making the http client async will be the moment that we make people using the library a little angry as most would not want an async http client for compute work.
Thank You,
Philip Schwartz
Software Engineering
LexisNexis RS
O - 561 999 4472
C - 954 290 4024
-----Original Message-----
From: Upayavira [mailto:uv@odoko.co.uk]
Sent: Monday, February 07, 2011 3:43 PM
To: libcloud@incubator.apache.org
Subject: Re: [libcloud] libcloud roadmap and plans for the future
Personally, I'd be concerned if Libcloud took on Twisted. I (as
something of an outsider) see Libcloud as a lightweight library for
interfacing with cloud providers. If you add Twisted, you add a
heavyweight dependency to a lightweight library, and suddenly deploying
your Libcloud based apps becomes a whole lot harder, especially if
you've never used Twisted.
Upayavira
On Mon, 07 Feb 2011 19:03 +0100, "Tomaz Muraus" <to...@cloudkick.com>
wrote:
> On Mon, Feb 7, 2011 at 4:04 PM, Jerry Chen <je...@apache.org> wrote:
>
> >
> > On Feb 7, 2011, at 8:48 AM, Tomaz Muraus wrote:
> >
> > > On Mon, Feb 7, 2011 at 4:57 AM, Jed Smith <je...@jedsmith.org> wrote:
> > I agree. Speak up, people!
> >
> > BTW, I started up a wish list on the wiki a while back:
> > http://wiki.apache.org/incubator/LibcloudWishlist
> >
> > I'm not convinced it has the best voting mechanism (it acts like a IRC
> > topic vote!) but at least we can get an idea of the weighting.
> >
>
> Cool, I have updated the wiki with other ideas.
>
> On Mon, Feb 7, 2011 at 4:24 PM, Jed Smith <je...@jedsmith.org> wrote:
>
> > On Mon, Feb 7, 2011 at 10:04 AM, Jerry Chen <je...@apache.org> wrote:
> > >
> > > As long as the abstraction doesn't create weird race conditions, nor
> > forces us to modify each driver for async stuff to work, I would be okay
> > with this capability. I just can't wrap my head around how one would test
> > for correctness and stability.
> >
> > I believe it would. You now need to track what requests you have out
> > and call the appropriate callback, a very difficult (and tedious
> > task). A supposedly stateless library just inherited a /lot/ of state.
> >
>
> In Twisted and other event-driven system everything runs inside a single
> thread (reactor) and as long you are only using asynchronous API, you
> don't
> need to worry about race conditions.
>
> In any case, adapting the library for async stuff would definitely
> require
> some refactoring, but I could argue that this might not be such a bad
> idea.
>
> Anecdote: I solved this problem for the Linode iPhone app. If you're
> > not familiar with UIKit/Foundation development for iOS, it's all
> > asynchronous. I spent very many hours debugging the code that did the
> > linkage between APIRequest instances and the callback event on them.
> > It got so difficult to do properly with Apple's APIs that I eventually
> > stumbled upon a hopelessly inelegant solution: every request the
> > iPhone app makes is a batch, with a test.echo action containing an ID
> > which is a key into the pending-requests dictionary. Seriously.
> >
> > That's what asynchronous ends up becoming in a request-receive
> > environment. It's a rabbit hole of inelegant solutions and a lot of
> > tedious state tracking in the library.
>
>
> Yes, I agree, debugging in a lot of callback oriented frameworks can be a
> huge pain. One of the good examples of this is NodeJS - with a lot of
> nested
> callbacks and no long stack traces finding a root cause can be pretty
> painful (Twisted partially solves this with defer.SetDebugging(True)).
>
> So, yes, in general debugging and writing asynchronous code can be
> painful,
> but Twited has enough of syntactic sugar which makes it a lot less
> painful
> (deferreds, inlineCallbacks, ...).
>
>
> > --
> > Jed Smith
> > jed@jedsmith.org
> >
>
This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message. Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Upayavira <uv...@odoko.co.uk>.
Personally, I'd be concerned if Libcloud took on Twisted. I (as
something of an outsider) see Libcloud as a lightweight library for
interfacing with cloud providers. If you add Twisted, you add a
heavyweight dependency to a lightweight library, and suddenly deploying
your Libcloud based apps becomes a whole lot harder, especially if
you've never used Twisted.
Upayavira
On Mon, 07 Feb 2011 19:03 +0100, "Tomaz Muraus" <to...@cloudkick.com>
wrote:
> On Mon, Feb 7, 2011 at 4:04 PM, Jerry Chen <je...@apache.org> wrote:
>
> >
> > On Feb 7, 2011, at 8:48 AM, Tomaz Muraus wrote:
> >
> > > On Mon, Feb 7, 2011 at 4:57 AM, Jed Smith <je...@jedsmith.org> wrote:
> > I agree. Speak up, people!
> >
> > BTW, I started up a wish list on the wiki a while back:
> > http://wiki.apache.org/incubator/LibcloudWishlist
> >
> > I'm not convinced it has the best voting mechanism (it acts like a IRC
> > topic vote!) but at least we can get an idea of the weighting.
> >
>
> Cool, I have updated the wiki with other ideas.
>
> On Mon, Feb 7, 2011 at 4:24 PM, Jed Smith <je...@jedsmith.org> wrote:
>
> > On Mon, Feb 7, 2011 at 10:04 AM, Jerry Chen <je...@apache.org> wrote:
> > >
> > > As long as the abstraction doesn't create weird race conditions, nor
> > forces us to modify each driver for async stuff to work, I would be okay
> > with this capability. I just can't wrap my head around how one would test
> > for correctness and stability.
> >
> > I believe it would. You now need to track what requests you have out
> > and call the appropriate callback, a very difficult (and tedious
> > task). A supposedly stateless library just inherited a /lot/ of state.
> >
>
> In Twisted and other event-driven system everything runs inside a single
> thread (reactor) and as long you are only using asynchronous API, you
> don't
> need to worry about race conditions.
>
> In any case, adapting the library for async stuff would definitely
> require
> some refactoring, but I could argue that this might not be such a bad
> idea.
>
> Anecdote: I solved this problem for the Linode iPhone app. If you're
> > not familiar with UIKit/Foundation development for iOS, it's all
> > asynchronous. I spent very many hours debugging the code that did the
> > linkage between APIRequest instances and the callback event on them.
> > It got so difficult to do properly with Apple's APIs that I eventually
> > stumbled upon a hopelessly inelegant solution: every request the
> > iPhone app makes is a batch, with a test.echo action containing an ID
> > which is a key into the pending-requests dictionary. Seriously.
> >
> > That's what asynchronous ends up becoming in a request-receive
> > environment. It's a rabbit hole of inelegant solutions and a lot of
> > tedious state tracking in the library.
>
>
> Yes, I agree, debugging in a lot of callback oriented frameworks can be a
> huge pain. One of the good examples of this is NodeJS - with a lot of
> nested
> callbacks and no long stack traces finding a root cause can be pretty
> painful (Twisted partially solves this with defer.SetDebugging(True)).
>
> So, yes, in general debugging and writing asynchronous code can be
> painful,
> but Twited has enough of syntactic sugar which makes it a lot less
> painful
> (deferreds, inlineCallbacks, ...).
>
>
> > --
> > Jed Smith
> > jed@jedsmith.org
> >
>
RE: [libcloud] libcloud roadmap and plans for the future
Posted by "Schwartz, Philip Marc (LNG-BCT)" <Ph...@lexisnexis.com>.
Tom also sent me an email outside of this thread. Here is my post as I have nothing against twisted, just with the usage of it in libcloud.
I have nothing against twisted. I use it on a day to day basis in my work along with the OpenStack project. My concerns are using it in libcloud as Twisted is not lightweight.
My current usage of libcloud is 2 fold. Embedded into c++ via the python interpreter and in a python module that is used on demand in an application server written on top of Facebook’s Tornado server for libevent based async. To add the requirement of twisted on an app like mine would just add to the memory foot print and when working in a libevent spawned thread pool of 12-48 threads or more at a time can be a downfall of my usage of libcloud.
I can see many people that run large scale usages of libcloud running into the same issue.
Thank You,
Philip Schwartz
Software Engineering
LexisNexis RS
O - 561 999 4472
C - 954 290 4024
-----Original Message-----
From: Jed Smith [mailto:jed@jedsmith.org]
Sent: Tuesday, February 08, 2011 10:57 AM
To: libcloud@incubator.apache.org
Subject: Re: [libcloud] libcloud roadmap and plans for the future
On Tue, Feb 8, 2011 at 12:24 AM, Paul Querna <pa...@querna.org> wrote:
>
> Right now this just feels like witch burning of Twisted Python
Tom mailed me off-list to ask me the same thing, so I'll clarify publicly:
I can't witch hunt something I've never used, I can merely resist a
dependency as (perceivably) heavy as Twisted. I resisted M2Crypto for
the single purpose we needed it for, and I will continue to resist any
added dependencies that don't benefit the library as a whole.
--
Jed Smith
jed@jedsmith.org
This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message. Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Jerry Chen <je...@apache.org>.
On Feb 8, 2011, at 9:57 AM, Jed Smith wrote:
> On Tue, Feb 8, 2011 at 12:24 AM, Paul Querna <pa...@querna.org> wrote:
>>
>> Right now this just feels like witch burning of Twisted Python
>
> Tom mailed me off-list to ask me the same thing, so I'll clarify publicly:
>
> I can't witch hunt something I've never used, I can merely resist a
> dependency as (perceivably) heavy as Twisted. I resisted M2Crypto for
> the single purpose we needed it for, and I will continue to resist any
> added dependencies that don't benefit the library as a whole.
Kill it with fire!!
But in all seriousness, Twisted often polarizes the discussion and I'm glad we all brought the thread back to the pros and cons of implementing async.
Personally, I think we have more important components to address in the immediate future, but as Tomaz mentions, this is just something to think about for further down the road.
Cheers,
Jerry
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Jed Smith <je...@jedsmith.org>.
On Tue, Feb 8, 2011 at 12:24 AM, Paul Querna <pa...@querna.org> wrote:
>
> Right now this just feels like witch burning of Twisted Python
Tom mailed me off-list to ask me the same thing, so I'll clarify publicly:
I can't witch hunt something I've never used, I can merely resist a
dependency as (perceivably) heavy as Twisted. I resisted M2Crypto for
the single purpose we needed it for, and I will continue to resist any
added dependencies that don't benefit the library as a whole.
--
Jed Smith
jed@jedsmith.org
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Paul Querna <pa...@querna.org>.
On Mon, Feb 7, 2011 at 11:08 AM, Jed Smith <je...@jedsmith.org> wrote:
> On Mon, Feb 7, 2011 at 1:03 PM, Tomaz Muraus <to...@cloudkick.com> wrote:
>> but Twited has enough of syntactic sugar which makes it a lot less painful
>
> I still consider asynchronous a bad idea, but I consider Twisted even worse.
>
> I will strongly -1 any Twisted integration.
Lets slow down, I think we are getting caught up by other things.
Right now this just feels like witch burning of Twisted Python (and
don't get me wrong, I have plenty of issues with Twisted Python)
I think we could implement asynchronous support in libcloud in a
manner that was non-disruptive and without breaking the existing
public API.
I think if we look hard at our existing compute drivers, we see
several common patterns, which in some places we have well abstracted,
in others not as much.
The pattern I see in list_node (and I think most other operations):
1) Preparing the request of information. (Driver.list_node)
2) Making the request (Connection.request)
3) Parsing the results of that request. (Response.parse_body, Driver.list_node
The Rackspace driver (speaking of it because its complicated, not
because of my (new) employer) is slightly harder because of how it
gets its authentication token.
But the core pattern still holds in my eyes.
Thinking abstractly, where async lives in this place, is purely in
abstraction of how step 2, Making the request, is done.
If a driver was changed to just get a callback with the results, it
could work in both an async and non-async version with absolutely no
differences inside the actual drivers.
I definitely agree that that this kind of abstraction is far more
important for for storage, where common operations are to upload 10gb
files -- or even consider the new S3 API that lets you upload multiple
chunks in parallel -- doing that with a blocking API just doesn't make
sense.
I see a push to break up steps 1, 2, and 3 from each other also makes
it much easier to write test cases, because you can more directly test
the parsing of the result in step 3, and because you can more easily
test if the request is constructed correctly in step 1.
I'm not saying there is one solution to this, but lets just keep the
mind open before throwing around -1s before any technical
implementation beyond a very raw proposal has been done -- none of
even know how it would work right now.
My own personal vision is that an optional dependency on Twisted
Python is a good thing, it does expose you to more environments. I
agreee adding a hard dependency on Twisted python is a non-starter --
the core as we know it today, must remain clean. I think this would
could be structed by bundling the 'default' blocking reactor that
handles step 2, making the request, using httplib. An separate
module, libcloud.async.twisted, would import all the same symbols, but
change the request reactor to use Twisted's http library, instead of
httplib.
I think this is viable. I think drawing conclusions right now is too
early, we would need to prototype it to really see if it caused more
pain than the features it added.
Thanks,
Paul
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Jed Smith <je...@jedsmith.org>.
On Mon, Feb 7, 2011 at 1:03 PM, Tomaz Muraus <to...@cloudkick.com> wrote:
> but Twited has enough of syntactic sugar which makes it a lot less painful
I still consider asynchronous a bad idea, but I consider Twisted even worse.
I will strongly -1 any Twisted integration.
--
Jed Smith
jed@jedsmith.org
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Tomaz Muraus <to...@cloudkick.com>.
On Mon, Feb 7, 2011 at 4:04 PM, Jerry Chen <je...@apache.org> wrote:
>
> On Feb 7, 2011, at 8:48 AM, Tomaz Muraus wrote:
>
> > On Mon, Feb 7, 2011 at 4:57 AM, Jed Smith <je...@jedsmith.org> wrote:
> I agree. Speak up, people!
>
> BTW, I started up a wish list on the wiki a while back:
> http://wiki.apache.org/incubator/LibcloudWishlist
>
> I'm not convinced it has the best voting mechanism (it acts like a IRC
> topic vote!) but at least we can get an idea of the weighting.
>
Cool, I have updated the wiki with other ideas.
On Mon, Feb 7, 2011 at 4:24 PM, Jed Smith <je...@jedsmith.org> wrote:
> On Mon, Feb 7, 2011 at 10:04 AM, Jerry Chen <je...@apache.org> wrote:
> >
> > As long as the abstraction doesn't create weird race conditions, nor
> forces us to modify each driver for async stuff to work, I would be okay
> with this capability. I just can't wrap my head around how one would test
> for correctness and stability.
>
> I believe it would. You now need to track what requests you have out
> and call the appropriate callback, a very difficult (and tedious
> task). A supposedly stateless library just inherited a /lot/ of state.
>
In Twisted and other event-driven system everything runs inside a single
thread (reactor) and as long you are only using asynchronous API, you don't
need to worry about race conditions.
In any case, adapting the library for async stuff would definitely require
some refactoring, but I could argue that this might not be such a bad idea.
Anecdote: I solved this problem for the Linode iPhone app. If you're
> not familiar with UIKit/Foundation development for iOS, it's all
> asynchronous. I spent very many hours debugging the code that did the
> linkage between APIRequest instances and the callback event on them.
> It got so difficult to do properly with Apple's APIs that I eventually
> stumbled upon a hopelessly inelegant solution: every request the
> iPhone app makes is a batch, with a test.echo action containing an ID
> which is a key into the pending-requests dictionary. Seriously.
>
> That's what asynchronous ends up becoming in a request-receive
> environment. It's a rabbit hole of inelegant solutions and a lot of
> tedious state tracking in the library.
Yes, I agree, debugging in a lot of callback oriented frameworks can be a
huge pain. One of the good examples of this is NodeJS - with a lot of nested
callbacks and no long stack traces finding a root cause can be pretty
painful (Twisted partially solves this with defer.SetDebugging(True)).
So, yes, in general debugging and writing asynchronous code can be painful,
but Twited has enough of syntactic sugar which makes it a lot less painful
(deferreds, inlineCallbacks, ...).
> --
> Jed Smith
> jed@jedsmith.org
>
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Jerry Chen <je...@apache.org>.
On Feb 7, 2011, at 8:48 AM, Tomaz Muraus wrote:
> On Mon, Feb 7, 2011 at 4:57 AM, Jed Smith <je...@jedsmith.org> wrote:
>
>> At the risk of sounding dense, how does asynchronous benefit the
>> library as a whole? It seems like a (potential) benefit for storage,
>> but not compute.
>
> Even though most of the actions are by nature blocking, this doesn't mean
> that all of our users are using the library in a blocking way as well.
>
> Currently, if you want to launch multiple number of nodes or, for example,
> you are constantly polling for node status on multiple accounts, you need to
> wait for a previous call to return or use something like threading which is
> in most cases less than ideal.
As long as the abstraction doesn't create weird race conditions, nor forces us to modify each driver for async stuff to work, I would be okay with this capability. I just can't wrap my head around how one would test for correctness and stability.
> And yes, like you have already pointed out, this can be particularly useful
> when the storage part is in place.
>
> I know that the development of the Storage part has started just recently,
> but my thread was more about the long term plans for libcloud and not
> necessary about something which needs be developed as soon as possible.
>
> In the end, it would be nice to hear how most of our users are currently
> using libcloud and if asynchronous API would benefit them.
I agree. Speak up, people!
BTW, I started up a wish list on the wiki a while back: http://wiki.apache.org/incubator/LibcloudWishlist
I'm not convinced it has the best voting mechanism (it acts like a IRC topic vote!) but at least we can get an idea of the weighting.
Cheers,
Jerry
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Tomaz Muraus <to...@cloudkick.com>.
On Mon, Feb 7, 2011 at 4:57 AM, Jed Smith <je...@jedsmith.org> wrote:
> At the risk of sounding dense, how does asynchronous benefit the
> library as a whole? It seems like a (potential) benefit for storage,
> but not compute.
>
Even though most of the actions are by nature blocking, this doesn't mean
that all of our users are using the library in a blocking way as well.
Currently, if you want to launch multiple number of nodes or, for example,
you are constantly polling for node status on multiple accounts, you need to
wait for a previous call to return or use something like threading which is
in most cases less than ideal.
And yes, like you have already pointed out, this can be particularly useful
when the storage part is in place.
I know that the development of the Storage part has started just recently,
but my thread was more about the long term plans for libcloud and not
necessary about something which needs be developed as soon as possible.
In the end, it would be nice to hear how most of our users are currently
using libcloud and if asynchronous API would benefit them.
Thanks,
Tomaz
> Almost every use case on the compute side is, by nature, blocking.
> Create a node, delete a node, list nodes - the calling application or
> library is going to wait on these. On the other hand, there is a
> benefit to the storage side in that the calling app can put a file
> asynchronously. Am I reading this correctly, or is there something I'm
> missing?
>
> I think in this case implementing asynchronous behavior is going to be
> important to a subset of users. Right now, storage isn't finished, so
> it poses no benefit to current use cases, does it? Are there a bunch
> of cases where a transition to a callback model would benefit the
> calling app? Would it be more effective for the calling application to
> wrap libcloud in an asynchronous framework if it so desires
>
> I'm just not seeing the benefit, that's all.
>
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Jed Smith <je...@jedsmith.org>.
At the risk of sounding dense, how does asynchronous benefit the
library as a whole? It seems like a (potential) benefit for storage,
but not compute.
Almost every use case on the compute side is, by nature, blocking.
Create a node, delete a node, list nodes - the calling application or
library is going to wait on these. On the other hand, there is a
benefit to the storage side in that the calling app can put a file
asynchronously. Am I reading this correctly, or is there something I'm
missing?
I think in this case implementing asynchronous behavior is going to be
important to a subset of users. Right now, storage isn't finished, so
it poses no benefit to current use cases, does it? Are there a bunch
of cases where a transition to a callback model would benefit the
calling app? Would it be more effective for the calling application to
wrap libcloud in an asynchronous framework if it so desires
I'm just not seeing the benefit, that's all.
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Tom Davis <to...@dislocatedday.com>.
On Feb 6, 2011, at 4:25 PM, Mark Nottingham <mn...@mnot.net> wrote:
>
> On 07/02/2011, at 6:11 AM, Tomaz Muraus wrote:
>
>> Yes, Twisted is a big beast, but in my opinion it is still a lot better to
>> use Twisted than something like asyncore.
>>
>> If we would use something like asyncore this means that we need to implement
>> our own http client and a lot of other things which already exist in Twisted
>> and have been battle tested.
>
> True, but it's not that hard. I did an async HTTP client in a few hundred lines of Python, and it has been battle-tested.
Regardless of anything else, I agree with Tomaz that asyncore is an unnecessarily low level of abstraction for libcloud's needs. Unless your http client has all the necessary features and a robust test suite, it seems like another existing solution would be preferable.
>
>
>> On the other hand, we are not building a non-blocking Python framework so
>> using something like epoll, kqueue, etc. directly and implementing our own
>> abstractions is out of the question imo.
>>
>> There are also a lot of other non-blocking frameworks available for Python,
>> but Twisted is pretty much the de-facto standard.
>
> Instead of buying into Python's moral equivalent of J2EE, how about changing languages? You can get very far, very quickly with node.js, for example, and it's async from the ground up.
My vote is for Twisted, but I'm pretty biased towards it. As for changing languages, I think that should be a last resort considering you lose the copious amount of libcloud that's been implemented with no specific requirement for synchronous code (use inlineCallbacks in Twisted and you could probably just make a new Request class, stick "yield" in front of most calls to request(), and call it a day). There are other Python solutions that would allow code reuse, unless I'm missing something about node's ability to run Python code. Reinventing the wheel is hardly justified by a dislike of one choice, in my opinion.
>
>
>> On Sun, Feb 6, 2011 at 12:17 AM, Grig Gheorghiu <gr...@gmail.com>wrote:
>>
>>> I agree regarding Twisted. It's a road that can lead to perdition ;-)
>>>
>>> On Sat, Feb 5, 2011 at 3:06 PM, Mark Nottingham <mn...@mnot.net> wrote:
>>>> FWIW, my experience is that creating event-driven applications is easier
>>> if you just use asyncore and/or epoll directly. Twisted is too complex /
>>> over-engineered / under-documented for me (and many others).
>>>>
>>>>
>>>> On 06/02/2011, at 7:31 AM, Tomaz Muraus wrote:
>>>>
>>>>> I know that the development of the storage API has started just
>>> recently,
>>>>> but I would still love talk about the libcloud roadmap and plans for the
>>>>> future.
>>>>>
>>>>> I would eventually like to see the following features:
>>>>>
>>>>> - non-blocking / asynchronous version of libcloud - I know that this is
>>> a
>>>>> major feature and requires a lot of work, but it would be very cool if,
>>> in
>>>>> the future you could do something like this:
>>>>>
>>>>> from libcloud.compute.drivers.sync import EC2Driver - imports blocking
>>>>> version of the driver
>>>>> from libcloud.compute.drivers.async import EC2Driver - imports
>>> not-blocking
>>>>> version of the driver
>>>>>
>>>>> Since Twisted is pretty much a de-facto standard for creating
>>> non-blocking
>>>>> Python applications, I think we should also use it for a non-blocking
>>>>> version of out library.
>>>>>
>>>>> - "Resources" concept - this feature is also be a pretty major one and
>>>>> requires a lot of thought.
>>>>>
>>>>> A "Resource" would be a generic concept which would represent a some
>>> kind of
>>>>> "Cloud resource" - a resource could be an IP address, load balancer,
>>> etc.
>>>>>
>>>>> This would basically allow us to implement a lot of (currently) provider
>>>>> specific feature, but in a generic way so it would later be easy to
>>> adapt it
>>>>> to work with a different provider.
>>>>>
>>>>> - Python 3.x support - Jed has already suggested this on IRC. I actually
>>>>> haven't tested libcloud with Python 3.x yet, so I don't know how many
>>> things
>>>>> needs to be changed to make it work (without using 2to3 or a similar
>>> tool).
>>>>>
>>>>> Thoughts, opinions?
>>>>>
>>>>> Thanks,
>>>>> Tomaz
>>>>
>>>> --
>>>> Mark Nottingham http://www.mnot.net/
>>>>
>>>>
>>>>
>>>>
>>>
>
> --
> Mark Nottingham http://www.mnot.net/
>
>
>
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Mark Nottingham <mn...@mnot.net>.
On 07/02/2011, at 6:11 AM, Tomaz Muraus wrote:
> Yes, Twisted is a big beast, but in my opinion it is still a lot better to
> use Twisted than something like asyncore.
>
> If we would use something like asyncore this means that we need to implement
> our own http client and a lot of other things which already exist in Twisted
> and have been battle tested.
True, but it's not that hard. I did an async HTTP client in a few hundred lines of Python, and it has been battle-tested.
> On the other hand, we are not building a non-blocking Python framework so
> using something like epoll, kqueue, etc. directly and implementing our own
> abstractions is out of the question imo.
>
> There are also a lot of other non-blocking frameworks available for Python,
> but Twisted is pretty much the de-facto standard.
Instead of buying into Python's moral equivalent of J2EE, how about changing languages? You can get very far, very quickly with node.js, for example, and it's async from the ground up.
> On Sun, Feb 6, 2011 at 12:17 AM, Grig Gheorghiu <gr...@gmail.com>wrote:
>
>> I agree regarding Twisted. It's a road that can lead to perdition ;-)
>>
>> On Sat, Feb 5, 2011 at 3:06 PM, Mark Nottingham <mn...@mnot.net> wrote:
>>> FWIW, my experience is that creating event-driven applications is easier
>> if you just use asyncore and/or epoll directly. Twisted is too complex /
>> over-engineered / under-documented for me (and many others).
>>>
>>>
>>> On 06/02/2011, at 7:31 AM, Tomaz Muraus wrote:
>>>
>>>> I know that the development of the storage API has started just
>> recently,
>>>> but I would still love talk about the libcloud roadmap and plans for the
>>>> future.
>>>>
>>>> I would eventually like to see the following features:
>>>>
>>>> - non-blocking / asynchronous version of libcloud - I know that this is
>> a
>>>> major feature and requires a lot of work, but it would be very cool if,
>> in
>>>> the future you could do something like this:
>>>>
>>>> from libcloud.compute.drivers.sync import EC2Driver - imports blocking
>>>> version of the driver
>>>> from libcloud.compute.drivers.async import EC2Driver - imports
>> not-blocking
>>>> version of the driver
>>>>
>>>> Since Twisted is pretty much a de-facto standard for creating
>> non-blocking
>>>> Python applications, I think we should also use it for a non-blocking
>>>> version of out library.
>>>>
>>>> - "Resources" concept - this feature is also be a pretty major one and
>>>> requires a lot of thought.
>>>>
>>>> A "Resource" would be a generic concept which would represent a some
>> kind of
>>>> "Cloud resource" - a resource could be an IP address, load balancer,
>> etc.
>>>>
>>>> This would basically allow us to implement a lot of (currently) provider
>>>> specific feature, but in a generic way so it would later be easy to
>> adapt it
>>>> to work with a different provider.
>>>>
>>>> - Python 3.x support - Jed has already suggested this on IRC. I actually
>>>> haven't tested libcloud with Python 3.x yet, so I don't know how many
>> things
>>>> needs to be changed to make it work (without using 2to3 or a similar
>> tool).
>>>>
>>>> Thoughts, opinions?
>>>>
>>>> Thanks,
>>>> Tomaz
>>>
>>> --
>>> Mark Nottingham http://www.mnot.net/
>>>
>>>
>>>
>>>
>>
--
Mark Nottingham http://www.mnot.net/
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Tomaz Muraus <to...@cloudkick.com>.
Yes, Twisted is a big beast, but in my opinion it is still a lot better to
use Twisted than something like asyncore.
If we would use something like asyncore this means that we need to implement
our own http client and a lot of other things which already exist in Twisted
and have been battle tested.
On the other hand, we are not building a non-blocking Python framework so
using something like epoll, kqueue, etc. directly and implementing our own
abstractions is out of the question imo.
There are also a lot of other non-blocking frameworks available for Python,
but Twisted is pretty much the de-facto standard.
On Sun, Feb 6, 2011 at 12:17 AM, Grig Gheorghiu <gr...@gmail.com>wrote:
> I agree regarding Twisted. It's a road that can lead to perdition ;-)
>
> On Sat, Feb 5, 2011 at 3:06 PM, Mark Nottingham <mn...@mnot.net> wrote:
> > FWIW, my experience is that creating event-driven applications is easier
> if you just use asyncore and/or epoll directly. Twisted is too complex /
> over-engineered / under-documented for me (and many others).
> >
> >
> > On 06/02/2011, at 7:31 AM, Tomaz Muraus wrote:
> >
> >> I know that the development of the storage API has started just
> recently,
> >> but I would still love talk about the libcloud roadmap and plans for the
> >> future.
> >>
> >> I would eventually like to see the following features:
> >>
> >> - non-blocking / asynchronous version of libcloud - I know that this is
> a
> >> major feature and requires a lot of work, but it would be very cool if,
> in
> >> the future you could do something like this:
> >>
> >> from libcloud.compute.drivers.sync import EC2Driver - imports blocking
> >> version of the driver
> >> from libcloud.compute.drivers.async import EC2Driver - imports
> not-blocking
> >> version of the driver
> >>
> >> Since Twisted is pretty much a de-facto standard for creating
> non-blocking
> >> Python applications, I think we should also use it for a non-blocking
> >> version of out library.
> >>
> >> - "Resources" concept - this feature is also be a pretty major one and
> >> requires a lot of thought.
> >>
> >> A "Resource" would be a generic concept which would represent a some
> kind of
> >> "Cloud resource" - a resource could be an IP address, load balancer,
> etc.
> >>
> >> This would basically allow us to implement a lot of (currently) provider
> >> specific feature, but in a generic way so it would later be easy to
> adapt it
> >> to work with a different provider.
> >>
> >> - Python 3.x support - Jed has already suggested this on IRC. I actually
> >> haven't tested libcloud with Python 3.x yet, so I don't know how many
> things
> >> needs to be changed to make it work (without using 2to3 or a similar
> tool).
> >>
> >> Thoughts, opinions?
> >>
> >> Thanks,
> >> Tomaz
> >
> > --
> > Mark Nottingham http://www.mnot.net/
> >
> >
> >
> >
>
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Grig Gheorghiu <gr...@gmail.com>.
I agree regarding Twisted. It's a road that can lead to perdition ;-)
On Sat, Feb 5, 2011 at 3:06 PM, Mark Nottingham <mn...@mnot.net> wrote:
> FWIW, my experience is that creating event-driven applications is easier if you just use asyncore and/or epoll directly. Twisted is too complex / over-engineered / under-documented for me (and many others).
>
>
> On 06/02/2011, at 7:31 AM, Tomaz Muraus wrote:
>
>> I know that the development of the storage API has started just recently,
>> but I would still love talk about the libcloud roadmap and plans for the
>> future.
>>
>> I would eventually like to see the following features:
>>
>> - non-blocking / asynchronous version of libcloud - I know that this is a
>> major feature and requires a lot of work, but it would be very cool if, in
>> the future you could do something like this:
>>
>> from libcloud.compute.drivers.sync import EC2Driver - imports blocking
>> version of the driver
>> from libcloud.compute.drivers.async import EC2Driver - imports not-blocking
>> version of the driver
>>
>> Since Twisted is pretty much a de-facto standard for creating non-blocking
>> Python applications, I think we should also use it for a non-blocking
>> version of out library.
>>
>> - "Resources" concept - this feature is also be a pretty major one and
>> requires a lot of thought.
>>
>> A "Resource" would be a generic concept which would represent a some kind of
>> "Cloud resource" - a resource could be an IP address, load balancer, etc.
>>
>> This would basically allow us to implement a lot of (currently) provider
>> specific feature, but in a generic way so it would later be easy to adapt it
>> to work with a different provider.
>>
>> - Python 3.x support - Jed has already suggested this on IRC. I actually
>> haven't tested libcloud with Python 3.x yet, so I don't know how many things
>> needs to be changed to make it work (without using 2to3 or a similar tool).
>>
>> Thoughts, opinions?
>>
>> Thanks,
>> Tomaz
>
> --
> Mark Nottingham http://www.mnot.net/
>
>
>
>
Re: [libcloud] libcloud roadmap and plans for the future
Posted by Mark Nottingham <mn...@mnot.net>.
FWIW, my experience is that creating event-driven applications is easier if you just use asyncore and/or epoll directly. Twisted is too complex / over-engineered / under-documented for me (and many others).
On 06/02/2011, at 7:31 AM, Tomaz Muraus wrote:
> I know that the development of the storage API has started just recently,
> but I would still love talk about the libcloud roadmap and plans for the
> future.
>
> I would eventually like to see the following features:
>
> - non-blocking / asynchronous version of libcloud - I know that this is a
> major feature and requires a lot of work, but it would be very cool if, in
> the future you could do something like this:
>
> from libcloud.compute.drivers.sync import EC2Driver - imports blocking
> version of the driver
> from libcloud.compute.drivers.async import EC2Driver - imports not-blocking
> version of the driver
>
> Since Twisted is pretty much a de-facto standard for creating non-blocking
> Python applications, I think we should also use it for a non-blocking
> version of out library.
>
> - "Resources" concept - this feature is also be a pretty major one and
> requires a lot of thought.
>
> A "Resource" would be a generic concept which would represent a some kind of
> "Cloud resource" - a resource could be an IP address, load balancer, etc.
>
> This would basically allow us to implement a lot of (currently) provider
> specific feature, but in a generic way so it would later be easy to adapt it
> to work with a different provider.
>
> - Python 3.x support - Jed has already suggested this on IRC. I actually
> haven't tested libcloud with Python 3.x yet, so I don't know how many things
> needs to be changed to make it work (without using 2to3 or a similar tool).
>
> Thoughts, opinions?
>
> Thanks,
> Tomaz
--
Mark Nottingham http://www.mnot.net/