You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@libcloud.apache.org by Tomaz Muraus <to...@apache.org> on 2013/06/12 07:59:31 UTC
[dev] Libcloud Design Day Recap (improving Libcloud and plans for the future)
First of all I want to say thanks to everyone for coming. We had a lot of
good debates and received some first hand feedback from our users.
Hopefully most of this debates will result into some actual code soon :)
I've included a short recap and some action items bellow. Each section
first contains a short problem description and a solution we agreed upon
afterwards.
*Pricing data distribution*
Currently we bundle JSON file with provider pricing data with every
release. There is no documented and easy way for user to update this file
or for us to ship an updated version without releasing a new version of
Libcloud.
We should provide easy way to update this file which includes the following:
* Command line tool for updating the file - user can specify a custom URL
to the file. By default it should point to the latest version from the
version control (
https://svn.apache.org/viewvc/libcloud/trunk/libcloud/data/pricing.json?view=co
)
* Modify the code so it also tries to find the default pricing data file in
/home/user/.libcloud_data/pricing.json (or whatever) directory
* Document this process and add it to the documentation page
* Note: Most of code for this functionality already exists (
https://github.com/apache/libcloud/blob/trunk/libcloud/pricing.py). We just
need to add a code for downloading a file from a remote server and the cli
tool.
*Exception reporting for partial failures in the methods which result in
multiple API calls / HTTP requests*
*
*
Currently we have no standard interface for exceptions which get raised
during a partial failure in a method which has side affects. Partial
failure means that a function which performs multiple API calls failed half
way through and this potentially resulted in some (but not all) resources
getting created.
We should provide a special exception for cases like this. This exception
should contain information about resources which got created. Users can
then use this information to perform "rollback" / cleanup partially created
resources.
*Support for the async API in the core*
*
*
Currently we only expose sync / blocking API. This mostly works OK for
simple command line scripts, but it's less than ideal for long running
process or applications which performs a lot of API calls which could be
made more efficient and faster if we supported an async api.
As part of this change we would eventually need to refactor every driver
which would also provide other side benefits such as:
* More maintainable code (smaller functions)
* Easier way to handle partial failures / exceptions
We didn't come to a consensus on everything, but we agreed on a couple of
things:
* Change should be backward compatible, we should default to sync API and
existing behavior code / shouldn't change
* New async API should be available separately (e.g. under libcloud.async.*
or whatever it ends up being)
* We should refactor internal code to follow new interfaces defined in the
PEP 3156 (http://www.python.org/dev/peps/pep-3156/)
* We should start with a small scale experiment and try this in a single
method in a single compute driver
*Documentation*
We are weak on the documentation side.
Going forward we should strictly enforce that every patch which adds new
code / functionality contains documentation and appropriate docstrings.
*Migrating to git*
We want to make contributing as easy as possible and SVN doesn't help with
that and increases barrier to entry.
Action item here is me opening an Apache Infrastructure ticket for
switching to git.
*Dropping support for Python 2.5*
Supporting Python 2.5 adds code complexity which we would like to avoid.
Main problem is that a bunch of CLI tools based on Libcloud usually also
run on older versions of Linux (e.g. RHEL 5) which still ship with Python
2.5.
We didn't managed to actually talk about this one, but we should explore an
option of removing support for Python 2.5 in the future release.
If you have attended the meetup, feel free to share your notes and or /
feedback. For other people who didn't attend - you are still more than
welcome and encouraged to share your feedback with us here on the mailing
list.
- Tomaz
Re: [dev] Libcloud Design Day Recap (improving Libcloud and plans for
the future)
Posted by Tomaz Muraus <to...@apache.org>.
Dies, thanks for this insight.
At the meetup, I did actually mention that jclouds recently moved away from
this model (I've been following jclouds Github account and mailing list and
I noticed this discussion there a while ago).
I totally agree about the code maintenance part in general (the less code
the better) and that's why I also want to drop support for Python 2.5 as
soon as possible.
In theory it should be easier to do this in Python, but we will see how it
looks in practice once we implement a prototype.
On Sun, Jun 16, 2013 at 3:37 PM, Koper, Dies <di...@fast.au.fujitsu.com>wrote:
> > *Support for the async API in the core*
> > *
> > *
> > Currently we only expose sync / blocking API. This mostly works OK for
>
> FYI,
> You may be interested to know jclouds had both sync and async APIs and
> recently dropped support for the async APIs.
> It was found that they're a big maintenance burden (keeping the sync and
> sync interfaces in sync), few users actually used them and there were
> good alternatives offered by third-party utility libraries (google's
> guava, which is used in jclouds already anyway) that users can use to
> achieve the same thing.
>
> Maybe the maintenance burden is less with python, Java's strict
> type-checking forced us to have two sets of interfaces, one with actual
> result values and the other returning 'futures' to retrieve the actual
> result values from once the operations complete.
>
> I wasn't involved in the decision to drop async support, it was part of
> a number of refactorings to reduce the size of the codebase and thereby
> its maintenance cost, but if you're interested, you may be able to find
> out more on the jclouds mailing lists.
>
> Regards,
> Dies Koper
>
>
> > -----Original Message-----
> > From: Tomaz Muraus [mailto:tomaz@apache.org]
> > Sent: Wednesday, 12 June 2013 4:00 PM
> > To: users@libcloud.apache.org; dev
> > Subject: [dev] Libcloud Design Day Recap (improving Libcloud and plans
> > for the future)
> >
> > First of all I want to say thanks to everyone for coming. We had a lot
> > of
> > good debates and received some first hand feedback from our users.
> > Hopefully most of this debates will result into some actual code soon
> :)
> >
> > I've included a short recap and some action items bellow. Each section
> > first contains a short problem description and a solution we agreed
> upon
> > afterwards.
> >
> > *Pricing data distribution*
> >
> > Currently we bundle JSON file with provider pricing data with every
> > release. There is no documented and easy way for user to update this
> file
> > or for us to ship an updated version without releasing a new version
> of
> > Libcloud.
> >
> > We should provide easy way to update this file which includes the
> > following:
> >
> > * Command line tool for updating the file - user can specify a custom
> > URL
> > to the file. By default it should point to the latest version from the
> > version control (
> > https://svn.apache.org/viewvc/libcloud/trunk/libcloud/data/pricing.j
> > son?view=co
> > )
> > * Modify the code so it also tries to find the default pricing data
> file
> > in
> > /home/user/.libcloud_data/pricing.json (or whatever) directory
> > * Document this process and add it to the documentation page
> >
> > * Note: Most of code for this functionality already exists (
> > https://github.com/apache/libcloud/blob/trunk/libcloud/pricing.py).
> > We just
> > need to add a code for downloading a file from a remote server and the
> > cli
> > tool.
> >
> > *Exception reporting for partial failures in the methods which result
> > in
> > multiple API calls / HTTP requests*
> > *
> > *
> > Currently we have no standard interface for exceptions which get
> raised
> > during a partial failure in a method which has side affects. Partial
> > failure means that a function which performs multiple API calls failed
> > half
> > way through and this potentially resulted in some (but not all)
> resources
> > getting created.
> >
> > We should provide a special exception for cases like this. This
> exception
> > should contain information about resources which got created. Users
> can
> > then use this information to perform "rollback" / cleanup partially
> > created
> > resources.
> >
> > *Support for the async API in the core*
> > *
> > *
> > Currently we only expose sync / blocking API. This mostly works OK for
> > simple command line scripts, but it's less than ideal for long running
> > process or applications which performs a lot of API calls which could
> > be
> > made more efficient and faster if we supported an async api.
> >
> > As part of this change we would eventually need to refactor every
> driver
> > which would also provide other side benefits such as:
> >
> > * More maintainable code (smaller functions)
> > * Easier way to handle partial failures / exceptions
> >
> > We didn't come to a consensus on everything, but we agreed on a couple
> > of
> > things:
> >
> > * Change should be backward compatible, we should default to sync API
> > and
> > existing behavior code / shouldn't change
> > * New async API should be available separately (e.g. under
> > libcloud.async.*
> > or whatever it ends up being)
> > * We should refactor internal code to follow new interfaces defined in
> > the
> > PEP 3156 (http://www.python.org/dev/peps/pep-3156/)
> > * We should start with a small scale experiment and try this in a
> single
> > method in a single compute driver
> >
> > *Documentation*
> >
> > We are weak on the documentation side.
> >
> > Going forward we should strictly enforce that every patch which adds
> new
> > code / functionality contains documentation and appropriate
> docstrings.
> >
> > *Migrating to git*
> >
> > We want to make contributing as easy as possible and SVN doesn't help
> > with
> > that and increases barrier to entry.
> >
> > Action item here is me opening an Apache Infrastructure ticket for
> > switching to git.
> >
> > *Dropping support for Python 2.5*
> >
> > Supporting Python 2.5 adds code complexity which we would like to
> avoid.
> > Main problem is that a bunch of CLI tools based on Libcloud usually
> also
> > run on older versions of Linux (e.g. RHEL 5) which still ship with
> Python
> > 2.5.
> >
> > We didn't managed to actually talk about this one, but we should
> explore
> > an
> > option of removing support for Python 2.5 in the future release.
> >
> > If you have attended the meetup, feel free to share your notes and or
> > /
> > feedback. For other people who didn't attend - you are still more than
> > welcome and encouraged to share your feedback with us here on the
> mailing
> > list.
> >
> > - Tomaz
>
>
RE: [dev] Libcloud Design Day Recap (improving Libcloud and plans for the future)
Posted by "Koper, Dies" <di...@fast.au.fujitsu.com>.
> *Support for the async API in the core*
> *
> *
> Currently we only expose sync / blocking API. This mostly works OK for
FYI,
You may be interested to know jclouds had both sync and async APIs and
recently dropped support for the async APIs.
It was found that they're a big maintenance burden (keeping the sync and
sync interfaces in sync), few users actually used them and there were
good alternatives offered by third-party utility libraries (google's
guava, which is used in jclouds already anyway) that users can use to
achieve the same thing.
Maybe the maintenance burden is less with python, Java's strict
type-checking forced us to have two sets of interfaces, one with actual
result values and the other returning 'futures' to retrieve the actual
result values from once the operations complete.
I wasn't involved in the decision to drop async support, it was part of
a number of refactorings to reduce the size of the codebase and thereby
its maintenance cost, but if you're interested, you may be able to find
out more on the jclouds mailing lists.
Regards,
Dies Koper
> -----Original Message-----
> From: Tomaz Muraus [mailto:tomaz@apache.org]
> Sent: Wednesday, 12 June 2013 4:00 PM
> To: users@libcloud.apache.org; dev
> Subject: [dev] Libcloud Design Day Recap (improving Libcloud and plans
> for the future)
>
> First of all I want to say thanks to everyone for coming. We had a lot
> of
> good debates and received some first hand feedback from our users.
> Hopefully most of this debates will result into some actual code soon
:)
>
> I've included a short recap and some action items bellow. Each section
> first contains a short problem description and a solution we agreed
upon
> afterwards.
>
> *Pricing data distribution*
>
> Currently we bundle JSON file with provider pricing data with every
> release. There is no documented and easy way for user to update this
file
> or for us to ship an updated version without releasing a new version
of
> Libcloud.
>
> We should provide easy way to update this file which includes the
> following:
>
> * Command line tool for updating the file - user can specify a custom
> URL
> to the file. By default it should point to the latest version from the
> version control (
> https://svn.apache.org/viewvc/libcloud/trunk/libcloud/data/pricing.j
> son?view=co
> )
> * Modify the code so it also tries to find the default pricing data
file
> in
> /home/user/.libcloud_data/pricing.json (or whatever) directory
> * Document this process and add it to the documentation page
>
> * Note: Most of code for this functionality already exists (
> https://github.com/apache/libcloud/blob/trunk/libcloud/pricing.py).
> We just
> need to add a code for downloading a file from a remote server and the
> cli
> tool.
>
> *Exception reporting for partial failures in the methods which result
> in
> multiple API calls / HTTP requests*
> *
> *
> Currently we have no standard interface for exceptions which get
raised
> during a partial failure in a method which has side affects. Partial
> failure means that a function which performs multiple API calls failed
> half
> way through and this potentially resulted in some (but not all)
resources
> getting created.
>
> We should provide a special exception for cases like this. This
exception
> should contain information about resources which got created. Users
can
> then use this information to perform "rollback" / cleanup partially
> created
> resources.
>
> *Support for the async API in the core*
> *
> *
> Currently we only expose sync / blocking API. This mostly works OK for
> simple command line scripts, but it's less than ideal for long running
> process or applications which performs a lot of API calls which could
> be
> made more efficient and faster if we supported an async api.
>
> As part of this change we would eventually need to refactor every
driver
> which would also provide other side benefits such as:
>
> * More maintainable code (smaller functions)
> * Easier way to handle partial failures / exceptions
>
> We didn't come to a consensus on everything, but we agreed on a couple
> of
> things:
>
> * Change should be backward compatible, we should default to sync API
> and
> existing behavior code / shouldn't change
> * New async API should be available separately (e.g. under
> libcloud.async.*
> or whatever it ends up being)
> * We should refactor internal code to follow new interfaces defined in
> the
> PEP 3156 (http://www.python.org/dev/peps/pep-3156/)
> * We should start with a small scale experiment and try this in a
single
> method in a single compute driver
>
> *Documentation*
>
> We are weak on the documentation side.
>
> Going forward we should strictly enforce that every patch which adds
new
> code / functionality contains documentation and appropriate
docstrings.
>
> *Migrating to git*
>
> We want to make contributing as easy as possible and SVN doesn't help
> with
> that and increases barrier to entry.
>
> Action item here is me opening an Apache Infrastructure ticket for
> switching to git.
>
> *Dropping support for Python 2.5*
>
> Supporting Python 2.5 adds code complexity which we would like to
avoid.
> Main problem is that a bunch of CLI tools based on Libcloud usually
also
> run on older versions of Linux (e.g. RHEL 5) which still ship with
Python
> 2.5.
>
> We didn't managed to actually talk about this one, but we should
explore
> an
> option of removing support for Python 2.5 in the future release.
>
> If you have attended the meetup, feel free to share your notes and or
> /
> feedback. For other people who didn't attend - you are still more than
> welcome and encouraged to share your feedback with us here on the
mailing
> list.
>
> - Tomaz
Re: [dev] Libcloud Design Day Recap (improving Libcloud and plans for
the future)
Posted by Tomaz Muraus <to...@apache.org>.
Sorry, I missed this one earlier.
We did discussion capabilities a bit, but we didn't really come to a
conclusion. Besides, that people were more interested in talking and
"fighting" about async support :-)
Another thing which I didn't mention in my first post and we discussed
quite a bit is providers with multiple locations. We mostly agreed that new
datacenter approach is better, but far from ideal and there is still a lot
we can to to make it better.
One of the options we discussed is using a special class instead of a
string value for the "datacenter" argument. This class would contain all
the logic and information which is required to talk to the provider (api
endpoint urls, supported locations, etc.)
There were also a lot of other ideas flying around. There are too many to
sum all of them up in this post, but I will try to write a slightly longer
blog post where I will dive deeper into the ideas we have discussed.
On Wed, Jun 12, 2013 at 12:23 PM, John Carr <jo...@me.com> wrote:
> Hi Tomaz,
>
> Thanks for sending this for those of us that couldn't make it! I've made a
> few comments inline below.
>
> Did you discuss capabilities at all?
>
> Cheers
> John
>
> On 12 Jun 2013, at 06:59, Tomaz Muraus <to...@apache.org> wrote:
>
> > First of all I want to say thanks to everyone for coming. We had a lot of
> > good debates and received some first hand feedback from our users.
> > Hopefully most of this debates will result into some actual code soon :)
> >
> > I've included a short recap and some action items bellow. Each section
> > first contains a short problem description and a solution we agreed upon
> > afterwards.
> >
> > *Pricing data distribution*
> >
> > Currently we bundle JSON file with provider pricing data with every
> > release. There is no documented and easy way for user to update this file
> > or for us to ship an updated version without releasing a new version of
> > Libcloud.
> >
> > We should provide easy way to update this file which includes the
> following:
> >
> > * Command line tool for updating the file - user can specify a custom URL
> > to the file. By default it should point to the latest version from the
> > version control (
> >
> https://svn.apache.org/viewvc/libcloud/trunk/libcloud/data/pricing.json?view=co
> > )
> > * Modify the code so it also tries to find the default pricing data file
> in
> > /home/user/.libcloud_data/pricing.json (or whatever) directory
> > * Document this process and add it to the documentation page
> >
> > * Note: Most of code for this functionality already exists (
> > https://github.com/apache/libcloud/blob/trunk/libcloud/pricing.py). We
> just
> > need to add a code for downloading a file from a remote server and the
> cli
> > tool.
>
> It would be good if apps using this could control where the cached file is
> kept. I'd want my scripts/apps to be able to update the file directly via
> the API too, without needing to use the command line tool.
>
> > *Exception reporting for partial failures in the methods which result in
> > multiple API calls / HTTP requests*
> > *
> > *
> > Currently we have no standard interface for exceptions which get raised
> > during a partial failure in a method which has side affects. Partial
> > failure means that a function which performs multiple API calls failed
> half
> > way through and this potentially resulted in some (but not all) resources
> > getting created.
> >
> > We should provide a special exception for cases like this. This exception
> > should contain information about resources which got created. Users can
> > then use this information to perform "rollback" / cleanup partially
> created
> > resources.
>
> What about having an addCleanup type stack like unittest has?
>
> Regardless of partial success it would be good if there was a mechanism
> for having "it failed, but it is safe to retry" exceptions.
>
> > *Support for the async API in the core*
> > *
> > *
> > Currently we only expose sync / blocking API. This mostly works OK for
> > simple command line scripts, but it's less than ideal for long running
> > process or applications which performs a lot of API calls which could be
> > made more efficient and faster if we supported an async api.
> >
> > As part of this change we would eventually need to refactor every driver
> > which would also provide other side benefits such as:
> >
> > * More maintainable code (smaller functions)
> > * Easier way to handle partial failures / exceptions
> >
> > We didn't come to a consensus on everything, but we agreed on a couple of
> > things:
> >
> > * Change should be backward compatible, we should default to sync API and
> > existing behavior code / shouldn't change
> > * New async API should be available separately (e.g. under
> libcloud.async.*
> > or whatever it ends up being)
> > * We should refactor internal code to follow new interfaces defined in
> the
> > PEP 3156 (http://www.python.org/dev/peps/pep-3156/)
> > * We should start with a small scale experiment and try this in a single
> > method in a single compute driver
>
> Doesn't PEP 3156 depend on 'yield from' (i.e. python 3.3)?
>
> > *Documentation*
> >
> > We are weak on the documentation side.
> >
> > Going forward we should strictly enforce that every patch which adds new
> > code / functionality contains documentation and appropriate docstrings.
>
> A related weakness is enforcing the interfaces we have defined, something
> I hoped to address as part of:
>
> https://issues.apache.org/jira/browse/LIBCLOUD-289
>
> The idea here is that the tests are more about testing the interface, and
> less about testing the implementation. Then each implementation has a stub
> test that tests each individual implementation conforms to the interface.
>
> As you can see in the diff, it identified cases where different
> implementations had different return values and simple things like where
> ports here strings instead of integers.
>
> > *Migrating to git*
> >
> > We want to make contributing as easy as possible and SVN doesn't help
> with
> > that and increases barrier to entry.
> >
> > Action item here is me opening an Apache Infrastructure ticket for
> > switching to git.
> >
> > *Dropping support for Python 2.5*
> >
> > Supporting Python 2.5 adds code complexity which we would like to avoid.
> > Main problem is that a bunch of CLI tools based on Libcloud usually also
> > run on older versions of Linux (e.g. RHEL 5) which still ship with Python
> > 2.5.
> >
> > We didn't managed to actually talk about this one, but we should explore
> an
> > option of removing support for Python 2.5 in the future release.
> >
> > If you have attended the meetup, feel free to share your notes and or /
> > feedback. For other people who didn't attend - you are still more than
> > welcome and encouraged to share your feedback with us here on the mailing
> > list.
> >
> > - Tomaz
>
>
Re: [dev] Libcloud Design Day Recap (improving Libcloud and plans for
the future)
Posted by John Carr <jo...@me.com>.
Hi Tomaz,
Thanks for sending this for those of us that couldn't make it! I've made a few comments inline below.
Did you discuss capabilities at all?
Cheers
John
On 12 Jun 2013, at 06:59, Tomaz Muraus <to...@apache.org> wrote:
> First of all I want to say thanks to everyone for coming. We had a lot of
> good debates and received some first hand feedback from our users.
> Hopefully most of this debates will result into some actual code soon :)
>
> I've included a short recap and some action items bellow. Each section
> first contains a short problem description and a solution we agreed upon
> afterwards.
>
> *Pricing data distribution*
>
> Currently we bundle JSON file with provider pricing data with every
> release. There is no documented and easy way for user to update this file
> or for us to ship an updated version without releasing a new version of
> Libcloud.
>
> We should provide easy way to update this file which includes the following:
>
> * Command line tool for updating the file - user can specify a custom URL
> to the file. By default it should point to the latest version from the
> version control (
> https://svn.apache.org/viewvc/libcloud/trunk/libcloud/data/pricing.json?view=co
> )
> * Modify the code so it also tries to find the default pricing data file in
> /home/user/.libcloud_data/pricing.json (or whatever) directory
> * Document this process and add it to the documentation page
>
> * Note: Most of code for this functionality already exists (
> https://github.com/apache/libcloud/blob/trunk/libcloud/pricing.py). We just
> need to add a code for downloading a file from a remote server and the cli
> tool.
It would be good if apps using this could control where the cached file is kept. I'd want my scripts/apps to be able to update the file directly via the API too, without needing to use the command line tool.
> *Exception reporting for partial failures in the methods which result in
> multiple API calls / HTTP requests*
> *
> *
> Currently we have no standard interface for exceptions which get raised
> during a partial failure in a method which has side affects. Partial
> failure means that a function which performs multiple API calls failed half
> way through and this potentially resulted in some (but not all) resources
> getting created.
>
> We should provide a special exception for cases like this. This exception
> should contain information about resources which got created. Users can
> then use this information to perform "rollback" / cleanup partially created
> resources.
What about having an addCleanup type stack like unittest has?
Regardless of partial success it would be good if there was a mechanism for having "it failed, but it is safe to retry" exceptions.
> *Support for the async API in the core*
> *
> *
> Currently we only expose sync / blocking API. This mostly works OK for
> simple command line scripts, but it's less than ideal for long running
> process or applications which performs a lot of API calls which could be
> made more efficient and faster if we supported an async api.
>
> As part of this change we would eventually need to refactor every driver
> which would also provide other side benefits such as:
>
> * More maintainable code (smaller functions)
> * Easier way to handle partial failures / exceptions
>
> We didn't come to a consensus on everything, but we agreed on a couple of
> things:
>
> * Change should be backward compatible, we should default to sync API and
> existing behavior code / shouldn't change
> * New async API should be available separately (e.g. under libcloud.async.*
> or whatever it ends up being)
> * We should refactor internal code to follow new interfaces defined in the
> PEP 3156 (http://www.python.org/dev/peps/pep-3156/)
> * We should start with a small scale experiment and try this in a single
> method in a single compute driver
Doesn't PEP 3156 depend on 'yield from' (i.e. python 3.3)?
> *Documentation*
>
> We are weak on the documentation side.
>
> Going forward we should strictly enforce that every patch which adds new
> code / functionality contains documentation and appropriate docstrings.
A related weakness is enforcing the interfaces we have defined, something I hoped to address as part of:
https://issues.apache.org/jira/browse/LIBCLOUD-289
The idea here is that the tests are more about testing the interface, and less about testing the implementation. Then each implementation has a stub test that tests each individual implementation conforms to the interface.
As you can see in the diff, it identified cases where different implementations had different return values and simple things like where ports here strings instead of integers.
> *Migrating to git*
>
> We want to make contributing as easy as possible and SVN doesn't help with
> that and increases barrier to entry.
>
> Action item here is me opening an Apache Infrastructure ticket for
> switching to git.
>
> *Dropping support for Python 2.5*
>
> Supporting Python 2.5 adds code complexity which we would like to avoid.
> Main problem is that a bunch of CLI tools based on Libcloud usually also
> run on older versions of Linux (e.g. RHEL 5) which still ship with Python
> 2.5.
>
> We didn't managed to actually talk about this one, but we should explore an
> option of removing support for Python 2.5 in the future release.
>
> If you have attended the meetup, feel free to share your notes and or /
> feedback. For other people who didn't attend - you are still more than
> welcome and encouraged to share your feedback with us here on the mailing
> list.
>
> - Tomaz
Re: [dev] Libcloud Design Day Recap (improving Libcloud and plans for
the future)
Posted by Tomaz Muraus <to...@apache.org>.
Quick response about the git migration - this means we would move to Apache
hosted git repository.
This still decreases barrier to entry, because people who are already using
git can use familiar branching model and Git has a good support for
multiple remotes which means people can still push their code to Github if
they wish.
On Wed, Jun 12, 2013 at 10:16 AM, Mahendra M <ma...@gmail.com> wrote:
> Hi Tomaz,
>
> Thanks a lot for the detailed summary :-)
> Was really sad about being unable to attend this meetup (being on the other
> side of the globe).
>
> Please see inline.
>
> On Wed, Jun 12, 2013 at 11:29 AM, Tomaz Muraus <to...@apache.org> wrote:
>
> > *Exception reporting for partial failures in the methods which result in
> > multiple API calls / HTTP requests*
> > Currently we have no standard interface for exceptions which get raised
> > during a partial failure in a method which has side affects. Partial
> > failure means that a function which performs multiple API calls failed
> half
> > way through and this potentially resulted in some (but not all) resources
> > getting created.
> >
> > We should provide a special exception for cases like this. This exception
> > should contain information about resources which got created. Users can
> > then use this information to perform "rollback" / cleanup partially
> created
> > resources.
>
>
> +1 to this feature - specially in cases of storage :-) - we can provide
> apis to sync folders with cloud storage.
>
> *Documentation*
> >
> > We are weak on the documentation side.
> >
> > Going forward we should strictly enforce that every patch which adds new
> > code / functionality contains documentation and appropriate docstrings.
> >
>
> Agree with that. I am one of the culprits in this crime :-)
> We could move the documentation along with the code (say under a docs
> directory).
> So any user who sends a patch will have to include the doc changes in the
> same pull request.
>
> This will ensure that doc changes are also part of the git/review workflow.
>
> Celery/Kombu projects do this and it is quite effective.
>
> *Migrating to git*
> >
> > We want to make contributing as easy as possible and SVN doesn't help
> with
> > that and increases barrier to entry.
> >
> > Action item here is me opening an Apache Infrastructure ticket for
> > switching to git.
> >
>
> Yaaay! Good news! Let me know if you need any help in this.
> Would this mean we will use Apache hosted git repo or can we move
> completely to github?
> I find github to be better for code reviews.
>
>
> > *Dropping support for Python 2.5*
> >
> > Supporting Python 2.5 adds code complexity which we would like to avoid.
> > Main problem is that a bunch of CLI tools based on Libcloud usually also
> > run on older versions of Linux (e.g. RHEL 5) which still ship with Python
> > 2.5.
> >
>
> This would be welcome. I would suggest that we do a small poll among the
> users to see if this would be an issue (before we make a change).
>
> Regards,
> Mahendra
>
> http://twitter.com/mahendra
>
Re: [dev] Libcloud Design Day Recap (improving Libcloud and plans for
the future)
Posted by Mahendra M <ma...@gmail.com>.
Hi Tomaz,
Thanks a lot for the detailed summary :-)
Was really sad about being unable to attend this meetup (being on the other
side of the globe).
Please see inline.
On Wed, Jun 12, 2013 at 11:29 AM, Tomaz Muraus <to...@apache.org> wrote:
> *Exception reporting for partial failures in the methods which result in
> multiple API calls / HTTP requests*
> Currently we have no standard interface for exceptions which get raised
> during a partial failure in a method which has side affects. Partial
> failure means that a function which performs multiple API calls failed half
> way through and this potentially resulted in some (but not all) resources
> getting created.
>
> We should provide a special exception for cases like this. This exception
> should contain information about resources which got created. Users can
> then use this information to perform "rollback" / cleanup partially created
> resources.
+1 to this feature - specially in cases of storage :-) - we can provide
apis to sync folders with cloud storage.
*Documentation*
>
> We are weak on the documentation side.
>
> Going forward we should strictly enforce that every patch which adds new
> code / functionality contains documentation and appropriate docstrings.
>
Agree with that. I am one of the culprits in this crime :-)
We could move the documentation along with the code (say under a docs
directory).
So any user who sends a patch will have to include the doc changes in the
same pull request.
This will ensure that doc changes are also part of the git/review workflow.
Celery/Kombu projects do this and it is quite effective.
*Migrating to git*
>
> We want to make contributing as easy as possible and SVN doesn't help with
> that and increases barrier to entry.
>
> Action item here is me opening an Apache Infrastructure ticket for
> switching to git.
>
Yaaay! Good news! Let me know if you need any help in this.
Would this mean we will use Apache hosted git repo or can we move
completely to github?
I find github to be better for code reviews.
> *Dropping support for Python 2.5*
>
> Supporting Python 2.5 adds code complexity which we would like to avoid.
> Main problem is that a bunch of CLI tools based on Libcloud usually also
> run on older versions of Linux (e.g. RHEL 5) which still ship with Python
> 2.5.
>
This would be welcome. I would suggest that we do a small poll among the
users to see if this would be an issue (before we make a change).
Regards,
Mahendra
http://twitter.com/mahendra
Re: [dev] Libcloud Design Day Recap (improving Libcloud and plans
for the future)
Posted by Trevor Powell <Tr...@rms.com>.
Excellent! Glad we put on this meet up. I look forward to the next one :-)
--
<http://www.rms.com/>
Trevor Alexander Powell
Principal Systems Engineer, Cloud Architecture
7575 Gateway Blvd. Newark, CA 94560
T: +1.510.713.3751
M: +1.650.325.7467
www.rms.com <http://www.rms.com/>
On 6/11/13 10:59 PM, "Tomaz Muraus" <to...@apache.org> wrote:
>First of all I want to say thanks to everyone for coming. We had a lot of
>good debates and received some first hand feedback from our users.
>Hopefully most of this debates will result into some actual code soon :)
>
>I've included a short recap and some action items bellow. Each section
>first contains a short problem description and a solution we agreed upon
>afterwards.
>
>*Pricing data distribution*
>
>Currently we bundle JSON file with provider pricing data with every
>release. There is no documented and easy way for user to update this file
>or for us to ship an updated version without releasing a new version of
>Libcloud.
>
>We should provide easy way to update this file which includes the
>following:
>
>* Command line tool for updating the file - user can specify a custom URL
>to the file. By default it should point to the latest version from the
>version control (
>https://svn.apache.org/viewvc/libcloud/trunk/libcloud/data/pricing.json?vi
>ew=co
>)
>* Modify the code so it also tries to find the default pricing data file
>in
>/home/user/.libcloud_data/pricing.json (or whatever) directory
>* Document this process and add it to the documentation page
>
>* Note: Most of code for this functionality already exists (
>https://github.com/apache/libcloud/blob/trunk/libcloud/pricing.py). We
>just
>need to add a code for downloading a file from a remote server and the cli
>tool.
>
>*Exception reporting for partial failures in the methods which result in
>multiple API calls / HTTP requests*
>*
>*
>Currently we have no standard interface for exceptions which get raised
>during a partial failure in a method which has side affects. Partial
>failure means that a function which performs multiple API calls failed
>half
>way through and this potentially resulted in some (but not all) resources
>getting created.
>
>We should provide a special exception for cases like this. This exception
>should contain information about resources which got created. Users can
>then use this information to perform "rollback" / cleanup partially
>created
>resources.
>
>*Support for the async API in the core*
>*
>*
>Currently we only expose sync / blocking API. This mostly works OK for
>simple command line scripts, but it's less than ideal for long running
>process or applications which performs a lot of API calls which could be
>made more efficient and faster if we supported an async api.
>
>As part of this change we would eventually need to refactor every driver
>which would also provide other side benefits such as:
>
>* More maintainable code (smaller functions)
>* Easier way to handle partial failures / exceptions
>
>We didn't come to a consensus on everything, but we agreed on a couple of
>things:
>
>* Change should be backward compatible, we should default to sync API and
>existing behavior code / shouldn't change
>* New async API should be available separately (e.g. under
>libcloud.async.*
>or whatever it ends up being)
>* We should refactor internal code to follow new interfaces defined in the
>PEP 3156 (http://www.python.org/dev/peps/pep-3156/)
>* We should start with a small scale experiment and try this in a single
>method in a single compute driver
>
>*Documentation*
>
>We are weak on the documentation side.
>
>Going forward we should strictly enforce that every patch which adds new
>code / functionality contains documentation and appropriate docstrings.
>
>*Migrating to git*
>
>We want to make contributing as easy as possible and SVN doesn't help with
>that and increases barrier to entry.
>
>Action item here is me opening an Apache Infrastructure ticket for
>switching to git.
>
>*Dropping support for Python 2.5*
>
>Supporting Python 2.5 adds code complexity which we would like to avoid.
>Main problem is that a bunch of CLI tools based on Libcloud usually also
>run on older versions of Linux (e.g. RHEL 5) which still ship with Python
>2.5.
>
>We didn't managed to actually talk about this one, but we should explore
>an
>option of removing support for Python 2.5 in the future release.
>
>If you have attended the meetup, feel free to share your notes and or /
>feedback. For other people who didn't attend - you are still more than
>welcome and encouraged to share your feedback with us here on the mailing
>list.
>
>- Tomaz
This message and any attachments contain information that may be RMS Inc. confidential and/or privileged. If you are not the intended recipient (or authorized to receive for the intended recipient), and have received this message in error, any use, disclosure or distribution is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to the e-mail and permanently deleting the message from your computer and/or storage system.