You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@libcloud.apache.org by anthony shaw <an...@apache.org> on 2018/03/08 23:16:09 UTC
[dev] [DISCUSS] Future of Apache Libcloud
Hi All,
The "cloud market" when Apache Libcloud was conceived in 2010 is very
different to how it looks today, some trends we are seeing
1- IaaS (our compute API) is one of many features in public clouds. Amazon,
Azure and GCP have 100's of individual services now [1]. Text-to-speech,
functions, automation, API gateways, it's impossible to keep up
2- Private Clouds have seen a continued decline but are still popular [2]
3- The advent of containers means it is now "easier" to deploy an
application to multiple clouds
4- The big 3 public clouds, Amazon, Azure and Google make up most of the
cloud market [3] if you compare Apache Libcloud downloads with boto (the
native AWS Python client) downloads, it's a massive delta. Boto is in the
top 10 most popular PyPi packages
In terms of users, I've pulled 3 snapshots of PyPi downloads, January 2016,
2017 and 2018 [4]
Annual downloads of Apache-Libcloud have seen a slight increase, but the
2016 - 2,778,687
2017 - 2,958,591
Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is a huge
drop. [4]
*I would like to propose a drastic (depending on your perspective) plan to
take Apache Libcloud through to 2020*
*1. Focus on specific use cases*
Lack of consistency between implementations of the base class means in
practicality it's difficult to have abstracted applications.
Apache Libcloud should (imo) come with a toolbox of utilities to both
demonstrate and validate cloud abstraction use cases, such as:
- Migrating storage objects from Cloud X to Cloud Y
- Splitting an application workload across multiple clouds
*2. Improve performance by adopting asyncio*
In almost all use cases, Libcloud would benefit from non-blocking calls.
Listing VM's requires multiple calls for the pages, uploading storage
objects can be done in multiple futures, deleting DNS records would be
better done in async.
I'm suggesting we introduce a Python 3.5+ only API, move to
requests-futures or aiohttp for the base HTTP client. Yes, *I am suggesting
we drop Python 2 support in the future*.
I've been researching how we could make this switch without breaking 64% of
our users..
Pip now has a way to choose versions based on Python runtimes
https://hackernoon.com/phasing-out-python-runtimes-gracefully-956f112f33c4
We could have apache-libcloud 3+ for Python 3.5 users and then maintain
2.3 patches for Python 2.7 and 3.4 users.
I think we can come up with a way of continuing to maintain the existing
code base for 2.7 users but move forward with a new API for async and
Python 3.5+ users.
The downloads [4] show that Python 2.7 is still the majority of users but
this is declining quickly and by 2020 the tables will have turned. We need
to be ready for that.
*3. Change our positioning on dependencies of 3rd party packages*
We aren't seeing enough community contribution to keep up with the rapid
pace at which Microsoft, Amazon and Google are changing their APIs. Google
have been contributing to their driver for years. We haven't seen that from
either Amazon or Microsoft. Alibaba have contributed to theirs, many other
cloud providers have contributed, but it seems to be after APIs are
changed, not in advance.
* 1. Docker is another example, that API changes almost every month. The
driver we have is unstable and doesn't support API versioning correctly. *
Please consider these options,
I would like to hear from users how you are currently using Apache Libcloud
and how you are using it
NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly
because they represent +80% of the cloud market [3]. This is a community
developed project, has no affiliation to those vendors and these are my
opinions, not those of the project.
[1] https://www.amazonaws.cn/en/products/
[2]
http://www.computerweekly.com/news/450280991/IDC-research-predicts-gradual-decline-in-on-premise-hardware-spend-as-cloud-adoption-rises
[3]
https://www.crn.com.au/news/microsoft-ate-into-aws-market-share-in-q4-481240
[4]
https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-vPIYPer2VFtSo_Q/edit?usp=sharing
Re: [dev] [DISCUSS] Future of Apache Libcloud
Posted by anthony shaw <an...@gmail.com>.
I also want to reiterate that I'm not bashing on the current design. It
absolutely serves a purpose, has many happy users and delivers on many of
the goals initially set out.
But, things have changed hugely in 5 years.
--
John >>
"primitives for waiting, paginating, retrying, etc" < absolutely. Each
driver does it differently today and this is a requirement across all. Most
APIs should give page count on the first request or support a HEAD request
so pages should be able to be fetched in parallel.
Jed >>
"A data-driven approach to generating bindings is really, really
interesting"
+1. I'd spent a bit of time trying to understand how Microsoft build their
Python SDK and it's almost entirely automated now, see
https://github.com/Azure/azure-sdk-for-python/blob/master/swagger_to_sdk_config.json
for how they use autorest to create bindings.
I have honestly tried to build out the Azure driver and it improve it. It
ended up costing me $100's and I broke the API.
---
Another project for us to look into is Terraform.
Terraform doesn't do any abstraction, but it does more closely declare the
dependencies between infrastructure components.
Within Libcloud today there isn't a way to declare which infrastructure
assets any ex_ methods are provisioning, fetching or updating. What
attributes those assets should have and their behaviours or dependencies.
Therefore, trying to do something like Terraform would be hugely
challenging. A colleague of mine did try on this project -
https://github.com/DimensionDataCBUSydney/plumbery
On Fri, Mar 9, 2018 at 12:18 PM, Jed Smith <je...@jedsmith.org> wrote:
> + a million to John, and I’m actually in the same boat. I stepped away
> due to a conflict, though I technically remain on the PMC (it seems).
> Every time I’ve wanted to come back the current design stops me, and
> that’s not speaking negatively, just that I have a difficult time
> approaching modern libcloud.
> I work for a company much more supportive of open source now, and I’d
> love to put my volunteer hours into libcloud again. I agree with
> everything architectural discussed in this thread, and moving that
> direction would rouse me from hibernation.
> What would I add to the discussion is that the individual layers should
> strive for requests-level simplicity and thought. I agree the ex_
> approach is offputting. I think Python, especially modern Python, is
> expressive enough to do this aspect well.
> A data-driven approach to generating bindings is really, really
> interesting. If that ends up being too bold for libcloud, I might help
> explore that separately. That’s a really interesting idea.
> -J
>
>
> On Thu, Mar 8, 2018, at 19:18, John Carr wrote:
> > Hi Anthony,
> >
> > I used to be an active contributor to libcloud, and am
> > technically still> a committer although I don’t think I’ve been active
> at all since you
> > stepped into the fold. I just wanted to start by saying thanks
> > for your> hard work!
> >
> > I fully support your proposal. It will take a while to be ready
> > to fully> exploit py3, we should get started.
> >
> > For me what went wrong in the current codebase is that the common
> > interfaces weren’t common enough. Even within the simple use
> > cases there> were often paper cuts (inconsistencies in exceptions
> raised). And the> ex_ approach was off-putting too.
> >
> > Ultimately I ended up only targeting AWS and it was a no
> > brainer to use> botocore.
> >
> > An asyncio libcloud could be really compelling - but it should learn a>
> few tricks from boto/botocore.
> >
> > I think there should be a low level API that targets the actual API as>
> closely as possible. It should provide some commonality -
> > primitives for> waiting, paginating, retrying, etc. Maybe some well known
> > exceptions. It> should probably map to basic types rather than exposing
> XML.
> > Fundamentally AWS APIs are inconsistent but these botocore
> > utilities and> patterns make up for it mostly.
> >
> > High level abstractions should be built on top of this layer
> > and only do> what is actually common between APIs - no iffy ex_ params.
> >
> > I wonder if we can make libcloud more attractive to providers
> > wanting to> write a python sdk?
> >
> > One idea I had a long time ago was to see how well the botocore data
> > driven model could extend to other providers. I don’t really think we>
> have the resources to do this. But the thinking was we could generate>
> requests, asyncio and twisted bindings from the same data and
> > potentially share the data with an node cloud driver for example.
> >
> > Cheers
> > John
> >
> > > On 8 Mar 2018, at 23:16, anthony shaw <an...@apache.org>
> > > wrote:> >
> > > Hi All,
> > >
> > > The "cloud market" when Apache Libcloud was conceived in 2010
> > > is very> > different to how it looks today, some trends we are seeing
> > >
> > > 1- IaaS (our compute API) is one of many features in public clouds.
> > > Amazon,> > Azure and GCP have 100's of individual services now [1].
> Text-to-
> > > speech,> > functions, automation, API gateways, it's impossible to
> keep up
> > > 2- Private Clouds have seen a continued decline but are still
> > > popular [2]> > 3- The advent of containers means it is now "easier" to
> deploy an
> > > application to multiple clouds
> > > 4- The big 3 public clouds, Amazon, Azure and Google make up most
> > > of the> > cloud market [3] if you compare Apache Libcloud downloads
> with
> > > boto (the> > native AWS Python client) downloads, it's a massive
> delta. Boto is
> > > in the> > top 10 most popular PyPi packages
> > >
> > > In terms of users, I've pulled 3 snapshots of PyPi downloads,
> > > January 2016,> > 2017 and 2018 [4]
> > > Annual downloads of Apache-Libcloud have seen a slight increase,
> > > but the> > 2016 - 2,778,687
> > > 2017 - 2,958,591
> > >
> > > Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is
> > > a huge> > drop. [4]
> > >
> > > *I would like to propose a drastic (depending on your perspective)
> > > plan to> > take Apache Libcloud through to 2020*
> > >
> > > *1. Focus on specific use cases*
> > >
> > > Lack of consistency between implementations of the base class
> > > means in> > practicality it's difficult to have abstracted
> applications.
> > >
> > > Apache Libcloud should (imo) come with a toolbox of utilities
> > > to both> > demonstrate and validate cloud abstraction use cases, such
> as:
> > > - Migrating storage objects from Cloud X to Cloud Y
> > > - Splitting an application workload across multiple clouds
> > >
> > > *2. Improve performance by adopting asyncio*
> > >
> > > In almost all use cases, Libcloud would benefit from non-blocking
> > > calls.> > Listing VM's requires multiple calls for the pages, uploading
> > > storage> > objects can be done in multiple futures, deleting DNS
> records
> > > would be> > better done in async.
> > >
> > > I'm suggesting we introduce a Python 3.5+ only API, move to
> > > requests-futures or aiohttp for the base HTTP client. Yes, *I am
> > > suggesting> > we drop Python 2 support in the future*.
> > >
> > > I've been researching how we could make this switch without breaking
> > > 64% of> > our users..
> > > Pip now has a way to choose versions based on Python runtimes
> > > https://hackernoon.com/phasing-out-python-runtimes-
> gracefully-956f112f33c4> > We could have apache-libcloud 3+ for Python
> 3.5 users and then
> > > maintain> > 2.3 patches for Python 2.7 and 3.4 users.
> > >
> > > I think we can come up with a way of continuing to maintain the
> > > existing> > code base for 2.7 users but move forward with a new API for
> > > async and> > Python 3.5+ users.
> > >
> > > The downloads [4] show that Python 2.7 is still the majority of
> > > users but> > this is declining quickly and by 2020 the tables will
> have turned.
> > > We need> > to be ready for that.
> > >
> > > *3. Change our positioning on dependencies of 3rd party packages*
> > >
> > > We aren't seeing enough community contribution to keep up with
> > > the rapid> > pace at which Microsoft, Amazon and Google are changing
> their APIs.
> > > Google> > have been contributing to their driver for years. We haven't
> seen
> > > that from> > either Amazon or Microsoft. Alibaba have contributed to
> theirs,
> > > many other> > cloud providers have contributed, but it seems to be
> after APIs are> > changed, not in advance.
> > >
> > >
> > > * 1. Docker is another example, that API changes almost every
> > > month. The> > driver we have is unstable and doesn't support API
> versioning
> > > correctly. *> > Please consider these options,
> > >
> > > I would like to hear from users how you are currently using Apache
> > > Libcloud> > and how you are using it
> > >
> > > NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly>
> > because they represent +80% of the cloud market [3]. This is a
> > > community> > developed project, has no affiliation to those vendors
> and these
> > > are my> > opinions, not those of the project.
> > >
> > > [1] https://www.amazonaws.cn/en/products/
> > > [2]
> > > http://www.computerweekly.com/news/450280991/IDC-research-
> predicts-gradual-decline-in-on-premise-hardware-spend-as-
> cloud-adoption-rises> > [3]
> > > https://www.crn.com.au/news/microsoft-ate-into-aws-market-
> share-in-q4-481240> > [4]
> > > https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-
> vPIYPer2VFtSo_Q/edit?usp=sharing>
>
>
Re: [dev] [DISCUSS] Future of Apache Libcloud
Posted by Samuel Marks <sa...@gmail.com>.
Hmm, I would actually go further and rebuild libcloud as a queue-oriented
multithreaded multiprocessing asynchronous server.
Samuel Marks
http://linkedin.com/in/samuelmarks
On Fri, Mar 9, 2018 at 7:23 PM, John Carr <jo...@unrouted.co.uk> wrote:
> On 9 Mar 2018, at 01:18, Jed Smith <je...@jedsmith.org> wrote:
> >
> > A data-driven approach to generating bindings is really, really
> > interesting. If that ends up being too bold for libcloud, I might help
> > explore that separately. That’s a really interesting idea.
> > -J
> >
>
> Here’s a prototype I started working on in 2015. I didn’t get very far,
> but if you check out the tests I did have one code base supporting sync and
> async. I had a middleware-esque approach where the data described which
> serialisers, request signers, etc to mash together.
>
> https://github.com/Jc2k/libcloudcore <https://github.com/Jc2k/libcloudcore
> >
>
> John
>
> >
> > On Thu, Mar 8, 2018, at 19:18, John Carr wrote:
> >> Hi Anthony,
> >>
> >> I used to be an active contributor to libcloud, and am
> >> technically still> a committer although I don’t think I’ve been active
> at all since you
> >> stepped into the fold. I just wanted to start by saying thanks
> >> for your> hard work!
> >>
> >> I fully support your proposal. It will take a while to be ready
> >> to fully> exploit py3, we should get started.
> >>
> >> For me what went wrong in the current codebase is that the common
> >> interfaces weren’t common enough. Even within the simple use
> >> cases there> were often paper cuts (inconsistencies in exceptions
> raised). And the> ex_ approach was off-putting too.
> >>
> >> Ultimately I ended up only targeting AWS and it was a no
> >> brainer to use> botocore.
> >>
> >> An asyncio libcloud could be really compelling - but it should learn a>
> few tricks from boto/botocore.
> >>
> >> I think there should be a low level API that targets the actual API as>
> closely as possible. It should provide some commonality -
> >> primitives for> waiting, paginating, retrying, etc. Maybe some well
> known
> >> exceptions. It> should probably map to basic types rather than exposing
> XML.
> >> Fundamentally AWS APIs are inconsistent but these botocore
> >> utilities and> patterns make up for it mostly.
> >>
> >> High level abstractions should be built on top of this layer
> >> and only do> what is actually common between APIs - no iffy ex_ params.
> >>
> >> I wonder if we can make libcloud more attractive to providers
> >> wanting to> write a python sdk?
> >>
> >> One idea I had a long time ago was to see how well the botocore data
> >> driven model could extend to other providers. I don’t really think we>
> have the resources to do this. But the thinking was we could generate>
> requests, asyncio and twisted bindings from the same data and
> >> potentially share the data with an node cloud driver for example.
> >>
> >> Cheers
> >> John
> >>
> >>> On 8 Mar 2018, at 23:16, anthony shaw <an...@apache.org>
> >>> wrote:> >
> >>> Hi All,
> >>>
> >>> The "cloud market" when Apache Libcloud was conceived in 2010
> >>> is very> > different to how it looks today, some trends we are seeing
> >>>
> >>> 1- IaaS (our compute API) is one of many features in public clouds.
> >>> Amazon,> > Azure and GCP have 100's of individual services now [1].
> Text-to-
> >>> speech,> > functions, automation, API gateways, it's impossible to
> keep up
> >>> 2- Private Clouds have seen a continued decline but are still
> >>> popular [2]> > 3- The advent of containers means it is now "easier" to
> deploy an
> >>> application to multiple clouds
> >>> 4- The big 3 public clouds, Amazon, Azure and Google make up most
> >>> of the> > cloud market [3] if you compare Apache Libcloud downloads
> with
> >>> boto (the> > native AWS Python client) downloads, it's a massive
> delta. Boto is
> >>> in the> > top 10 most popular PyPi packages
> >>>
> >>> In terms of users, I've pulled 3 snapshots of PyPi downloads,
> >>> January 2016,> > 2017 and 2018 [4]
> >>> Annual downloads of Apache-Libcloud have seen a slight increase,
> >>> but the> > 2016 - 2,778,687
> >>> 2017 - 2,958,591
> >>>
> >>> Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is
> >>> a huge> > drop. [4]
> >>>
> >>> *I would like to propose a drastic (depending on your perspective)
> >>> plan to> > take Apache Libcloud through to 2020*
> >>>
> >>> *1. Focus on specific use cases*
> >>>
> >>> Lack of consistency between implementations of the base class
> >>> means in> > practicality it's difficult to have abstracted
> applications.
> >>>
> >>> Apache Libcloud should (imo) come with a toolbox of utilities
> >>> to both> > demonstrate and validate cloud abstraction use cases, such
> as:
> >>> - Migrating storage objects from Cloud X to Cloud Y
> >>> - Splitting an application workload across multiple clouds
> >>>
> >>> *2. Improve performance by adopting asyncio*
> >>>
> >>> In almost all use cases, Libcloud would benefit from non-blocking
> >>> calls.> > Listing VM's requires multiple calls for the pages, uploading
> >>> storage> > objects can be done in multiple futures, deleting DNS
> records
> >>> would be> > better done in async.
> >>>
> >>> I'm suggesting we introduce a Python 3.5+ only API, move to
> >>> requests-futures or aiohttp for the base HTTP client. Yes, *I am
> >>> suggesting> > we drop Python 2 support in the future*.
> >>>
> >>> I've been researching how we could make this switch without breaking
> >>> 64% of> > our users..
> >>> Pip now has a way to choose versions based on Python runtimes
> >>> https://hackernoon.com/phasing-out-python-runtimes-
> gracefully-956f112f33c4> > We could have apache-libcloud 3+ for Python
> 3.5 users and then
> >>> maintain> > 2.3 patches for Python 2.7 and 3.4 users.
> >>>
> >>> I think we can come up with a way of continuing to maintain the
> >>> existing> > code base for 2.7 users but move forward with a new API for
> >>> async and> > Python 3.5+ users.
> >>>
> >>> The downloads [4] show that Python 2.7 is still the majority of
> >>> users but> > this is declining quickly and by 2020 the tables will
> have turned.
> >>> We need> > to be ready for that.
> >>>
> >>> *3. Change our positioning on dependencies of 3rd party packages*
> >>>
> >>> We aren't seeing enough community contribution to keep up with
> >>> the rapid> > pace at which Microsoft, Amazon and Google are changing
> their APIs.
> >>> Google> > have been contributing to their driver for years. We haven't
> seen
> >>> that from> > either Amazon or Microsoft. Alibaba have contributed to
> theirs,
> >>> many other> > cloud providers have contributed, but it seems to be
> after APIs are> > changed, not in advance.
> >>>
> >>>
> >>> * 1. Docker is another example, that API changes almost every
> >>> month. The> > driver we have is unstable and doesn't support API
> versioning
> >>> correctly. *> > Please consider these options,
> >>>
> >>> I would like to hear from users how you are currently using Apache
> >>> Libcloud> > and how you are using it
> >>>
> >>> NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly>
> > because they represent +80% of the cloud market [3]. This is a
> >>> community> > developed project, has no affiliation to those vendors
> and these
> >>> are my> > opinions, not those of the project.
> >>>
> >>> [1] https://www.amazonaws.cn/en/products/
> >>> [2]
> >>> http://www.computerweekly.com/news/450280991/IDC-research-
> predicts-gradual-decline-in-on-premise-hardware-spend-as-
> cloud-adoption-rises> > [3]
> >>> https://www.crn.com.au/news/microsoft-ate-into-aws-market-
> share-in-q4-481240> > [4]
> >>> https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-
> vPIYPer2VFtSo_Q/edit?usp=sharing>
> >
>
>
Re: [dev] [DISCUSS] Future of Apache Libcloud
Posted by John Carr <jo...@unrouted.co.uk>.
On 9 Mar 2018, at 01:18, Jed Smith <je...@jedsmith.org> wrote:
>
> A data-driven approach to generating bindings is really, really
> interesting. If that ends up being too bold for libcloud, I might help
> explore that separately. That’s a really interesting idea.
> -J
>
Here’s a prototype I started working on in 2015. I didn’t get very far, but if you check out the tests I did have one code base supporting sync and async. I had a middleware-esque approach where the data described which serialisers, request signers, etc to mash together.
https://github.com/Jc2k/libcloudcore <https://github.com/Jc2k/libcloudcore>
John
>
> On Thu, Mar 8, 2018, at 19:18, John Carr wrote:
>> Hi Anthony,
>>
>> I used to be an active contributor to libcloud, and am
>> technically still> a committer although I don’t think I’ve been active at all since you
>> stepped into the fold. I just wanted to start by saying thanks
>> for your> hard work!
>>
>> I fully support your proposal. It will take a while to be ready
>> to fully> exploit py3, we should get started.
>>
>> For me what went wrong in the current codebase is that the common
>> interfaces weren’t common enough. Even within the simple use
>> cases there> were often paper cuts (inconsistencies in exceptions raised). And the> ex_ approach was off-putting too.
>>
>> Ultimately I ended up only targeting AWS and it was a no
>> brainer to use> botocore.
>>
>> An asyncio libcloud could be really compelling - but it should learn a> few tricks from boto/botocore.
>>
>> I think there should be a low level API that targets the actual API as> closely as possible. It should provide some commonality -
>> primitives for> waiting, paginating, retrying, etc. Maybe some well known
>> exceptions. It> should probably map to basic types rather than exposing XML.
>> Fundamentally AWS APIs are inconsistent but these botocore
>> utilities and> patterns make up for it mostly.
>>
>> High level abstractions should be built on top of this layer
>> and only do> what is actually common between APIs - no iffy ex_ params.
>>
>> I wonder if we can make libcloud more attractive to providers
>> wanting to> write a python sdk?
>>
>> One idea I had a long time ago was to see how well the botocore data
>> driven model could extend to other providers. I don’t really think we> have the resources to do this. But the thinking was we could generate> requests, asyncio and twisted bindings from the same data and
>> potentially share the data with an node cloud driver for example.
>>
>> Cheers
>> John
>>
>>> On 8 Mar 2018, at 23:16, anthony shaw <an...@apache.org>
>>> wrote:> >
>>> Hi All,
>>>
>>> The "cloud market" when Apache Libcloud was conceived in 2010
>>> is very> > different to how it looks today, some trends we are seeing
>>>
>>> 1- IaaS (our compute API) is one of many features in public clouds.
>>> Amazon,> > Azure and GCP have 100's of individual services now [1]. Text-to-
>>> speech,> > functions, automation, API gateways, it's impossible to keep up
>>> 2- Private Clouds have seen a continued decline but are still
>>> popular [2]> > 3- The advent of containers means it is now "easier" to deploy an
>>> application to multiple clouds
>>> 4- The big 3 public clouds, Amazon, Azure and Google make up most
>>> of the> > cloud market [3] if you compare Apache Libcloud downloads with
>>> boto (the> > native AWS Python client) downloads, it's a massive delta. Boto is
>>> in the> > top 10 most popular PyPi packages
>>>
>>> In terms of users, I've pulled 3 snapshots of PyPi downloads,
>>> January 2016,> > 2017 and 2018 [4]
>>> Annual downloads of Apache-Libcloud have seen a slight increase,
>>> but the> > 2016 - 2,778,687
>>> 2017 - 2,958,591
>>>
>>> Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is
>>> a huge> > drop. [4]
>>>
>>> *I would like to propose a drastic (depending on your perspective)
>>> plan to> > take Apache Libcloud through to 2020*
>>>
>>> *1. Focus on specific use cases*
>>>
>>> Lack of consistency between implementations of the base class
>>> means in> > practicality it's difficult to have abstracted applications.
>>>
>>> Apache Libcloud should (imo) come with a toolbox of utilities
>>> to both> > demonstrate and validate cloud abstraction use cases, such as:
>>> - Migrating storage objects from Cloud X to Cloud Y
>>> - Splitting an application workload across multiple clouds
>>>
>>> *2. Improve performance by adopting asyncio*
>>>
>>> In almost all use cases, Libcloud would benefit from non-blocking
>>> calls.> > Listing VM's requires multiple calls for the pages, uploading
>>> storage> > objects can be done in multiple futures, deleting DNS records
>>> would be> > better done in async.
>>>
>>> I'm suggesting we introduce a Python 3.5+ only API, move to
>>> requests-futures or aiohttp for the base HTTP client. Yes, *I am
>>> suggesting> > we drop Python 2 support in the future*.
>>>
>>> I've been researching how we could make this switch without breaking
>>> 64% of> > our users..
>>> Pip now has a way to choose versions based on Python runtimes
>>> https://hackernoon.com/phasing-out-python-runtimes-gracefully-956f112f33c4> > We could have apache-libcloud 3+ for Python 3.5 users and then
>>> maintain> > 2.3 patches for Python 2.7 and 3.4 users.
>>>
>>> I think we can come up with a way of continuing to maintain the
>>> existing> > code base for 2.7 users but move forward with a new API for
>>> async and> > Python 3.5+ users.
>>>
>>> The downloads [4] show that Python 2.7 is still the majority of
>>> users but> > this is declining quickly and by 2020 the tables will have turned.
>>> We need> > to be ready for that.
>>>
>>> *3. Change our positioning on dependencies of 3rd party packages*
>>>
>>> We aren't seeing enough community contribution to keep up with
>>> the rapid> > pace at which Microsoft, Amazon and Google are changing their APIs.
>>> Google> > have been contributing to their driver for years. We haven't seen
>>> that from> > either Amazon or Microsoft. Alibaba have contributed to theirs,
>>> many other> > cloud providers have contributed, but it seems to be after APIs are> > changed, not in advance.
>>>
>>>
>>> * 1. Docker is another example, that API changes almost every
>>> month. The> > driver we have is unstable and doesn't support API versioning
>>> correctly. *> > Please consider these options,
>>>
>>> I would like to hear from users how you are currently using Apache
>>> Libcloud> > and how you are using it
>>>
>>> NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly> > because they represent +80% of the cloud market [3]. This is a
>>> community> > developed project, has no affiliation to those vendors and these
>>> are my> > opinions, not those of the project.
>>>
>>> [1] https://www.amazonaws.cn/en/products/
>>> [2]
>>> http://www.computerweekly.com/news/450280991/IDC-research-predicts-gradual-decline-in-on-premise-hardware-spend-as-cloud-adoption-rises> > [3]
>>> https://www.crn.com.au/news/microsoft-ate-into-aws-market-share-in-q4-481240> > [4]
>>> https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-vPIYPer2VFtSo_Q/edit?usp=sharing>
>
Re: [dev] [DISCUSS] Future of Apache Libcloud
Posted by Jed Smith <je...@jedsmith.org>.
+ a million to John, and I’m actually in the same boat. I stepped away
due to a conflict, though I technically remain on the PMC (it seems).
Every time I’ve wanted to come back the current design stops me, and
that’s not speaking negatively, just that I have a difficult time
approaching modern libcloud.
I work for a company much more supportive of open source now, and I’d
love to put my volunteer hours into libcloud again. I agree with
everything architectural discussed in this thread, and moving that
direction would rouse me from hibernation.
What would I add to the discussion is that the individual layers should
strive for requests-level simplicity and thought. I agree the ex_
approach is offputting. I think Python, especially modern Python, is
expressive enough to do this aspect well.
A data-driven approach to generating bindings is really, really
interesting. If that ends up being too bold for libcloud, I might help
explore that separately. That’s a really interesting idea.
-J
On Thu, Mar 8, 2018, at 19:18, John Carr wrote:
> Hi Anthony,
>
> I used to be an active contributor to libcloud, and am
> technically still> a committer although I don’t think I’ve been active at all since you
> stepped into the fold. I just wanted to start by saying thanks
> for your> hard work!
>
> I fully support your proposal. It will take a while to be ready
> to fully> exploit py3, we should get started.
>
> For me what went wrong in the current codebase is that the common
> interfaces weren’t common enough. Even within the simple use
> cases there> were often paper cuts (inconsistencies in exceptions raised). And the> ex_ approach was off-putting too.
>
> Ultimately I ended up only targeting AWS and it was a no
> brainer to use> botocore.
>
> An asyncio libcloud could be really compelling - but it should learn a> few tricks from boto/botocore.
>
> I think there should be a low level API that targets the actual API as> closely as possible. It should provide some commonality -
> primitives for> waiting, paginating, retrying, etc. Maybe some well known
> exceptions. It> should probably map to basic types rather than exposing XML.
> Fundamentally AWS APIs are inconsistent but these botocore
> utilities and> patterns make up for it mostly.
>
> High level abstractions should be built on top of this layer
> and only do> what is actually common between APIs - no iffy ex_ params.
>
> I wonder if we can make libcloud more attractive to providers
> wanting to> write a python sdk?
>
> One idea I had a long time ago was to see how well the botocore data
> driven model could extend to other providers. I don’t really think we> have the resources to do this. But the thinking was we could generate> requests, asyncio and twisted bindings from the same data and
> potentially share the data with an node cloud driver for example.
>
> Cheers
> John
>
> > On 8 Mar 2018, at 23:16, anthony shaw <an...@apache.org>
> > wrote:> >
> > Hi All,
> >
> > The "cloud market" when Apache Libcloud was conceived in 2010
> > is very> > different to how it looks today, some trends we are seeing
> >
> > 1- IaaS (our compute API) is one of many features in public clouds.
> > Amazon,> > Azure and GCP have 100's of individual services now [1]. Text-to-
> > speech,> > functions, automation, API gateways, it's impossible to keep up
> > 2- Private Clouds have seen a continued decline but are still
> > popular [2]> > 3- The advent of containers means it is now "easier" to deploy an
> > application to multiple clouds
> > 4- The big 3 public clouds, Amazon, Azure and Google make up most
> > of the> > cloud market [3] if you compare Apache Libcloud downloads with
> > boto (the> > native AWS Python client) downloads, it's a massive delta. Boto is
> > in the> > top 10 most popular PyPi packages
> >
> > In terms of users, I've pulled 3 snapshots of PyPi downloads,
> > January 2016,> > 2017 and 2018 [4]
> > Annual downloads of Apache-Libcloud have seen a slight increase,
> > but the> > 2016 - 2,778,687
> > 2017 - 2,958,591
> >
> > Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is
> > a huge> > drop. [4]
> >
> > *I would like to propose a drastic (depending on your perspective)
> > plan to> > take Apache Libcloud through to 2020*
> >
> > *1. Focus on specific use cases*
> >
> > Lack of consistency between implementations of the base class
> > means in> > practicality it's difficult to have abstracted applications.
> >
> > Apache Libcloud should (imo) come with a toolbox of utilities
> > to both> > demonstrate and validate cloud abstraction use cases, such as:
> > - Migrating storage objects from Cloud X to Cloud Y
> > - Splitting an application workload across multiple clouds
> >
> > *2. Improve performance by adopting asyncio*
> >
> > In almost all use cases, Libcloud would benefit from non-blocking
> > calls.> > Listing VM's requires multiple calls for the pages, uploading
> > storage> > objects can be done in multiple futures, deleting DNS records
> > would be> > better done in async.
> >
> > I'm suggesting we introduce a Python 3.5+ only API, move to
> > requests-futures or aiohttp for the base HTTP client. Yes, *I am
> > suggesting> > we drop Python 2 support in the future*.
> >
> > I've been researching how we could make this switch without breaking
> > 64% of> > our users..
> > Pip now has a way to choose versions based on Python runtimes
> > https://hackernoon.com/phasing-out-python-runtimes-gracefully-956f112f33c4> > We could have apache-libcloud 3+ for Python 3.5 users and then
> > maintain> > 2.3 patches for Python 2.7 and 3.4 users.
> >
> > I think we can come up with a way of continuing to maintain the
> > existing> > code base for 2.7 users but move forward with a new API for
> > async and> > Python 3.5+ users.
> >
> > The downloads [4] show that Python 2.7 is still the majority of
> > users but> > this is declining quickly and by 2020 the tables will have turned.
> > We need> > to be ready for that.
> >
> > *3. Change our positioning on dependencies of 3rd party packages*
> >
> > We aren't seeing enough community contribution to keep up with
> > the rapid> > pace at which Microsoft, Amazon and Google are changing their APIs.
> > Google> > have been contributing to their driver for years. We haven't seen
> > that from> > either Amazon or Microsoft. Alibaba have contributed to theirs,
> > many other> > cloud providers have contributed, but it seems to be after APIs are> > changed, not in advance.
> >
> >
> > * 1. Docker is another example, that API changes almost every
> > month. The> > driver we have is unstable and doesn't support API versioning
> > correctly. *> > Please consider these options,
> >
> > I would like to hear from users how you are currently using Apache
> > Libcloud> > and how you are using it
> >
> > NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly> > because they represent +80% of the cloud market [3]. This is a
> > community> > developed project, has no affiliation to those vendors and these
> > are my> > opinions, not those of the project.
> >
> > [1] https://www.amazonaws.cn/en/products/
> > [2]
> > http://www.computerweekly.com/news/450280991/IDC-research-predicts-gradual-decline-in-on-premise-hardware-spend-as-cloud-adoption-rises> > [3]
> > https://www.crn.com.au/news/microsoft-ate-into-aws-market-share-in-q4-481240> > [4]
> > https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-vPIYPer2VFtSo_Q/edit?usp=sharing>
Re: [dev] [DISCUSS] Future of Apache Libcloud
Posted by John Carr <jo...@unrouted.co.uk>.
Hi Anthony,
I used to be an active contributor to libcloud, and am technically still a committer although I don’t think I’ve been active at all since you stepped into the fold. I just wanted to start by saying thanks for your hard work!
I fully support your proposal. It will take a while to be ready to fully exploit py3, we should get started.
For me what went wrong in the current codebase is that the common interfaces weren’t common enough. Even within the simple use cases there were often paper cuts (inconsistencies in exceptions raised). And the ex_ approach was off-putting too.
Ultimately I ended up only targeting AWS and it was a no brainer to use botocore.
An asyncio libcloud could be really compelling - but it should learn a few tricks from boto/botocore.
I think there should be a low level API that targets the actual API as closely as possible. It should provide some commonality - primitives for waiting, paginating, retrying, etc. Maybe some well known exceptions. It should probably map to basic types rather than exposing XML. Fundamentally AWS APIs are inconsistent but these botocore utilities and patterns make up for it mostly.
High level abstractions should be built on top of this layer and only do what is actually common between APIs - no iffy ex_ params.
I wonder if we can make libcloud more attractive to providers wanting to write a python sdk?
One idea I had a long time ago was to see how well the botocore data driven model could extend to other providers. I don’t really think we have the resources to do this. But the thinking was we could generate requests, asyncio and twisted bindings from the same data and potentially share the data with an node cloud driver for example.
Cheers
John
> On 8 Mar 2018, at 23:16, anthony shaw <an...@apache.org> wrote:
>
> Hi All,
>
> The "cloud market" when Apache Libcloud was conceived in 2010 is very
> different to how it looks today, some trends we are seeing
>
> 1- IaaS (our compute API) is one of many features in public clouds. Amazon,
> Azure and GCP have 100's of individual services now [1]. Text-to-speech,
> functions, automation, API gateways, it's impossible to keep up
> 2- Private Clouds have seen a continued decline but are still popular [2]
> 3- The advent of containers means it is now "easier" to deploy an
> application to multiple clouds
> 4- The big 3 public clouds, Amazon, Azure and Google make up most of the
> cloud market [3] if you compare Apache Libcloud downloads with boto (the
> native AWS Python client) downloads, it's a massive delta. Boto is in the
> top 10 most popular PyPi packages
>
> In terms of users, I've pulled 3 snapshots of PyPi downloads, January 2016,
> 2017 and 2018 [4]
> Annual downloads of Apache-Libcloud have seen a slight increase, but the
> 2016 - 2,778,687
> 2017 - 2,958,591
>
> Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is a huge
> drop. [4]
>
> *I would like to propose a drastic (depending on your perspective) plan to
> take Apache Libcloud through to 2020*
>
> *1. Focus on specific use cases*
>
> Lack of consistency between implementations of the base class means in
> practicality it's difficult to have abstracted applications.
>
> Apache Libcloud should (imo) come with a toolbox of utilities to both
> demonstrate and validate cloud abstraction use cases, such as:
> - Migrating storage objects from Cloud X to Cloud Y
> - Splitting an application workload across multiple clouds
>
> *2. Improve performance by adopting asyncio*
>
> In almost all use cases, Libcloud would benefit from non-blocking calls.
> Listing VM's requires multiple calls for the pages, uploading storage
> objects can be done in multiple futures, deleting DNS records would be
> better done in async.
>
> I'm suggesting we introduce a Python 3.5+ only API, move to
> requests-futures or aiohttp for the base HTTP client. Yes, *I am suggesting
> we drop Python 2 support in the future*.
>
> I've been researching how we could make this switch without breaking 64% of
> our users..
> Pip now has a way to choose versions based on Python runtimes
> https://hackernoon.com/phasing-out-python-runtimes-gracefully-956f112f33c4
> We could have apache-libcloud 3+ for Python 3.5 users and then maintain
> 2.3 patches for Python 2.7 and 3.4 users.
>
> I think we can come up with a way of continuing to maintain the existing
> code base for 2.7 users but move forward with a new API for async and
> Python 3.5+ users.
>
> The downloads [4] show that Python 2.7 is still the majority of users but
> this is declining quickly and by 2020 the tables will have turned. We need
> to be ready for that.
>
> *3. Change our positioning on dependencies of 3rd party packages*
>
> We aren't seeing enough community contribution to keep up with the rapid
> pace at which Microsoft, Amazon and Google are changing their APIs. Google
> have been contributing to their driver for years. We haven't seen that from
> either Amazon or Microsoft. Alibaba have contributed to theirs, many other
> cloud providers have contributed, but it seems to be after APIs are
> changed, not in advance.
>
>
> * 1. Docker is another example, that API changes almost every month. The
> driver we have is unstable and doesn't support API versioning correctly. *
> Please consider these options,
>
> I would like to hear from users how you are currently using Apache Libcloud
> and how you are using it
>
> NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly
> because they represent +80% of the cloud market [3]. This is a community
> developed project, has no affiliation to those vendors and these are my
> opinions, not those of the project.
>
> [1] https://www.amazonaws.cn/en/products/
> [2]
> http://www.computerweekly.com/news/450280991/IDC-research-predicts-gradual-decline-in-on-premise-hardware-spend-as-cloud-adoption-rises
> [3]
> https://www.crn.com.au/news/microsoft-ate-into-aws-market-share-in-q4-481240
> [4]
> https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-vPIYPer2VFtSo_Q/edit?usp=sharing
Re: [dev] [DISCUSS] Future of Apache Libcloud
Posted by John Carr <jo...@unrouted.co.uk>.
Hi Anthony,
I used to be an active contributor to libcloud, and am technically still a committer although I don’t think I’ve been active at all since you stepped into the fold. I just wanted to start by saying thanks for your hard work!
I fully support your proposal. It will take a while to be ready to fully exploit py3, we should get started.
For me what went wrong in the current codebase is that the common interfaces weren’t common enough. Even within the simple use cases there were often paper cuts (inconsistencies in exceptions raised). And the ex_ approach was off-putting too.
Ultimately I ended up only targeting AWS and it was a no brainer to use botocore.
An asyncio libcloud could be really compelling - but it should learn a few tricks from boto/botocore.
I think there should be a low level API that targets the actual API as closely as possible. It should provide some commonality - primitives for waiting, paginating, retrying, etc. Maybe some well known exceptions. It should probably map to basic types rather than exposing XML. Fundamentally AWS APIs are inconsistent but these botocore utilities and patterns make up for it mostly.
High level abstractions should be built on top of this layer and only do what is actually common between APIs - no iffy ex_ params.
I wonder if we can make libcloud more attractive to providers wanting to write a python sdk?
One idea I had a long time ago was to see how well the botocore data driven model could extend to other providers. I don’t really think we have the resources to do this. But the thinking was we could generate requests, asyncio and twisted bindings from the same data and potentially share the data with an node cloud driver for example.
Cheers
John
> On 8 Mar 2018, at 23:16, anthony shaw <an...@apache.org> wrote:
>
> Hi All,
>
> The "cloud market" when Apache Libcloud was conceived in 2010 is very
> different to how it looks today, some trends we are seeing
>
> 1- IaaS (our compute API) is one of many features in public clouds. Amazon,
> Azure and GCP have 100's of individual services now [1]. Text-to-speech,
> functions, automation, API gateways, it's impossible to keep up
> 2- Private Clouds have seen a continued decline but are still popular [2]
> 3- The advent of containers means it is now "easier" to deploy an
> application to multiple clouds
> 4- The big 3 public clouds, Amazon, Azure and Google make up most of the
> cloud market [3] if you compare Apache Libcloud downloads with boto (the
> native AWS Python client) downloads, it's a massive delta. Boto is in the
> top 10 most popular PyPi packages
>
> In terms of users, I've pulled 3 snapshots of PyPi downloads, January 2016,
> 2017 and 2018 [4]
> Annual downloads of Apache-Libcloud have seen a slight increase, but the
> 2016 - 2,778,687
> 2017 - 2,958,591
>
> Python 2.7 represented 90% of users in 2017 and 64% in 2018. This is a huge
> drop. [4]
>
> *I would like to propose a drastic (depending on your perspective) plan to
> take Apache Libcloud through to 2020*
>
> *1. Focus on specific use cases*
>
> Lack of consistency between implementations of the base class means in
> practicality it's difficult to have abstracted applications.
>
> Apache Libcloud should (imo) come with a toolbox of utilities to both
> demonstrate and validate cloud abstraction use cases, such as:
> - Migrating storage objects from Cloud X to Cloud Y
> - Splitting an application workload across multiple clouds
>
> *2. Improve performance by adopting asyncio*
>
> In almost all use cases, Libcloud would benefit from non-blocking calls.
> Listing VM's requires multiple calls for the pages, uploading storage
> objects can be done in multiple futures, deleting DNS records would be
> better done in async.
>
> I'm suggesting we introduce a Python 3.5+ only API, move to
> requests-futures or aiohttp for the base HTTP client. Yes, *I am suggesting
> we drop Python 2 support in the future*.
>
> I've been researching how we could make this switch without breaking 64% of
> our users..
> Pip now has a way to choose versions based on Python runtimes
> https://hackernoon.com/phasing-out-python-runtimes-gracefully-956f112f33c4
> We could have apache-libcloud 3+ for Python 3.5 users and then maintain
> 2.3 patches for Python 2.7 and 3.4 users.
>
> I think we can come up with a way of continuing to maintain the existing
> code base for 2.7 users but move forward with a new API for async and
> Python 3.5+ users.
>
> The downloads [4] show that Python 2.7 is still the majority of users but
> this is declining quickly and by 2020 the tables will have turned. We need
> to be ready for that.
>
> *3. Change our positioning on dependencies of 3rd party packages*
>
> We aren't seeing enough community contribution to keep up with the rapid
> pace at which Microsoft, Amazon and Google are changing their APIs. Google
> have been contributing to their driver for years. We haven't seen that from
> either Amazon or Microsoft. Alibaba have contributed to theirs, many other
> cloud providers have contributed, but it seems to be after APIs are
> changed, not in advance.
>
>
> * 1. Docker is another example, that API changes almost every month. The
> driver we have is unstable and doesn't support API versioning correctly. *
> Please consider these options,
>
> I would like to hear from users how you are currently using Apache Libcloud
> and how you are using it
>
> NB: I have mentioned Azure, AWS and GCP a lot in this thread, mainly
> because they represent +80% of the cloud market [3]. This is a community
> developed project, has no affiliation to those vendors and these are my
> opinions, not those of the project.
>
> [1] https://www.amazonaws.cn/en/products/
> [2]
> http://www.computerweekly.com/news/450280991/IDC-research-predicts-gradual-decline-in-on-premise-hardware-spend-as-cloud-adoption-rises
> [3]
> https://www.crn.com.au/news/microsoft-ate-into-aws-market-share-in-q4-481240
> [4]
> https://docs.google.com/spreadsheets/d/17KEs8Lr_bCQ1XI7QzqmNVe279d7-vPIYPer2VFtSo_Q/edit?usp=sharing