[PROPOSAL] add native container orchestration service

[Murali Reddy <mu...@gmail.com>]

All,

I would like propose native functionality into CloudStack to provide a container service through which users out-of-the box can use to launch container based application. Idea is to support ability to orchestrate the resources and automate aspects of setting up container orchestrator through CloudStack. Public IAAS service providers AWS with its ECS [1] and google with GKE [2] already provides ability container applications.  Competitive cloud orchestration platforms already have native support for container service. Users of CloudStack both as public cloud providers and users with private clouds will benefit with such functionality.

While container orchestrator of user choice can be provisioned on top of CloudStack (with out CloudStack being involved) with tools like TerraForm[3], Ansible[4] etc, advantage of having native orchestration is giving user a nice cohesive integration. This proposal would like add a notion of first class CloudStack entity called container cluster which can be used to provision resources, scale up, scale down, start and stop the cluster of VM’s on which containerised applications can be run. For actual container orchestration we will still need container orchestrator like docker swarm, marathon, kubernetes, but CloudStack container service can automate setting up of control place automatically.

Please see the detailed FS [5], and jira ticket [6]. 

[1]http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html
[2]https://cloud.google.com/container-engine/
[3] https://github.com/skippbox/ansible-cloudstack
[4]https://github.com/chiradeep/terraform-cloudstack-examples/tree/master/kubernetes-coreos-isolated-nw
[5] https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+container+service+functional+specification+and+design+document
[6]https://issues.apache.org/jira/browse/CLOUDSTACK-9760

Re: [PROPOSAL] add native container orchestration service

[Murali Reddy <mu...@shapeblue.com>]

Wido,

This proposal is not suggesting CloudStack do the job of container orchestration like you mentioned we have number of purpose built options for that. 



As described in the FS, running containers on VM’s provisioned by IAAS is popular deployment model. If user wants to do same on CloudStack, user can provision VM’s and use ansible/terraform to deploy container orchestrator control plane and cluster nodes. What this proposal is suggesting to add higher abstraction that can automate this and additional operations to act on a cluster of VM’s as a cohesive unit. Proposed container service can have plugin for swarm, marathon, nomad, etc to automate setting up container orchestrator control place. For this proposal CloudStack container service will integrate with Kubernetes to start with.

As analogy, any body can deploy container orchestrator on the AWS provisioned VM, but there is value why AWS provides native functionality with AWS ECS. Same goes with google's GKE.


On 27/01/17, 3:20 PM, "Wido den Hollander" <wi...@widodh.nl> wrote:

>
>> Op 27 januari 2017 om 7:20 schreef Murali Reddy <mu...@gmail.com>:
>> 
>> 
>> All,
>> 
>> I would like propose native functionality into CloudStack to provide a container service through which users out-of-the box can use to launch container based application. Idea is to support ability to orchestrate the resources and automate aspects of setting up container orchestrator through CloudStack. Public IAAS service providers AWS with its ECS [1] and google with GKE [2] already provides ability container applications.  Competitive cloud orchestration platforms already have native support for container service. Users of CloudStack both as public cloud providers and users with private clouds will benefit with such functionality.
>> 
>
>Sorry, how much I appreciate the effort and work you already invested in this I'm against this.
>
>There are many, many tools which can run inside KVM/Xen/VMWare and do the container orchestration much, much better then I think CloudStack will ever do.
>
>We tried with LXC, but that failed.
>
>For example:
>- Kubernetes
>- Nomad
>- Docker Swarm
>- Apache Mesos
>- CoreOS in the basics
>
>Just a small number of tools which run Containers just fine while being INSIDE a VM.
>
>We still have a long road to go in perfectioning the existing HVs inside CloudStack. I wouldn't want to put resources on testing such a feature which in my opinion does not benefit the project.
>
>CloudStack should imho be mean and lean'. Be damn good and simple in what it does: Running VMs and their network(s).
>
>Wido
>
>> While container orchestrator of user choice can be provisioned on top of CloudStack (with out CloudStack being involved) with tools like TerraForm[3], Ansible[4] etc, advantage of having native orchestration is giving user a nice cohesive integration. This proposal would like add a notion of first class CloudStack entity called container cluster which can be used to provision resources, scale up, scale down, start and stop the cluster of VM’s on which containerised applications can be run. For actual container orchestration we will still need container orchestrator like docker swarm, marathon, kubernetes, but CloudStack container service can automate setting up of control place automatically.
>> 
>> Please see the detailed FS [5], and jira ticket [6]. 
>> 
>> [1]http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html
>> [2]https://cloud.google.com/container-engine/
>> [3] https://github.com/skippbox/ansible-cloudstack
>> [4]https://github.com/chiradeep/terraform-cloudstack-examples/tree/master/kubernetes-coreos-isolated-nw
>> [5] https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+container+service+functional+specification+and+design+document
>> [6]https://issues.apache.org/jira/browse/CLOUDSTACK-9760
>> 
>>

murali.reddy@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: [PROPOSAL] add native container orchestration service

[Wido den Hollander <wi...@widodh.nl>]

> Op 27 januari 2017 om 7:20 schreef Murali Reddy <mu...@gmail.com>:
> 
> 
> All,
> 
> I would like propose native functionality into CloudStack to provide a container service through which users out-of-the box can use to launch container based application. Idea is to support ability to orchestrate the resources and automate aspects of setting up container orchestrator through CloudStack. Public IAAS service providers AWS with its ECS [1] and google with GKE [2] already provides ability container applications.  Competitive cloud orchestration platforms already have native support for container service. Users of CloudStack both as public cloud providers and users with private clouds will benefit with such functionality.
> 

Sorry, how much I appreciate the effort and work you already invested in this I'm against this.

There are many, many tools which can run inside KVM/Xen/VMWare and do the container orchestration much, much better then I think CloudStack will ever do.

We tried with LXC, but that failed.

For example:
- Kubernetes
- Nomad
- Docker Swarm
- Apache Mesos
- CoreOS in the basics

Just a small number of tools which run Containers just fine while being INSIDE a VM.

We still have a long road to go in perfectioning the existing HVs inside CloudStack. I wouldn't want to put resources on testing such a feature which in my opinion does not benefit the project.

CloudStack should imho be mean and lean'. Be damn good and simple in what it does: Running VMs and their network(s).

Wido

> While container orchestrator of user choice can be provisioned on top of CloudStack (with out CloudStack being involved) with tools like TerraForm[3], Ansible[4] etc, advantage of having native orchestration is giving user a nice cohesive integration. This proposal would like add a notion of first class CloudStack entity called container cluster which can be used to provision resources, scale up, scale down, start and stop the cluster of VM’s on which containerised applications can be run. For actual container orchestration we will still need container orchestrator like docker swarm, marathon, kubernetes, but CloudStack container service can automate setting up of control place automatically.
> 
> Please see the detailed FS [5], and jira ticket [6]. 
> 
> [1]http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html
> [2]https://cloud.google.com/container-engine/
> [3] https://github.com/skippbox/ansible-cloudstack
> [4]https://github.com/chiradeep/terraform-cloudstack-examples/tree/master/kubernetes-coreos-isolated-nw
> [5] https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+container+service+functional+specification+and+design+document
> [6]https://issues.apache.org/jira/browse/CLOUDSTACK-9760
> 
>

Re: [PROPOSAL] add native container orchestration service

[Rafael Weingärtner <ra...@gmail.com>]

I have been following the discussion on this thread here. I agree with
everything that has been posted about the responsibility of
maintaining/supporting plugins that sometimes we feel are being left aside
and we do not know to what extent they are being used. However, as Will and
others highlighted, we need to work on new features/integrations;
otherwise, we may start losing market share to alternatives. In my opinion,
if the plugin/integration is well designed and coded/developed, I am more
than ok with it. And of course, let's not re-create something that already
exists. Of course, let's not re-create something that already exists;
integration is one thing re-create something that is already out there is
another.

I do not wish to cause a detour here; I only want to make a comment
regarding the ETSI and their specification for NFV (network functions)
management. Their idea is to manage the networking layer of the cloud. They
talk about managing NFVs (e.g. load balancers, virtual routers, firewalls
and others). ACS does that already (it deploys and manages these underlying
systems, am I mistaken?!). Their idea is to put this management process in
another layer out of the cloud orchestrator. I have seen their blueprints,
deployed and tested the system they have so far, and I am still a little
skeptical with the design (but that is probably only me). I have already
spoken with them, and they are open to integrate with other cloud computing
systems. They use OpenStack because most people working within the OPNFV
project comes from companies that use OpenStack, and their software right
now is quite tied to OpenStack. As a matter of fact, because of the lack of
contact with other communities, they are adapting the specs to OpenStack
needs, which seems a rather unusual process to do with the specification of
a standard. If some of you are interested, we could create a task force and
initiate contact and create a bridge from our project to theirs (by bridge
I mean open a channel of communications). If we do this, we will certainly
bring quite some attention to ACS (there a big telecommunication companies
behind OPNFV and ETSI ). I am working with NFVs management, and I would
much rather use ACS than OpenStack.


On Mon, Jan 30, 2017 at 2:54 AM, Murali Reddy <mu...@shapeblue.com>
wrote:

>
> I agree with some good views Will has shared and I also agree to the
> concerns raised by Wido and Eric. IMO we need balance of staying
> relevant/add new features vs stability of CloudStack and take corrective
> action if needed. We have two good examples for both. When SDN was hot
> technology CloudStack ended up with bunch of SDN controller integrations.
> Few years later, now CloudStack is carrying baggage with no maintainers for
> those plugins, with probably not many of CloudStack users needing overlays.
> On the other hand, other than attempt to liaison with ETSI for NFV no
> effort was done. OpenStack has become de-facto for NFV. Now for OpenStack,
> arguably NFV has become larger use case than cloud it self. I concur with
> Will’s point that with minimal viable solution that does not change the
> core of CloudStack, and can keep CloudStack relevant is worth to be taken
> in.
>
> Will,
>
> To your question of how different is from ShapeBlue’s container service,
> its design, implementation and API semantics etc remain same. ShapeBlue’s
> container service was true drop in plug-in to CloudStack, with this
> proposal I am trying to re-work to make it a core CloudStack pluggable
> service which is part of CloudStack.
>
> Key concepts that this proposal is trying to add
>
>     - add notion of ‘container cluster’ as a first class entity in
> CloudStack. Which is bacially collection of other CloudStack resources
> (like VM’s, networks, public ip, network rules etc)
>     - life cycle operation to manage ‘container cluster’ like create,
> delete, start, stop, scale-up, scale-down, heal etc
>     - orchestrate container orchestrator control plane on top of
> provisioned resources
>
> At-least for k8s (which is what this proposal is targeting), integration
> with k8s is bare minimum. There are cloud-config scripts that automatically
> setup k8s cluster master and node VM’s. All CloudStack is doing in passing
> the cloud-config to the core OS VM’s as user data.
>
> Regards
> Murali Reddy
>
>
>
>
>
>
>
> On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on behalf
> of wstevens@cloudops.com> wrote:
>
> >I agree that we need to be careful what we take on and own inside
> >CloudStack.  I feel like some of the plugins or integrations which we have
> >been "maintaining" may serve us better to abandon, but I feel like that is
> >a whole discussion on its own.
> >
> >In this case, I feel like there is a minimum viable solution which puts
> >CloudStack in a pretty good place to enable container orchestration.  For
> >example, one of the biggest challenges with K8S is the fact that it is
> >single tenant.  CloudStack has good multi tenancy support and has the
> >ability to orchestrate the underlying infra quite well.  We will have to
> be
> >very careful not to try to own too deep into the K8S world though, in my
> >opinion.  We only want to be responsible for providing the infra (and a
> way
> >to bootstrap K8S ideally) and be able to scale the infra, everything else
> >should be owned by the K8S on top.  That is the way I see it anyway, but
> >please add your input.
> >
> >I think it is a liability to try to go too deep, for the same reasons Wido
> >and Erik have mentioned.  But I also think we need to take it seriously
> >because that train is moving and this may be a good opportunity to stay
> >relevant in a rapidly changing market.
> >
> >*Will STEVENS*
> >Lead Developer
> >
> ><https://goo.gl/NYZ8KK>
> >
> >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander <wi...@widodh.nl>
> wrote:
> >
> >>
> >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> wstevens@cloudops.com
> >> >:
> >> >
> >> >
> >> > Hey Murali,
> >> > How different is this proposal than what ShapeBlue already built.  It
> >> looks
> >> > pretty consistent with the functionality that you guys open sourced in
> >> > Seville.
> >> >
> >> > I have not yet used this functionality, but I have reports that it
> works
> >> > quite well.
> >> >
> >> > I believe the premise here is to only orchestrate the VM layer and
> >> > basically expose a "group" of running VMs to the user.  The user is
> >> > responsible for configuring K8S or whatever other container
> orchestrator
> >> on
> >> > top.  I saw mention of the "cloud-config" scripts in the FS, how are
> >> those
> >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
> >> >
> >> > I believe the core feature that is being requested to be added is the
> >> > ability to create a group of VMs which will be kept active as a group
> if
> >> at
> >> > all possible.  ACS would be responsible for making sure that the
> number
> >> of
> >> > VMs specified for the group are in running state and it would spin up
> new
> >> > VMs as needed in order to satisfy the group settings.  In general, it
> is
> >> > understood that any application running on this group would have to be
> >> > fault tolerant enough to be able to rediscover a new VM if one fails
> and
> >> is
> >> > replaced by a fresh copy.  Is that fair to say?  How is it expected
> that
> >> > this service discovery is done, just by VMs being present on the
> network?
> >> >
> >> > As for some of the other people's concerns in this thread.
> >> >
> >> > - Regarding Wido's remarks.  I understand that there is some added
> >> > complexity, but I don't feel like the scope of the addition is
> >> > unrealistic.  I think the LXC integration was a lot farther out of the
> >> > scope of what ACS does then this is.  This does not change the
> "things"
> >> > which ACS orchestrates, it just adds the concept of a grouping of
> things
> >> > which ACS already manages.  I think this is the right approach since
> it
> >> is
> >> > not trying to be a container orchestrator.  We will never compete with
> >> K8S,
> >> > for example, and we should not try, but K8S is here and the market
> wants
> >> > it.  I do think we should be keeping our head up about that fact
> because
> >> > being able to provide a the underlay for K8S is very valuable in the
> >> > current marketplace.  I see this functionality as a way to enable K8S
> >> > adoption on top of ACS without changing our core values.
> >> >
> >> > - Regarding Erik's remarks.  The container space is moving fast, but
> so
> >> is
> >> > the industry.  If we want to remain relevant, we need to be able to
> >> adapt a
> >> > bit.  I don't think this is a big shift in what we do, but it is one
> that
> >> > enables people to be able to start running with something like K8S on
> top
> >> > of their existing ACS.  This is something we are interested in doing
> and
> >> so
> >> > are our customers.  If we can have a thin layer in ACS which helps
> enable
> >> > the use of K8S (or other container orchestrators) by orchestrating
> >> > infrastructure, as we already do, and making it easier to adopt a
> >> container
> >> > orchestrator running on top of ACS, I think that gives us a nice
> foothold
> >> > in the market.  I don't really feel it is fair to compare containers
> to
> >> > IPv6.  IPv6 has been out forever and it has taken almost a decade to
> get
> >> > anyone to adopt it.  Containers have really only been here for like 2
> >> years
> >> > and they are changing the market landscape in a very real way.
> >> >
> >> > Kind of on topic and kind of off topic.  I think understanding our
> >> approach
> >> > to containers is going to be important for the ACS community as a
> whole.
> >> > If we don't offer that market anything, then we will not be considered
> >> and
> >> > we will lose market share we can't afford to lose.  If we try to hitch
> >> our
> >> > horse to that cart too much, we will not be able to be agile enough
> and
> >> > will fail.  I feel like the right approach is for us to know that it
> is a
> >> > thriving market and continue to do what we do, but to extend an olive
> >> > branch to that market.  I think this sort of implementation is the
> right
> >> > approach because we are not trying to do too much.  We are simply
> giving
> >> a
> >> > foundation on which the next big thing in the container orchestration
> >> world
> >> > can adopt without us having to compete directly in that space.  I
> think
> >> we
> >> > have to focus on what we do best, but at the same time, think about
> what
> >> we
> >> > can do to enable that huge market of users to adopt ACS as their
> >> > foundation.  The ability to offer VMs and containers in the same data
> >> plane
> >> > is something we have the ability to do, especially with this approach,
> >> and
> >> > is something that most other softwares can not do.  The adoption of
> >> > containers by bigger organizations will be only part of their
> workload,
> >> > they will still be running VMs for the foreseeable future. Being able
> to
> >> > appeal to that market is going to be important for us.
> >> >
> >> > Hopefully I don't have too many strong opinions here, but I do think
> we
> >> > need to be thinking about how we move forward in a world which is
> >> adopting
> >> > containers in a very real way.
> >> >
> >>
> >> Understood. I just want to prevent that we add more features to
> CloudStack
> >> which are poorly maintained. Not judging Murali here at all, but I'd
> rather
> >> see CloudStack loose code then have it added.
> >>
> >> Thinking about LXC, would like to see that removed together with various
> >> other network plugins which I think are rarely used.
> >>
> >> Now, the idea of Murali was misunderstood by me. I think it would be
> worth
> >> more if we would improve our cloud-init support and integration in
> various
> >> tools which makes it much easier to deploy VMs running containers ON
> >> CloudStack.
> >>
> >> Not so much changing CloudStack code, but rather tooling around it.
> >>
> >> If we have CloudStack talking to Kubernetes we suddenly have to maintain
> >> that API integration. Who's going to do that. Realistically.
> >>
> >> That's my main worry in this regard.
> >>
> >> We have so much more work to do in ACS in total that I don't know if
> this
> >> is the realistic route. I talk to many people who are not looking at
> >> containers and are still working with VMs.
> >>
> >> There is not a single truth which is true, it really depends on who you
> >> ask.
> >>
> >> Wido
> >>
> >> > Cheers,
> >> >
> >> > Will
> >> >
> >> > *Will STEVENS*
> >> > Lead Developer
> >> >
> >> > <https://goo.gl/NYZ8KK>
> >> >
> >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com>
> wrote:
> >> >
> >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> muralimmreddy@gmail.com
> >> >
> >> > > wrote:
> >> > > > All,
> >> > > >
> >> > > > I would like propose native functionality into CloudStack to
> provide
> >> a
> >> > > container service through which users out-of-the box can use to
> launch
> >> > > container based application. Idea is to support ability to
> orchestrate
> >> the
> >> > > resources and automate aspects of setting up container orchestrator
> >> through
> >> > > CloudStack. Public IAAS service providers AWS with its ECS [1] and
> >> google
> >> > > with GKE [2] already provides ability container applications.
> >> Competitive
> >> > > cloud orchestration platforms already have native support for
> container
> >> > > service. Users of CloudStack both as public cloud providers and
> users
> >> with
> >> > > private clouds will benefit with such functionality.
> >> > > >
> >> > > > While container orchestrator of user choice can be provisioned on
> >> top of
> >> > > CloudStack (with out CloudStack being involved) with tools like
> >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> orchestration
> >> is
> >> > > giving user a nice cohesive integration. This proposal would like
> add a
> >> > > notion of first class CloudStack entity called container cluster
> which
> >> can
> >> > > be used to provision resources, scale up, scale down, start and stop
> >> the
> >> > > cluster of VM’s on which containerised applications can be run. For
> >> actual
> >> > > container orchestration we will still need container orchestrator
> like
> >> > > docker swarm, marathon, kubernetes, but CloudStack container service
> >> can
> >> > > automate setting up of control place automatically.
> >> > > >
> >> > >
> >> > > To be honest I'm torn on this one.
> >> > >
> >> > > Containers are a rapid changing thing, and while docker swam,
> >> > > kubernetes, rancher or whatnot is popular today, they might not be
> >> > > tomorrow.
> >> > > They might use CoreOS today, but might not tomorrow.
> >> > >
> >> > > We have a rather poor track record of staying up to date with new
> >> > > features/versions, and adding a feature that is so rapidly changing
> >> > > is, I fear, going to be hard to maintain.
> >> > > Want an example, look at xenserver. It is one of the most used
> >> > > hypervisors we support, yet it took 6 months or so for us to support
> >> > > the latest release.
> >> > > Or IPv6...
> >> > >
> >> > > I don't mean to bash at maintainers/implementers of those features,
> I
> >> > > appreciate all the work being done in every aspect, but I believe we
> >> > > should be realistic and realize that we have issues with keeping
> stuff
> >> > > up to date.
> >> > >
> >> > > I'd say focus on making sure other tools can do their job well
> against
> >> > > CloudStack (kops, rancher, ++), but that does not mean I will -1 the
> >> > > idea if anyone really wants to go through with it.
> >> > >
> >> > > --
> >> > > Erik
> >> > >
> >>
>
> murali.reddy@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>


-- 
Rafael Weingärtner

Re: [PROPOSAL] add native container orchestration service

[Paul Angus <pa...@shapeblue.com>]

Will is 100% correct.  As I mentioned the Title is misleading.  However, Murali did clarify in his explanation; this is really about vm cluster orchestration.



________________________________
From: Will Stevens <ws...@cloudops.com>
Sent: 6 Feb 2017 22:54
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service

​My understanding is that what Paul is talking about is what is already
built and IS what the thread is talking about.​

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani <
rajesh.ramchandani@accelerite.com> wrote:

> Hi Paul - I think this is different from what the thread was about. The
> conversation was specifically about adding support for container
> orchestrators. You are talking about provisioning a group of VMs. Although,
> this is something I think several Cloudstack users have requested before
> and we should propose a solution to this.
>
> Raj
>
> ________________________________
> From: Paul Angus <pa...@shapeblue.com>
> Sent: Monday, February 6, 2017 11:16:41 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] add native container orchestration service
>
> #WhatHeSaid
>
> The title is misleading.  The proposal is purely to add the notion of
> Clusters of VMs to CloudStack.  These may be for databases, containers or
> anything else that needs 'clusters' of compute. Self-healing and
> autoscaling are logical next steps to be added.
>
> Those guys at ShapeBlue have open-sourced their whole k8s container
> service piece.  If/when the 'cluster' part of that work is added into
> CloudStack, the k8s specific pieces can be used by anyone who wants to,
> alternatively they could be used for reference in order to create another
> types of cluster.  (or ignored completely).
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Will Stevens [mailto:williamstevens@gmail.com]
> Sent: 31 January 2017 13:26
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> s/cloud-init/cloud-config/
>
> On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:
>
> > I think that is covered in this proposal. There is nothing k8s
> > specific in this integration (from what I understand), all the k8s
> > details are passed in via the cloud-init configuration after the cluster
> has been provisioned.
> >
> > On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com>
> > wrote:
> >
> >
> > There are many container orchestrators. Those container orchestrators
> > are happy to run on any VMs or bare metal machines. K8s is just one of
> > them and there will be more in the future. It may not be a good idea
> > to make CloudStack to be k8s aware. IMO, the relationship between k8s
> > and cloudstack should be similar to application and os. It probably
> > not a good idea to make your OS to be aware of any specific
> > applications so IMHO I don’t think k8s should be native to CloudStack.
> > It makes more sense to provide some generic services that many
> > applications can take advantages of. Some generic resource grouping
> > service makes sense so a group of VMs, baremetal machines or network
> > can be treated as a single entity. Some life cycle management will be
> > necessary for these entities too. We can deploy k8s, swarm, dcos or
> > even non-container specific services on top of CloudStack. The k8s is
> > changing fast. One single tenant installation may need more than one
> > VM group and may actually requires more (k8s federation). It will be a
> > struggle to be in sync if we try to bring k8s specific knowledge into
> cloudstack.
> >
> > Regards,
> >
> >
> > --
> > Lianghwa Jou
> > VP Engineering, Accelerite
> > email: lianghwa.jou@accelerite.com
> >
> >
> >
> >
> >
> > On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
> >
> >
> >     I agree with some good views Will has shared and I also agree to
> > the concerns raised by Wido and Eric. IMO we need balance of staying
> > relevant/add new features vs stability of CloudStack and take
> > corrective action if needed. We have two good examples for both. When
> > SDN was hot technology CloudStack ended up with bunch of SDN controller
> integrations.
> > Few years later, now CloudStack is carrying baggage with no
> > maintainers for those plugins, with probably not many of CloudStack
> users needing overlays.
> > On the other hand, other than attempt to liaison with ETSI for NFV no
> > effort was done. OpenStack has become de-facto for NFV. Now for
> > OpenStack, arguably NFV has become larger use case than cloud it self.
> > I concur with Will’s point that with minimal viable solution that does
> > not change the core of CloudStack, and can keep CloudStack relevant is
> > worth to be taken in.
> >
> >     Will,
> >
> >     To your question of how different is from ShapeBlue’s container
> > service, its design, implementation and API semantics etc remain same.
> > ShapeBlue’s container service was true drop in plug-in to CloudStack,
> > with this proposal I am trying to re-work to make it a core CloudStack
> > pluggable service which is part of CloudStack.
> >
> >     Key concepts that this proposal is trying to add
> >
> >         - add notion of ‘container cluster’ as a first class entity in
> > CloudStack. Which is bacially collection of other CloudStack resources
> > (like VM’s, networks, public ip, network rules etc)
> >         - life cycle operation to manage ‘container cluster’ like
> > create, delete, start, stop, scale-up, scale-down, heal etc
> >         - orchestrate container orchestrator control plane on top of
> > provisioned resources
> >
> >     At-least for k8s (which is what this proposal is targeting),
> > integration with k8s is bare minimum. There are cloud-config scripts
> > that automatically setup k8s cluster master and node VM’s. All
> > CloudStack is doing in passing the cloud-config to the core OS VM’s as
> user data.
> >
> >     Regards
> >     Murali Reddy
> >
> >
> >
> >
> >
> >
> >
> >     On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
> > behalf of wstevens@cloudops.com> wrote:
> >
> >     >I agree that we need to be careful what we take on and own inside
> >     >CloudStack.  I feel like some of the plugins or integrations
> > which we have
> >     >been "maintaining" may serve us better to abandon, but I feel
> > like that is
> >     >a whole discussion on its own.
> >     >
> >     >In this case, I feel like there is a minimum viable solution
> > which puts
> >     >CloudStack in a pretty good place to enable container orchestration.
> > For
> >     >example, one of the biggest challenges with K8S is the fact that it
> is
> >     >single tenant.  CloudStack has good multi tenancy support and has
> the
> >     >ability to orchestrate the underlying infra quite well.  We will
> > have to be
> >     >very careful not to try to own too deep into the K8S world
> > though, in my
> >     >opinion.  We only want to be responsible for providing the infra
> > (and a way
> >     >to bootstrap K8S ideally) and be able to scale the infra,
> > everything else
> >     >should be owned by the K8S on top.  That is the way I see it
> > anyway, but
> >     >please add your input.
> >     >
> >     >I think it is a liability to try to go too deep, for the same
> > reasons Wido
> >     >and Erik have mentioned.  But I also think we need to take it
> > seriously
> >     >because that train is moving and this may be a good opportunity
> > to stay
> >     >relevant in a rapidly changing market.
> >     >
> >     >*Will STEVENS*
> >     >Lead Developer
> >     >
> >     ><https://goo.gl/NYZ8KK>
> >     >
> >     >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > <wi...@widodh.nl>
> > wrote:
> >     >
> >     >>
> >     >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > wstevens@cloudops.com
> >     >> >:
> >     >> >
> >     >> >
> >     >> > Hey Murali,
> >     >> > How different is this proposal than what ShapeBlue already
> > built.  It
> >     >> looks
> >     >> > pretty consistent with the functionality that you guys open
> > sourced in
> >     >> > Seville.
> >     >> >
> >     >> > I have not yet used this functionality, but I have reports
> > that it works
> >     >> > quite well.
> >     >> >
> >     >> > I believe the premise here is to only orchestrate the VM layer
> and
> >     >> > basically expose a "group" of running VMs to the user.  The
> > user is
> >     >> > responsible for configuring K8S or whatever other container
> > orchestrator
> >     >> on
> >     >> > top.  I saw mention of the "cloud-config" scripts in the FS,
> > how are
> >     >> those
> >     >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
> >     >> >
> >     >> > I believe the core feature that is being requested to be
> > added is the
> >     >> > ability to create a group of VMs which will be kept active as
> > a group if
> >     >> at
> >     >> > all possible.  ACS would be responsible for making sure that
> > the number
> >     >> of
> >     >> > VMs specified for the group are in running state and it would
> > spin up new
> >     >> > VMs as needed in order to satisfy the group settings.  In
> > general, it is
> >     >> > understood that any application running on this group would
> > have to be
> >     >> > fault tolerant enough to be able to rediscover a new VM if
> > one fails and
> >     >> is
> >     >> > replaced by a fresh copy.  Is that fair to say?  How is it
> > expected that
> >     >> > this service discovery is done, just by VMs being present on
> > the network?
> >     >> >
> >     >> > As for some of the other people's concerns in this thread.
> >     >> >
> >     >> > - Regarding Wido's remarks.  I understand that there is some
> added
> >     >> > complexity, but I don't feel like the scope of the addition is
> >     >> > unrealistic.  I think the LXC integration was a lot farther
> > out of the
> >     >> > scope of what ACS does then this is.  This does not change
> > the "things"
> >     >> > which ACS orchestrates, it just adds the concept of a
> > grouping of things
> >     >> > which ACS already manages.  I think this is the right
> > approach since it
> >     >> is
> >     >> > not trying to be a container orchestrator.  We will never
> > compete with
> >     >> K8S,
> >     >> > for example, and we should not try, but K8S is here and the
> > market wants
> >     >> > it.  I do think we should be keeping our head up about that
> > fact because
> >     >> > being able to provide a the underlay for K8S is very valuable
> > in the
> >     >> > current marketplace.  I see this functionality as a way to
> > enable K8S
> >     >> > adoption on top of ACS without changing our core values.
> >     >> >
> >     >> > - Regarding Erik's remarks.  The container space is moving
> > fast, but so
> >     >> is
> >     >> > the industry.  If we want to remain relevant, we need to be
> > able to
> >     >> adapt a
> >     >> > bit.  I don't think this is a big shift in what we do, but it
> > is one that
> >     >> > enables people to be able to start running with something
> > like K8S on top
> >     >> > of their existing ACS.  This is something we are interested
> > in doing and
> >     >> so
> >     >> > are our customers.  If we can have a thin layer in ACS which
> > helps enable
> >     >> > the use of K8S (or other container orchestrators) by
> orchestrating
> >     >> > infrastructure, as we already do, and making it easier to adopt
> a
> >     >> container
> >     >> > orchestrator running on top of ACS, I think that gives us a
> > nice foothold
> >     >> > in the market.  I don't really feel it is fair to compare
> > containers to
> >     >> > IPv6.  IPv6 has been out forever and it has taken almost a
> > decade to get
> >     >> > anyone to adopt it.  Containers have really only been here
> > for like 2
> >     >> years
> >     >> > and they are changing the market landscape in a very real way.
> >     >> >
> >     >> > Kind of on topic and kind of off topic.  I think understanding
> our
> >     >> approach
> >     >> > to containers is going to be important for the ACS community
> > as a whole.
> >     >> > If we don't offer that market anything, then we will not be
> > considered
> >     >> and
> >     >> > we will lose market share we can't afford to lose.  If we try
> > to hitch
> >     >> our
> >     >> > horse to that cart too much, we will not be able to be agile
> > enough and
> >     >> > will fail.  I feel like the right approach is for us to know
> > that it is a
> >     >> > thriving market and continue to do what we do, but to extend
> > an olive
> >     >> > branch to that market.  I think this sort of implementation
> > is the right
> >     >> > approach because we are not trying to do too much.  We are
> > simply giving
> >     >> a
> >     >> > foundation on which the next big thing in the container
> > orchestration
> >     >> world
> >     >> > can adopt without us having to compete directly in that
> > space.  I think
> >     >> we
> >     >> > have to focus on what we do best, but at the same time, think
> > about what
> >     >> we
> >     >> > can do to enable that huge market of users to adopt ACS as their
> >     >> > foundation.  The ability to offer VMs and containers in the
> > same data
> >     >> plane
> >     >> > is something we have the ability to do, especially with this
> > approach,
> >     >> and
> >     >> > is something that most other softwares can not do.  The
> > adoption of
> >     >> > containers by bigger organizations will be only part of their
> > workload,
> >     >> > they will still be running VMs for the foreseeable future.
> > Being able to
> >     >> > appeal to that market is going to be important for us.
> >     >> >
> >     >> > Hopefully I don't have too many strong opinions here, but I
> > do think we
> >     >> > need to be thinking about how we move forward in a world which
> is
> >     >> adopting
> >     >> > containers in a very real way.
> >     >> >
> >     >>
> >     >> Understood. I just want to prevent that we add more features to
> > CloudStack
> >     >> which are poorly maintained. Not judging Murali here at all,
> > but I'd rather
> >     >> see CloudStack loose code then have it added.
> >     >>
> >     >> Thinking about LXC, would like to see that removed together
> > with various
> >     >> other network plugins which I think are rarely used.
> >     >>
> >     >> Now, the idea of Murali was misunderstood by me. I think it
> > would be worth
> >     >> more if we would improve our cloud-init support and integration
> > in various
> >     >> tools which makes it much easier to deploy VMs running containers
> ON
> >     >> CloudStack.
> >     >>
> >     >> Not so much changing CloudStack code, but rather tooling around
> it.
> >     >>
> >     >> If we have CloudStack talking to Kubernetes we suddenly have to
> > maintain
> >     >> that API integration. Who's going to do that. Realistically.
> >     >>
> >     >> That's my main worry in this regard.
> >     >>
> >     >> We have so much more work to do in ACS in total that I don't
> > know if this
> >     >> is the realistic route. I talk to many people who are not looking
> at
> >     >> containers and are still working with VMs.
> >     >>
> >     >> There is not a single truth which is true, it really depends on
> > who you
> >     >> ask.
> >     >>
> >     >> Wido
> >     >>
> >     >> > Cheers,
> >     >> >
> >     >> > Will
> >     >> >
> >     >> > *Will STEVENS*
> >     >> > Lead Developer
> >     >> >
> >     >> > <https://goo.gl/NYZ8KK>
> >     >> >
> >     >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > <te...@gmail.com>
> > wrote:
> >     >> >
> >     >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > muralimmreddy@gmail.com
> >     >> >
> >     >> > > wrote:
> >     >> > > > All,
> >     >> > > >
> >     >> > > > I would like propose native functionality into CloudStack
> > to provide
> >     >> a
> >     >> > > container service through which users out-of-the box can
> > use to launch
> >     >> > > container based application. Idea is to support ability to
> > orchestrate
> >     >> the
> >     >> > > resources and automate aspects of setting up container
> > orchestrator
> >     >> through
> >     >> > > CloudStack. Public IAAS service providers AWS with its ECS
> > [1] and
> >     >> google
> >     >> > > with GKE [2] already provides ability container applications.
> >     >> Competitive
> >     >> > > cloud orchestration platforms already have native support
> > for container
> >     >> > > service. Users of CloudStack both as public cloud providers
> > and users
> >     >> with
> >     >> > > private clouds will benefit with such functionality.
> >     >> > > >
> >     >> > > > While container orchestrator of user choice can be
> > provisioned on
> >     >> top of
> >     >> > > CloudStack (with out CloudStack being involved) with tools
> like
> >     >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> > orchestration
> >     >> is
> >     >> > > giving user a nice cohesive integration. This proposal
> > would like add a
> >     >> > > notion of first class CloudStack entity called container
> > cluster which
> >     >> can
> >     >> > > be used to provision resources, scale up, scale down, start
> > and stop
> >     >> the
> >     >> > > cluster of VM’s on which containerised applications can be
> run.
> > For
> >     >> actual
> >     >> > > container orchestration we will still need container
> > orchestrator like
> >     >> > > docker swarm, marathon, kubernetes, but CloudStack
> > container service
> >     >> can
> >     >> > > automate setting up of control place automatically.
> >     >> > > >
> >     >> > >
> >     >> > > To be honest I'm torn on this one.
> >     >> > >
> >     >> > > Containers are a rapid changing thing, and while docker swam,
> >     >> > > kubernetes, rancher or whatnot is popular today, they might
> > not be
> >     >> > > tomorrow.
> >     >> > > They might use CoreOS today, but might not tomorrow.
> >     >> > >
> >     >> > > We have a rather poor track record of staying up to date
> > with new
> >     >> > > features/versions, and adding a feature that is so rapidly
> > changing
> >     >> > > is, I fear, going to be hard to maintain.
> >     >> > > Want an example, look at xenserver. It is one of the most used
> >     >> > > hypervisors we support, yet it took 6 months or so for us
> > to support
> >     >> > > the latest release.
> >     >> > > Or IPv6...
> >     >> > >
> >     >> > > I don't mean to bash at maintainers/implementers of those
> > features, I
> >     >> > > appreciate all the work being done in every aspect, but I
> > believe we
> >     >> > > should be realistic and realize that we have issues with
> > keeping stuff
> >     >> > > up to date.
> >     >> > >
> >     >> > > I'd say focus on making sure other tools can do their job
> > well against
> >     >> > > CloudStack (kops, rancher, ++), but that does not mean I
> > will
> > -1 the
> >     >> > > idea if anyone really wants to go through with it.
> >     >> > >
> >     >> > > --
> >     >> > > Erik
> >     >> > >
> >     >>
> >
> >     murali.reddy@shapeblue.com
> >     www.shapeblue.com<http://www.shapeblue.com>
> >     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >     @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which
> > is the property of Accelerite, a Persistent Systems business. It is
> > intended only for the use of the individual or entity to which it is
> > addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Accelerite, a
> > Persistent Systems business does not accept any liability for virus
> infected mails.
> >
> >
> >
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>

paul.angus@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: [PROPOSAL] add native container orchestration service

[Will Stevens <ws...@cloudops.com>]

​My understanding is that what Paul is talking about is what is already
built and IS what the thread is talking about.​

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani <
rajesh.ramchandani@accelerite.com> wrote:

> Hi Paul - I think this is different from what the thread was about. The
> conversation was specifically about adding support for container
> orchestrators. You are talking about provisioning a group of VMs. Although,
> this is something I think several Cloudstack users have requested before
> and we should propose a solution to this.
>
> Raj
>
> ________________________________
> From: Paul Angus <pa...@shapeblue.com>
> Sent: Monday, February 6, 2017 11:16:41 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] add native container orchestration service
>
> #WhatHeSaid
>
> The title is misleading.  The proposal is purely to add the notion of
> Clusters of VMs to CloudStack.  These may be for databases, containers or
> anything else that needs 'clusters' of compute. Self-healing and
> autoscaling are logical next steps to be added.
>
> Those guys at ShapeBlue have open-sourced their whole k8s container
> service piece.  If/when the 'cluster' part of that work is added into
> CloudStack, the k8s specific pieces can be used by anyone who wants to,
> alternatively they could be used for reference in order to create another
> types of cluster.  (or ignored completely).
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Will Stevens [mailto:williamstevens@gmail.com]
> Sent: 31 January 2017 13:26
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> s/cloud-init/cloud-config/
>
> On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:
>
> > I think that is covered in this proposal. There is nothing k8s
> > specific in this integration (from what I understand), all the k8s
> > details are passed in via the cloud-init configuration after the cluster
> has been provisioned.
> >
> > On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com>
> > wrote:
> >
> >
> > There are many container orchestrators. Those container orchestrators
> > are happy to run on any VMs or bare metal machines. K8s is just one of
> > them and there will be more in the future. It may not be a good idea
> > to make CloudStack to be k8s aware. IMO, the relationship between k8s
> > and cloudstack should be similar to application and os. It probably
> > not a good idea to make your OS to be aware of any specific
> > applications so IMHO I don’t think k8s should be native to CloudStack.
> > It makes more sense to provide some generic services that many
> > applications can take advantages of. Some generic resource grouping
> > service makes sense so a group of VMs, baremetal machines or network
> > can be treated as a single entity. Some life cycle management will be
> > necessary for these entities too. We can deploy k8s, swarm, dcos or
> > even non-container specific services on top of CloudStack. The k8s is
> > changing fast. One single tenant installation may need more than one
> > VM group and may actually requires more (k8s federation). It will be a
> > struggle to be in sync if we try to bring k8s specific knowledge into
> cloudstack.
> >
> > Regards,
> >
> >
> > --
> > Lianghwa Jou
> > VP Engineering, Accelerite
> > email: lianghwa.jou@accelerite.com
> >
> >
> >
> >
> >
> > On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
> >
> >
> >     I agree with some good views Will has shared and I also agree to
> > the concerns raised by Wido and Eric. IMO we need balance of staying
> > relevant/add new features vs stability of CloudStack and take
> > corrective action if needed. We have two good examples for both. When
> > SDN was hot technology CloudStack ended up with bunch of SDN controller
> integrations.
> > Few years later, now CloudStack is carrying baggage with no
> > maintainers for those plugins, with probably not many of CloudStack
> users needing overlays.
> > On the other hand, other than attempt to liaison with ETSI for NFV no
> > effort was done. OpenStack has become de-facto for NFV. Now for
> > OpenStack, arguably NFV has become larger use case than cloud it self.
> > I concur with Will’s point that with minimal viable solution that does
> > not change the core of CloudStack, and can keep CloudStack relevant is
> > worth to be taken in.
> >
> >     Will,
> >
> >     To your question of how different is from ShapeBlue’s container
> > service, its design, implementation and API semantics etc remain same.
> > ShapeBlue’s container service was true drop in plug-in to CloudStack,
> > with this proposal I am trying to re-work to make it a core CloudStack
> > pluggable service which is part of CloudStack.
> >
> >     Key concepts that this proposal is trying to add
> >
> >         - add notion of ‘container cluster’ as a first class entity in
> > CloudStack. Which is bacially collection of other CloudStack resources
> > (like VM’s, networks, public ip, network rules etc)
> >         - life cycle operation to manage ‘container cluster’ like
> > create, delete, start, stop, scale-up, scale-down, heal etc
> >         - orchestrate container orchestrator control plane on top of
> > provisioned resources
> >
> >     At-least for k8s (which is what this proposal is targeting),
> > integration with k8s is bare minimum. There are cloud-config scripts
> > that automatically setup k8s cluster master and node VM’s. All
> > CloudStack is doing in passing the cloud-config to the core OS VM’s as
> user data.
> >
> >     Regards
> >     Murali Reddy
> >
> >
> >
> >
> >
> >
> >
> >     On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
> > behalf of wstevens@cloudops.com> wrote:
> >
> >     >I agree that we need to be careful what we take on and own inside
> >     >CloudStack.  I feel like some of the plugins or integrations
> > which we have
> >     >been "maintaining" may serve us better to abandon, but I feel
> > like that is
> >     >a whole discussion on its own.
> >     >
> >     >In this case, I feel like there is a minimum viable solution
> > which puts
> >     >CloudStack in a pretty good place to enable container orchestration.
> > For
> >     >example, one of the biggest challenges with K8S is the fact that it
> is
> >     >single tenant.  CloudStack has good multi tenancy support and has
> the
> >     >ability to orchestrate the underlying infra quite well.  We will
> > have to be
> >     >very careful not to try to own too deep into the K8S world
> > though, in my
> >     >opinion.  We only want to be responsible for providing the infra
> > (and a way
> >     >to bootstrap K8S ideally) and be able to scale the infra,
> > everything else
> >     >should be owned by the K8S on top.  That is the way I see it
> > anyway, but
> >     >please add your input.
> >     >
> >     >I think it is a liability to try to go too deep, for the same
> > reasons Wido
> >     >and Erik have mentioned.  But I also think we need to take it
> > seriously
> >     >because that train is moving and this may be a good opportunity
> > to stay
> >     >relevant in a rapidly changing market.
> >     >
> >     >*Will STEVENS*
> >     >Lead Developer
> >     >
> >     ><https://goo.gl/NYZ8KK>
> >     >
> >     >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > <wi...@widodh.nl>
> > wrote:
> >     >
> >     >>
> >     >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > wstevens@cloudops.com
> >     >> >:
> >     >> >
> >     >> >
> >     >> > Hey Murali,
> >     >> > How different is this proposal than what ShapeBlue already
> > built.  It
> >     >> looks
> >     >> > pretty consistent with the functionality that you guys open
> > sourced in
> >     >> > Seville.
> >     >> >
> >     >> > I have not yet used this functionality, but I have reports
> > that it works
> >     >> > quite well.
> >     >> >
> >     >> > I believe the premise here is to only orchestrate the VM layer
> and
> >     >> > basically expose a "group" of running VMs to the user.  The
> > user is
> >     >> > responsible for configuring K8S or whatever other container
> > orchestrator
> >     >> on
> >     >> > top.  I saw mention of the "cloud-config" scripts in the FS,
> > how are
> >     >> those
> >     >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
> >     >> >
> >     >> > I believe the core feature that is being requested to be
> > added is the
> >     >> > ability to create a group of VMs which will be kept active as
> > a group if
> >     >> at
> >     >> > all possible.  ACS would be responsible for making sure that
> > the number
> >     >> of
> >     >> > VMs specified for the group are in running state and it would
> > spin up new
> >     >> > VMs as needed in order to satisfy the group settings.  In
> > general, it is
> >     >> > understood that any application running on this group would
> > have to be
> >     >> > fault tolerant enough to be able to rediscover a new VM if
> > one fails and
> >     >> is
> >     >> > replaced by a fresh copy.  Is that fair to say?  How is it
> > expected that
> >     >> > this service discovery is done, just by VMs being present on
> > the network?
> >     >> >
> >     >> > As for some of the other people's concerns in this thread.
> >     >> >
> >     >> > - Regarding Wido's remarks.  I understand that there is some
> added
> >     >> > complexity, but I don't feel like the scope of the addition is
> >     >> > unrealistic.  I think the LXC integration was a lot farther
> > out of the
> >     >> > scope of what ACS does then this is.  This does not change
> > the "things"
> >     >> > which ACS orchestrates, it just adds the concept of a
> > grouping of things
> >     >> > which ACS already manages.  I think this is the right
> > approach since it
> >     >> is
> >     >> > not trying to be a container orchestrator.  We will never
> > compete with
> >     >> K8S,
> >     >> > for example, and we should not try, but K8S is here and the
> > market wants
> >     >> > it.  I do think we should be keeping our head up about that
> > fact because
> >     >> > being able to provide a the underlay for K8S is very valuable
> > in the
> >     >> > current marketplace.  I see this functionality as a way to
> > enable K8S
> >     >> > adoption on top of ACS without changing our core values.
> >     >> >
> >     >> > - Regarding Erik's remarks.  The container space is moving
> > fast, but so
> >     >> is
> >     >> > the industry.  If we want to remain relevant, we need to be
> > able to
> >     >> adapt a
> >     >> > bit.  I don't think this is a big shift in what we do, but it
> > is one that
> >     >> > enables people to be able to start running with something
> > like K8S on top
> >     >> > of their existing ACS.  This is something we are interested
> > in doing and
> >     >> so
> >     >> > are our customers.  If we can have a thin layer in ACS which
> > helps enable
> >     >> > the use of K8S (or other container orchestrators) by
> orchestrating
> >     >> > infrastructure, as we already do, and making it easier to adopt
> a
> >     >> container
> >     >> > orchestrator running on top of ACS, I think that gives us a
> > nice foothold
> >     >> > in the market.  I don't really feel it is fair to compare
> > containers to
> >     >> > IPv6.  IPv6 has been out forever and it has taken almost a
> > decade to get
> >     >> > anyone to adopt it.  Containers have really only been here
> > for like 2
> >     >> years
> >     >> > and they are changing the market landscape in a very real way.
> >     >> >
> >     >> > Kind of on topic and kind of off topic.  I think understanding
> our
> >     >> approach
> >     >> > to containers is going to be important for the ACS community
> > as a whole.
> >     >> > If we don't offer that market anything, then we will not be
> > considered
> >     >> and
> >     >> > we will lose market share we can't afford to lose.  If we try
> > to hitch
> >     >> our
> >     >> > horse to that cart too much, we will not be able to be agile
> > enough and
> >     >> > will fail.  I feel like the right approach is for us to know
> > that it is a
> >     >> > thriving market and continue to do what we do, but to extend
> > an olive
> >     >> > branch to that market.  I think this sort of implementation
> > is the right
> >     >> > approach because we are not trying to do too much.  We are
> > simply giving
> >     >> a
> >     >> > foundation on which the next big thing in the container
> > orchestration
> >     >> world
> >     >> > can adopt without us having to compete directly in that
> > space.  I think
> >     >> we
> >     >> > have to focus on what we do best, but at the same time, think
> > about what
> >     >> we
> >     >> > can do to enable that huge market of users to adopt ACS as their
> >     >> > foundation.  The ability to offer VMs and containers in the
> > same data
> >     >> plane
> >     >> > is something we have the ability to do, especially with this
> > approach,
> >     >> and
> >     >> > is something that most other softwares can not do.  The
> > adoption of
> >     >> > containers by bigger organizations will be only part of their
> > workload,
> >     >> > they will still be running VMs for the foreseeable future.
> > Being able to
> >     >> > appeal to that market is going to be important for us.
> >     >> >
> >     >> > Hopefully I don't have too many strong opinions here, but I
> > do think we
> >     >> > need to be thinking about how we move forward in a world which
> is
> >     >> adopting
> >     >> > containers in a very real way.
> >     >> >
> >     >>
> >     >> Understood. I just want to prevent that we add more features to
> > CloudStack
> >     >> which are poorly maintained. Not judging Murali here at all,
> > but I'd rather
> >     >> see CloudStack loose code then have it added.
> >     >>
> >     >> Thinking about LXC, would like to see that removed together
> > with various
> >     >> other network plugins which I think are rarely used.
> >     >>
> >     >> Now, the idea of Murali was misunderstood by me. I think it
> > would be worth
> >     >> more if we would improve our cloud-init support and integration
> > in various
> >     >> tools which makes it much easier to deploy VMs running containers
> ON
> >     >> CloudStack.
> >     >>
> >     >> Not so much changing CloudStack code, but rather tooling around
> it.
> >     >>
> >     >> If we have CloudStack talking to Kubernetes we suddenly have to
> > maintain
> >     >> that API integration. Who's going to do that. Realistically.
> >     >>
> >     >> That's my main worry in this regard.
> >     >>
> >     >> We have so much more work to do in ACS in total that I don't
> > know if this
> >     >> is the realistic route. I talk to many people who are not looking
> at
> >     >> containers and are still working with VMs.
> >     >>
> >     >> There is not a single truth which is true, it really depends on
> > who you
> >     >> ask.
> >     >>
> >     >> Wido
> >     >>
> >     >> > Cheers,
> >     >> >
> >     >> > Will
> >     >> >
> >     >> > *Will STEVENS*
> >     >> > Lead Developer
> >     >> >
> >     >> > <https://goo.gl/NYZ8KK>
> >     >> >
> >     >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > <te...@gmail.com>
> > wrote:
> >     >> >
> >     >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > muralimmreddy@gmail.com
> >     >> >
> >     >> > > wrote:
> >     >> > > > All,
> >     >> > > >
> >     >> > > > I would like propose native functionality into CloudStack
> > to provide
> >     >> a
> >     >> > > container service through which users out-of-the box can
> > use to launch
> >     >> > > container based application. Idea is to support ability to
> > orchestrate
> >     >> the
> >     >> > > resources and automate aspects of setting up container
> > orchestrator
> >     >> through
> >     >> > > CloudStack. Public IAAS service providers AWS with its ECS
> > [1] and
> >     >> google
> >     >> > > with GKE [2] already provides ability container applications.
> >     >> Competitive
> >     >> > > cloud orchestration platforms already have native support
> > for container
> >     >> > > service. Users of CloudStack both as public cloud providers
> > and users
> >     >> with
> >     >> > > private clouds will benefit with such functionality.
> >     >> > > >
> >     >> > > > While container orchestrator of user choice can be
> > provisioned on
> >     >> top of
> >     >> > > CloudStack (with out CloudStack being involved) with tools
> like
> >     >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> > orchestration
> >     >> is
> >     >> > > giving user a nice cohesive integration. This proposal
> > would like add a
> >     >> > > notion of first class CloudStack entity called container
> > cluster which
> >     >> can
> >     >> > > be used to provision resources, scale up, scale down, start
> > and stop
> >     >> the
> >     >> > > cluster of VM’s on which containerised applications can be
> run.
> > For
> >     >> actual
> >     >> > > container orchestration we will still need container
> > orchestrator like
> >     >> > > docker swarm, marathon, kubernetes, but CloudStack
> > container service
> >     >> can
> >     >> > > automate setting up of control place automatically.
> >     >> > > >
> >     >> > >
> >     >> > > To be honest I'm torn on this one.
> >     >> > >
> >     >> > > Containers are a rapid changing thing, and while docker swam,
> >     >> > > kubernetes, rancher or whatnot is popular today, they might
> > not be
> >     >> > > tomorrow.
> >     >> > > They might use CoreOS today, but might not tomorrow.
> >     >> > >
> >     >> > > We have a rather poor track record of staying up to date
> > with new
> >     >> > > features/versions, and adding a feature that is so rapidly
> > changing
> >     >> > > is, I fear, going to be hard to maintain.
> >     >> > > Want an example, look at xenserver. It is one of the most used
> >     >> > > hypervisors we support, yet it took 6 months or so for us
> > to support
> >     >> > > the latest release.
> >     >> > > Or IPv6...
> >     >> > >
> >     >> > > I don't mean to bash at maintainers/implementers of those
> > features, I
> >     >> > > appreciate all the work being done in every aspect, but I
> > believe we
> >     >> > > should be realistic and realize that we have issues with
> > keeping stuff
> >     >> > > up to date.
> >     >> > >
> >     >> > > I'd say focus on making sure other tools can do their job
> > well against
> >     >> > > CloudStack (kops, rancher, ++), but that does not mean I
> > will
> > -1 the
> >     >> > > idea if anyone really wants to go through with it.
> >     >> > >
> >     >> > > --
> >     >> > > Erik
> >     >> > >
> >     >>
> >
> >     murali.reddy@shapeblue.com
> >     www.shapeblue.com<http://www.shapeblue.com>
> >     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >     @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which
> > is the property of Accelerite, a Persistent Systems business. It is
> > intended only for the use of the individual or entity to which it is
> > addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Accelerite, a
> > Persistent Systems business does not accept any liability for virus
> infected mails.
> >
> >
> >
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>

RE: [PROPOSAL] add native container orchestration service

[Rajesh Ramchandani <ra...@accelerite.com>]

Hi Paul - I think this is different from what the thread was about. The conversation was specifically about adding support for container orchestrators. You are talking about provisioning a group of VMs. Although, this is something I think several Cloudstack users have requested before and we should propose a solution to this.

Raj

________________________________
From: Paul Angus <pa...@shapeblue.com>
Sent: Monday, February 6, 2017 11:16:41 AM
To: dev@cloudstack.apache.org
Subject: RE: [PROPOSAL] add native container orchestration service

#WhatHeSaid

The title is misleading.  The proposal is purely to add the notion of Clusters of VMs to CloudStack.  These may be for databases, containers or anything else that needs 'clusters' of compute. Self-healing and autoscaling are logical next steps to be added.

Those guys at ShapeBlue have open-sourced their whole k8s container service piece.  If/when the 'cluster' part of that work is added into CloudStack, the k8s specific pieces can be used by anyone who wants to, alternatively they could be used for reference in order to create another types of cluster.  (or ignored completely).




paul.angus@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-----Original Message-----
From: Will Stevens [mailto:williamstevens@gmail.com]
Sent: 31 January 2017 13:26
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service

s/cloud-init/cloud-config/

On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:

> I think that is covered in this proposal. There is nothing k8s
> specific in this integration (from what I understand), all the k8s
> details are passed in via the cloud-init configuration after the cluster has been provisioned.
>
> On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com>
> wrote:
>
>
> There are many container orchestrators. Those container orchestrators
> are happy to run on any VMs or bare metal machines. K8s is just one of
> them and there will be more in the future. It may not be a good idea
> to make CloudStack to be k8s aware. IMO, the relationship between k8s
> and cloudstack should be similar to application and os. It probably
> not a good idea to make your OS to be aware of any specific
> applications so IMHO I don’t think k8s should be native to CloudStack.
> It makes more sense to provide some generic services that many
> applications can take advantages of. Some generic resource grouping
> service makes sense so a group of VMs, baremetal machines or network
> can be treated as a single entity. Some life cycle management will be
> necessary for these entities too. We can deploy k8s, swarm, dcos or
> even non-container specific services on top of CloudStack. The k8s is
> changing fast. One single tenant installation may need more than one
> VM group and may actually requires more (k8s federation). It will be a
> struggle to be in sync if we try to bring k8s specific knowledge into cloudstack.
>
> Regards,
>
>
> --
> Lianghwa Jou
> VP Engineering, Accelerite
> email: lianghwa.jou@accelerite.com
>
>
>
>
>
> On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
>
>
>     I agree with some good views Will has shared and I also agree to
> the concerns raised by Wido and Eric. IMO we need balance of staying
> relevant/add new features vs stability of CloudStack and take
> corrective action if needed. We have two good examples for both. When
> SDN was hot technology CloudStack ended up with bunch of SDN controller integrations.
> Few years later, now CloudStack is carrying baggage with no
> maintainers for those plugins, with probably not many of CloudStack users needing overlays.
> On the other hand, other than attempt to liaison with ETSI for NFV no
> effort was done. OpenStack has become de-facto for NFV. Now for
> OpenStack, arguably NFV has become larger use case than cloud it self.
> I concur with Will’s point that with minimal viable solution that does
> not change the core of CloudStack, and can keep CloudStack relevant is
> worth to be taken in.
>
>     Will,
>
>     To your question of how different is from ShapeBlue’s container
> service, its design, implementation and API semantics etc remain same.
> ShapeBlue’s container service was true drop in plug-in to CloudStack,
> with this proposal I am trying to re-work to make it a core CloudStack
> pluggable service which is part of CloudStack.
>
>     Key concepts that this proposal is trying to add
>
>         - add notion of ‘container cluster’ as a first class entity in
> CloudStack. Which is bacially collection of other CloudStack resources
> (like VM’s, networks, public ip, network rules etc)
>         - life cycle operation to manage ‘container cluster’ like
> create, delete, start, stop, scale-up, scale-down, heal etc
>         - orchestrate container orchestrator control plane on top of
> provisioned resources
>
>     At-least for k8s (which is what this proposal is targeting),
> integration with k8s is bare minimum. There are cloud-config scripts
> that automatically setup k8s cluster master and node VM’s. All
> CloudStack is doing in passing the cloud-config to the core OS VM’s as user data.
>
>     Regards
>     Murali Reddy
>
>
>
>
>
>
>
>     On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
> behalf of wstevens@cloudops.com> wrote:
>
>     >I agree that we need to be careful what we take on and own inside
>     >CloudStack.  I feel like some of the plugins or integrations
> which we have
>     >been "maintaining" may serve us better to abandon, but I feel
> like that is
>     >a whole discussion on its own.
>     >
>     >In this case, I feel like there is a minimum viable solution
> which puts
>     >CloudStack in a pretty good place to enable container orchestration.
> For
>     >example, one of the biggest challenges with K8S is the fact that it is
>     >single tenant.  CloudStack has good multi tenancy support and has the
>     >ability to orchestrate the underlying infra quite well.  We will
> have to be
>     >very careful not to try to own too deep into the K8S world
> though, in my
>     >opinion.  We only want to be responsible for providing the infra
> (and a way
>     >to bootstrap K8S ideally) and be able to scale the infra,
> everything else
>     >should be owned by the K8S on top.  That is the way I see it
> anyway, but
>     >please add your input.
>     >
>     >I think it is a liability to try to go too deep, for the same
> reasons Wido
>     >and Erik have mentioned.  But I also think we need to take it
> seriously
>     >because that train is moving and this may be a good opportunity
> to stay
>     >relevant in a rapidly changing market.
>     >
>     >*Will STEVENS*
>     >Lead Developer
>     >
>     ><https://goo.gl/NYZ8KK>
>     >
>     >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> <wi...@widodh.nl>
> wrote:
>     >
>     >>
>     >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> wstevens@cloudops.com
>     >> >:
>     >> >
>     >> >
>     >> > Hey Murali,
>     >> > How different is this proposal than what ShapeBlue already
> built.  It
>     >> looks
>     >> > pretty consistent with the functionality that you guys open
> sourced in
>     >> > Seville.
>     >> >
>     >> > I have not yet used this functionality, but I have reports
> that it works
>     >> > quite well.
>     >> >
>     >> > I believe the premise here is to only orchestrate the VM layer and
>     >> > basically expose a "group" of running VMs to the user.  The
> user is
>     >> > responsible for configuring K8S or whatever other container
> orchestrator
>     >> on
>     >> > top.  I saw mention of the "cloud-config" scripts in the FS,
> how are
>     >> those
>     >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
>     >> >
>     >> > I believe the core feature that is being requested to be
> added is the
>     >> > ability to create a group of VMs which will be kept active as
> a group if
>     >> at
>     >> > all possible.  ACS would be responsible for making sure that
> the number
>     >> of
>     >> > VMs specified for the group are in running state and it would
> spin up new
>     >> > VMs as needed in order to satisfy the group settings.  In
> general, it is
>     >> > understood that any application running on this group would
> have to be
>     >> > fault tolerant enough to be able to rediscover a new VM if
> one fails and
>     >> is
>     >> > replaced by a fresh copy.  Is that fair to say?  How is it
> expected that
>     >> > this service discovery is done, just by VMs being present on
> the network?
>     >> >
>     >> > As for some of the other people's concerns in this thread.
>     >> >
>     >> > - Regarding Wido's remarks.  I understand that there is some added
>     >> > complexity, but I don't feel like the scope of the addition is
>     >> > unrealistic.  I think the LXC integration was a lot farther
> out of the
>     >> > scope of what ACS does then this is.  This does not change
> the "things"
>     >> > which ACS orchestrates, it just adds the concept of a
> grouping of things
>     >> > which ACS already manages.  I think this is the right
> approach since it
>     >> is
>     >> > not trying to be a container orchestrator.  We will never
> compete with
>     >> K8S,
>     >> > for example, and we should not try, but K8S is here and the
> market wants
>     >> > it.  I do think we should be keeping our head up about that
> fact because
>     >> > being able to provide a the underlay for K8S is very valuable
> in the
>     >> > current marketplace.  I see this functionality as a way to
> enable K8S
>     >> > adoption on top of ACS without changing our core values.
>     >> >
>     >> > - Regarding Erik's remarks.  The container space is moving
> fast, but so
>     >> is
>     >> > the industry.  If we want to remain relevant, we need to be
> able to
>     >> adapt a
>     >> > bit.  I don't think this is a big shift in what we do, but it
> is one that
>     >> > enables people to be able to start running with something
> like K8S on top
>     >> > of their existing ACS.  This is something we are interested
> in doing and
>     >> so
>     >> > are our customers.  If we can have a thin layer in ACS which
> helps enable
>     >> > the use of K8S (or other container orchestrators) by orchestrating
>     >> > infrastructure, as we already do, and making it easier to adopt a
>     >> container
>     >> > orchestrator running on top of ACS, I think that gives us a
> nice foothold
>     >> > in the market.  I don't really feel it is fair to compare
> containers to
>     >> > IPv6.  IPv6 has been out forever and it has taken almost a
> decade to get
>     >> > anyone to adopt it.  Containers have really only been here
> for like 2
>     >> years
>     >> > and they are changing the market landscape in a very real way.
>     >> >
>     >> > Kind of on topic and kind of off topic.  I think understanding our
>     >> approach
>     >> > to containers is going to be important for the ACS community
> as a whole.
>     >> > If we don't offer that market anything, then we will not be
> considered
>     >> and
>     >> > we will lose market share we can't afford to lose.  If we try
> to hitch
>     >> our
>     >> > horse to that cart too much, we will not be able to be agile
> enough and
>     >> > will fail.  I feel like the right approach is for us to know
> that it is a
>     >> > thriving market and continue to do what we do, but to extend
> an olive
>     >> > branch to that market.  I think this sort of implementation
> is the right
>     >> > approach because we are not trying to do too much.  We are
> simply giving
>     >> a
>     >> > foundation on which the next big thing in the container
> orchestration
>     >> world
>     >> > can adopt without us having to compete directly in that
> space.  I think
>     >> we
>     >> > have to focus on what we do best, but at the same time, think
> about what
>     >> we
>     >> > can do to enable that huge market of users to adopt ACS as their
>     >> > foundation.  The ability to offer VMs and containers in the
> same data
>     >> plane
>     >> > is something we have the ability to do, especially with this
> approach,
>     >> and
>     >> > is something that most other softwares can not do.  The
> adoption of
>     >> > containers by bigger organizations will be only part of their
> workload,
>     >> > they will still be running VMs for the foreseeable future.
> Being able to
>     >> > appeal to that market is going to be important for us.
>     >> >
>     >> > Hopefully I don't have too many strong opinions here, but I
> do think we
>     >> > need to be thinking about how we move forward in a world which is
>     >> adopting
>     >> > containers in a very real way.
>     >> >
>     >>
>     >> Understood. I just want to prevent that we add more features to
> CloudStack
>     >> which are poorly maintained. Not judging Murali here at all,
> but I'd rather
>     >> see CloudStack loose code then have it added.
>     >>
>     >> Thinking about LXC, would like to see that removed together
> with various
>     >> other network plugins which I think are rarely used.
>     >>
>     >> Now, the idea of Murali was misunderstood by me. I think it
> would be worth
>     >> more if we would improve our cloud-init support and integration
> in various
>     >> tools which makes it much easier to deploy VMs running containers ON
>     >> CloudStack.
>     >>
>     >> Not so much changing CloudStack code, but rather tooling around it.
>     >>
>     >> If we have CloudStack talking to Kubernetes we suddenly have to
> maintain
>     >> that API integration. Who's going to do that. Realistically.
>     >>
>     >> That's my main worry in this regard.
>     >>
>     >> We have so much more work to do in ACS in total that I don't
> know if this
>     >> is the realistic route. I talk to many people who are not looking at
>     >> containers and are still working with VMs.
>     >>
>     >> There is not a single truth which is true, it really depends on
> who you
>     >> ask.
>     >>
>     >> Wido
>     >>
>     >> > Cheers,
>     >> >
>     >> > Will
>     >> >
>     >> > *Will STEVENS*
>     >> > Lead Developer
>     >> >
>     >> > <https://goo.gl/NYZ8KK>
>     >> >
>     >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> <te...@gmail.com>
> wrote:
>     >> >
>     >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> muralimmreddy@gmail.com
>     >> >
>     >> > > wrote:
>     >> > > > All,
>     >> > > >
>     >> > > > I would like propose native functionality into CloudStack
> to provide
>     >> a
>     >> > > container service through which users out-of-the box can
> use to launch
>     >> > > container based application. Idea is to support ability to
> orchestrate
>     >> the
>     >> > > resources and automate aspects of setting up container
> orchestrator
>     >> through
>     >> > > CloudStack. Public IAAS service providers AWS with its ECS
> [1] and
>     >> google
>     >> > > with GKE [2] already provides ability container applications.
>     >> Competitive
>     >> > > cloud orchestration platforms already have native support
> for container
>     >> > > service. Users of CloudStack both as public cloud providers
> and users
>     >> with
>     >> > > private clouds will benefit with such functionality.
>     >> > > >
>     >> > > > While container orchestrator of user choice can be
> provisioned on
>     >> top of
>     >> > > CloudStack (with out CloudStack being involved) with tools like
>     >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> orchestration
>     >> is
>     >> > > giving user a nice cohesive integration. This proposal
> would like add a
>     >> > > notion of first class CloudStack entity called container
> cluster which
>     >> can
>     >> > > be used to provision resources, scale up, scale down, start
> and stop
>     >> the
>     >> > > cluster of VM’s on which containerised applications can be run.
> For
>     >> actual
>     >> > > container orchestration we will still need container
> orchestrator like
>     >> > > docker swarm, marathon, kubernetes, but CloudStack
> container service
>     >> can
>     >> > > automate setting up of control place automatically.
>     >> > > >
>     >> > >
>     >> > > To be honest I'm torn on this one.
>     >> > >
>     >> > > Containers are a rapid changing thing, and while docker swam,
>     >> > > kubernetes, rancher or whatnot is popular today, they might
> not be
>     >> > > tomorrow.
>     >> > > They might use CoreOS today, but might not tomorrow.
>     >> > >
>     >> > > We have a rather poor track record of staying up to date
> with new
>     >> > > features/versions, and adding a feature that is so rapidly
> changing
>     >> > > is, I fear, going to be hard to maintain.
>     >> > > Want an example, look at xenserver. It is one of the most used
>     >> > > hypervisors we support, yet it took 6 months or so for us
> to support
>     >> > > the latest release.
>     >> > > Or IPv6...
>     >> > >
>     >> > > I don't mean to bash at maintainers/implementers of those
> features, I
>     >> > > appreciate all the work being done in every aspect, but I
> believe we
>     >> > > should be realistic and realize that we have issues with
> keeping stuff
>     >> > > up to date.
>     >> > >
>     >> > > I'd say focus on making sure other tools can do their job
> well against
>     >> > > CloudStack (kops, rancher, ++), but that does not mean I
> will
> -1 the
>     >> > > idea if anyone really wants to go through with it.
>     >> > >
>     >> > > --
>     >> > > Erik
>     >> > >
>     >>
>
>     murali.reddy@shapeblue.com
>     www.shapeblue.com<http://www.shapeblue.com>
>     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     @shapeblue
>
>
>
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not
> authorized to read, retain, copy, print, distribute or use this
> message. If you have received this communication in error, please
> notify the sender and delete all copies of this message. Accelerite, a
> Persistent Systems business does not accept any liability for virus infected mails.
>
>
>



DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.

RE: [PROPOSAL] add native container orchestration service

[Paul Angus <pa...@shapeblue.com>]

#WhatHeSaid

The title is misleading.  The proposal is purely to add the notion of Clusters of VMs to CloudStack.  These may be for databases, containers or anything else that needs 'clusters' of compute. Self-healing and autoscaling are logical next steps to be added. 

Those guys at ShapeBlue have open-sourced their whole k8s container service piece.  If/when the 'cluster' part of that work is added into CloudStack, the k8s specific pieces can be used by anyone who wants to, alternatively they could be used for reference in order to create another types of cluster.  (or ignored completely).




paul.angus@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-----Original Message-----
From: Will Stevens [mailto:williamstevens@gmail.com] 
Sent: 31 January 2017 13:26
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service

s/cloud-init/cloud-config/

On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:

> I think that is covered in this proposal. There is nothing k8s 
> specific in this integration (from what I understand), all the k8s 
> details are passed in via the cloud-init configuration after the cluster has been provisioned.
>
> On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com>
> wrote:
>
>
> There are many container orchestrators. Those container orchestrators 
> are happy to run on any VMs or bare metal machines. K8s is just one of 
> them and there will be more in the future. It may not be a good idea 
> to make CloudStack to be k8s aware. IMO, the relationship between k8s 
> and cloudstack should be similar to application and os. It probably 
> not a good idea to make your OS to be aware of any specific 
> applications so IMHO I don’t think k8s should be native to CloudStack. 
> It makes more sense to provide some generic services that many 
> applications can take advantages of. Some generic resource grouping 
> service makes sense so a group of VMs, baremetal machines or network 
> can be treated as a single entity. Some life cycle management will be 
> necessary for these entities too. We can deploy k8s, swarm, dcos or 
> even non-container specific services on top of CloudStack. The k8s is 
> changing fast. One single tenant installation may need more than one 
> VM group and may actually requires more (k8s federation). It will be a 
> struggle to be in sync if we try to bring k8s specific knowledge into cloudstack.
>
> Regards,
>
>
> --
> Lianghwa Jou
> VP Engineering, Accelerite
> email: lianghwa.jou@accelerite.com
>
>
>
>
>
> On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
>
>
>     I agree with some good views Will has shared and I also agree to 
> the concerns raised by Wido and Eric. IMO we need balance of staying 
> relevant/add new features vs stability of CloudStack and take 
> corrective action if needed. We have two good examples for both. When 
> SDN was hot technology CloudStack ended up with bunch of SDN controller integrations.
> Few years later, now CloudStack is carrying baggage with no 
> maintainers for those plugins, with probably not many of CloudStack users needing overlays.
> On the other hand, other than attempt to liaison with ETSI for NFV no 
> effort was done. OpenStack has become de-facto for NFV. Now for 
> OpenStack, arguably NFV has become larger use case than cloud it self. 
> I concur with Will’s point that with minimal viable solution that does 
> not change the core of CloudStack, and can keep CloudStack relevant is 
> worth to be taken in.
>
>     Will,
>
>     To your question of how different is from ShapeBlue’s container 
> service, its design, implementation and API semantics etc remain same.
> ShapeBlue’s container service was true drop in plug-in to CloudStack, 
> with this proposal I am trying to re-work to make it a core CloudStack 
> pluggable service which is part of CloudStack.
>
>     Key concepts that this proposal is trying to add
>
>         - add notion of ‘container cluster’ as a first class entity in 
> CloudStack. Which is bacially collection of other CloudStack resources 
> (like VM’s, networks, public ip, network rules etc)
>         - life cycle operation to manage ‘container cluster’ like 
> create, delete, start, stop, scale-up, scale-down, heal etc
>         - orchestrate container orchestrator control plane on top of 
> provisioned resources
>
>     At-least for k8s (which is what this proposal is targeting), 
> integration with k8s is bare minimum. There are cloud-config scripts 
> that automatically setup k8s cluster master and node VM’s. All 
> CloudStack is doing in passing the cloud-config to the core OS VM’s as user data.
>
>     Regards
>     Murali Reddy
>
>
>
>
>
>
>
>     On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on 
> behalf of wstevens@cloudops.com> wrote:
>
>     >I agree that we need to be careful what we take on and own inside
>     >CloudStack.  I feel like some of the plugins or integrations 
> which we have
>     >been "maintaining" may serve us better to abandon, but I feel 
> like that is
>     >a whole discussion on its own.
>     >
>     >In this case, I feel like there is a minimum viable solution 
> which puts
>     >CloudStack in a pretty good place to enable container orchestration.
> For
>     >example, one of the biggest challenges with K8S is the fact that it is
>     >single tenant.  CloudStack has good multi tenancy support and has the
>     >ability to orchestrate the underlying infra quite well.  We will 
> have to be
>     >very careful not to try to own too deep into the K8S world 
> though, in my
>     >opinion.  We only want to be responsible for providing the infra 
> (and a way
>     >to bootstrap K8S ideally) and be able to scale the infra, 
> everything else
>     >should be owned by the K8S on top.  That is the way I see it 
> anyway, but
>     >please add your input.
>     >
>     >I think it is a liability to try to go too deep, for the same 
> reasons Wido
>     >and Erik have mentioned.  But I also think we need to take it 
> seriously
>     >because that train is moving and this may be a good opportunity 
> to stay
>     >relevant in a rapidly changing market.
>     >
>     >*Will STEVENS*
>     >Lead Developer
>     >
>     ><https://goo.gl/NYZ8KK>
>     >
>     >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander 
> <wi...@widodh.nl>
> wrote:
>     >
>     >>
>     >> > Op 27 januari 2017 om 16:08 schreef Will Stevens < 
> wstevens@cloudops.com
>     >> >:
>     >> >
>     >> >
>     >> > Hey Murali,
>     >> > How different is this proposal than what ShapeBlue already 
> built.  It
>     >> looks
>     >> > pretty consistent with the functionality that you guys open 
> sourced in
>     >> > Seville.
>     >> >
>     >> > I have not yet used this functionality, but I have reports 
> that it works
>     >> > quite well.
>     >> >
>     >> > I believe the premise here is to only orchestrate the VM layer and
>     >> > basically expose a "group" of running VMs to the user.  The 
> user is
>     >> > responsible for configuring K8S or whatever other container 
> orchestrator
>     >> on
>     >> > top.  I saw mention of the "cloud-config" scripts in the FS, 
> how are
>     >> those
>     >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
>     >> >
>     >> > I believe the core feature that is being requested to be 
> added is the
>     >> > ability to create a group of VMs which will be kept active as 
> a group if
>     >> at
>     >> > all possible.  ACS would be responsible for making sure that 
> the number
>     >> of
>     >> > VMs specified for the group are in running state and it would 
> spin up new
>     >> > VMs as needed in order to satisfy the group settings.  In 
> general, it is
>     >> > understood that any application running on this group would 
> have to be
>     >> > fault tolerant enough to be able to rediscover a new VM if 
> one fails and
>     >> is
>     >> > replaced by a fresh copy.  Is that fair to say?  How is it 
> expected that
>     >> > this service discovery is done, just by VMs being present on 
> the network?
>     >> >
>     >> > As for some of the other people's concerns in this thread.
>     >> >
>     >> > - Regarding Wido's remarks.  I understand that there is some added
>     >> > complexity, but I don't feel like the scope of the addition is
>     >> > unrealistic.  I think the LXC integration was a lot farther 
> out of the
>     >> > scope of what ACS does then this is.  This does not change 
> the "things"
>     >> > which ACS orchestrates, it just adds the concept of a 
> grouping of things
>     >> > which ACS already manages.  I think this is the right 
> approach since it
>     >> is
>     >> > not trying to be a container orchestrator.  We will never 
> compete with
>     >> K8S,
>     >> > for example, and we should not try, but K8S is here and the 
> market wants
>     >> > it.  I do think we should be keeping our head up about that 
> fact because
>     >> > being able to provide a the underlay for K8S is very valuable 
> in the
>     >> > current marketplace.  I see this functionality as a way to 
> enable K8S
>     >> > adoption on top of ACS without changing our core values.
>     >> >
>     >> > - Regarding Erik's remarks.  The container space is moving 
> fast, but so
>     >> is
>     >> > the industry.  If we want to remain relevant, we need to be 
> able to
>     >> adapt a
>     >> > bit.  I don't think this is a big shift in what we do, but it 
> is one that
>     >> > enables people to be able to start running with something 
> like K8S on top
>     >> > of their existing ACS.  This is something we are interested 
> in doing and
>     >> so
>     >> > are our customers.  If we can have a thin layer in ACS which 
> helps enable
>     >> > the use of K8S (or other container orchestrators) by orchestrating
>     >> > infrastructure, as we already do, and making it easier to adopt a
>     >> container
>     >> > orchestrator running on top of ACS, I think that gives us a 
> nice foothold
>     >> > in the market.  I don't really feel it is fair to compare 
> containers to
>     >> > IPv6.  IPv6 has been out forever and it has taken almost a 
> decade to get
>     >> > anyone to adopt it.  Containers have really only been here 
> for like 2
>     >> years
>     >> > and they are changing the market landscape in a very real way.
>     >> >
>     >> > Kind of on topic and kind of off topic.  I think understanding our
>     >> approach
>     >> > to containers is going to be important for the ACS community 
> as a whole.
>     >> > If we don't offer that market anything, then we will not be 
> considered
>     >> and
>     >> > we will lose market share we can't afford to lose.  If we try 
> to hitch
>     >> our
>     >> > horse to that cart too much, we will not be able to be agile 
> enough and
>     >> > will fail.  I feel like the right approach is for us to know 
> that it is a
>     >> > thriving market and continue to do what we do, but to extend 
> an olive
>     >> > branch to that market.  I think this sort of implementation 
> is the right
>     >> > approach because we are not trying to do too much.  We are 
> simply giving
>     >> a
>     >> > foundation on which the next big thing in the container 
> orchestration
>     >> world
>     >> > can adopt without us having to compete directly in that 
> space.  I think
>     >> we
>     >> > have to focus on what we do best, but at the same time, think 
> about what
>     >> we
>     >> > can do to enable that huge market of users to adopt ACS as their
>     >> > foundation.  The ability to offer VMs and containers in the 
> same data
>     >> plane
>     >> > is something we have the ability to do, especially with this 
> approach,
>     >> and
>     >> > is something that most other softwares can not do.  The 
> adoption of
>     >> > containers by bigger organizations will be only part of their 
> workload,
>     >> > they will still be running VMs for the foreseeable future. 
> Being able to
>     >> > appeal to that market is going to be important for us.
>     >> >
>     >> > Hopefully I don't have too many strong opinions here, but I 
> do think we
>     >> > need to be thinking about how we move forward in a world which is
>     >> adopting
>     >> > containers in a very real way.
>     >> >
>     >>
>     >> Understood. I just want to prevent that we add more features to 
> CloudStack
>     >> which are poorly maintained. Not judging Murali here at all, 
> but I'd rather
>     >> see CloudStack loose code then have it added.
>     >>
>     >> Thinking about LXC, would like to see that removed together 
> with various
>     >> other network plugins which I think are rarely used.
>     >>
>     >> Now, the idea of Murali was misunderstood by me. I think it 
> would be worth
>     >> more if we would improve our cloud-init support and integration 
> in various
>     >> tools which makes it much easier to deploy VMs running containers ON
>     >> CloudStack.
>     >>
>     >> Not so much changing CloudStack code, but rather tooling around it.
>     >>
>     >> If we have CloudStack talking to Kubernetes we suddenly have to 
> maintain
>     >> that API integration. Who's going to do that. Realistically.
>     >>
>     >> That's my main worry in this regard.
>     >>
>     >> We have so much more work to do in ACS in total that I don't 
> know if this
>     >> is the realistic route. I talk to many people who are not looking at
>     >> containers and are still working with VMs.
>     >>
>     >> There is not a single truth which is true, it really depends on 
> who you
>     >> ask.
>     >>
>     >> Wido
>     >>
>     >> > Cheers,
>     >> >
>     >> > Will
>     >> >
>     >> > *Will STEVENS*
>     >> > Lead Developer
>     >> >
>     >> > <https://goo.gl/NYZ8KK>
>     >> >
>     >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber 
> <te...@gmail.com>
> wrote:
>     >> >
>     >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy < 
> muralimmreddy@gmail.com
>     >> >
>     >> > > wrote:
>     >> > > > All,
>     >> > > >
>     >> > > > I would like propose native functionality into CloudStack 
> to provide
>     >> a
>     >> > > container service through which users out-of-the box can 
> use to launch
>     >> > > container based application. Idea is to support ability to 
> orchestrate
>     >> the
>     >> > > resources and automate aspects of setting up container 
> orchestrator
>     >> through
>     >> > > CloudStack. Public IAAS service providers AWS with its ECS 
> [1] and
>     >> google
>     >> > > with GKE [2] already provides ability container applications.
>     >> Competitive
>     >> > > cloud orchestration platforms already have native support 
> for container
>     >> > > service. Users of CloudStack both as public cloud providers 
> and users
>     >> with
>     >> > > private clouds will benefit with such functionality.
>     >> > > >
>     >> > > > While container orchestrator of user choice can be 
> provisioned on
>     >> top of
>     >> > > CloudStack (with out CloudStack being involved) with tools like
>     >> > > TerraForm[3], Ansible[4] etc, advantage of having native 
> orchestration
>     >> is
>     >> > > giving user a nice cohesive integration. This proposal 
> would like add a
>     >> > > notion of first class CloudStack entity called container 
> cluster which
>     >> can
>     >> > > be used to provision resources, scale up, scale down, start 
> and stop
>     >> the
>     >> > > cluster of VM’s on which containerised applications can be run.
> For
>     >> actual
>     >> > > container orchestration we will still need container 
> orchestrator like
>     >> > > docker swarm, marathon, kubernetes, but CloudStack 
> container service
>     >> can
>     >> > > automate setting up of control place automatically.
>     >> > > >
>     >> > >
>     >> > > To be honest I'm torn on this one.
>     >> > >
>     >> > > Containers are a rapid changing thing, and while docker swam,
>     >> > > kubernetes, rancher or whatnot is popular today, they might 
> not be
>     >> > > tomorrow.
>     >> > > They might use CoreOS today, but might not tomorrow.
>     >> > >
>     >> > > We have a rather poor track record of staying up to date 
> with new
>     >> > > features/versions, and adding a feature that is so rapidly 
> changing
>     >> > > is, I fear, going to be hard to maintain.
>     >> > > Want an example, look at xenserver. It is one of the most used
>     >> > > hypervisors we support, yet it took 6 months or so for us 
> to support
>     >> > > the latest release.
>     >> > > Or IPv6...
>     >> > >
>     >> > > I don't mean to bash at maintainers/implementers of those 
> features, I
>     >> > > appreciate all the work being done in every aspect, but I 
> believe we
>     >> > > should be realistic and realize that we have issues with 
> keeping stuff
>     >> > > up to date.
>     >> > >
>     >> > > I'd say focus on making sure other tools can do their job 
> well against
>     >> > > CloudStack (kops, rancher, ++), but that does not mean I 
> will
> -1 the
>     >> > > idea if anyone really wants to go through with it.
>     >> > >
>     >> > > --
>     >> > > Erik
>     >> > >
>     >>
>
>     murali.reddy@shapeblue.com
>     www.shapeblue.com
>     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     @shapeblue
>
>
>
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which 
> is the property of Accelerite, a Persistent Systems business. It is 
> intended only for the use of the individual or entity to which it is 
> addressed. If you are not the intended recipient, you are not 
> authorized to read, retain, copy, print, distribute or use this 
> message. If you have received this communication in error, please 
> notify the sender and delete all copies of this message. Accelerite, a 
> Persistent Systems business does not accept any liability for virus infected mails.
>
>
>

Re: [PROPOSAL] add native container orchestration service

[Will Stevens <wi...@gmail.com>]

s/cloud-init/cloud-config/

On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:

> I think that is covered in this proposal. There is nothing k8s specific in
> this integration (from what I understand), all the k8s details are passed
> in via the cloud-init configuration after the cluster has been provisioned.
>
> On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com>
> wrote:
>
>
> There are many container orchestrators. Those container orchestrators are
> happy to run on any VMs or bare metal machines. K8s is just one of them and
> there will be more in the future. It may not be a good idea to make
> CloudStack to be k8s aware. IMO, the relationship between k8s and
> cloudstack should be similar to application and os. It probably not a good
> idea to make your OS to be aware of any specific applications so IMHO I
> don’t think k8s should be native to CloudStack. It makes more sense to
> provide some generic services that many applications can take advantages
> of. Some generic resource grouping service makes sense so a group of VMs,
> baremetal machines or network can be treated as a single entity. Some life
> cycle management will be necessary for these entities too. We can deploy
> k8s, swarm, dcos or even non-container specific services on top of
> CloudStack. The k8s is changing fast. One single tenant installation may
> need more than one VM group and may actually requires more (k8s
> federation). It will be a struggle to be in sync if we try to bring k8s
> specific knowledge into cloudstack.
>
> Regards,
>
>
> --
> Lianghwa Jou
> VP Engineering, Accelerite
> email: lianghwa.jou@accelerite.com
>
>
>
>
>
> On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
>
>
>     I agree with some good views Will has shared and I also agree to the
> concerns raised by Wido and Eric. IMO we need balance of staying
> relevant/add new features vs stability of CloudStack and take corrective
> action if needed. We have two good examples for both. When SDN was hot
> technology CloudStack ended up with bunch of SDN controller integrations.
> Few years later, now CloudStack is carrying baggage with no maintainers for
> those plugins, with probably not many of CloudStack users needing overlays.
> On the other hand, other than attempt to liaison with ETSI for NFV no
> effort was done. OpenStack has become de-facto for NFV. Now for OpenStack,
> arguably NFV has become larger use case than cloud it self. I concur with
> Will’s point that with minimal viable solution that does not change the
> core of CloudStack, and can keep CloudStack relevant is worth to be taken
> in.
>
>     Will,
>
>     To your question of how different is from ShapeBlue’s container
> service, its design, implementation and API semantics etc remain same.
> ShapeBlue’s container service was true drop in plug-in to CloudStack, with
> this proposal I am trying to re-work to make it a core CloudStack pluggable
> service which is part of CloudStack.
>
>     Key concepts that this proposal is trying to add
>
>         - add notion of ‘container cluster’ as a first class entity in
> CloudStack. Which is bacially collection of other CloudStack resources
> (like VM’s, networks, public ip, network rules etc)
>         - life cycle operation to manage ‘container cluster’ like create,
> delete, start, stop, scale-up, scale-down, heal etc
>         - orchestrate container orchestrator control plane on top of
> provisioned resources
>
>     At-least for k8s (which is what this proposal is targeting),
> integration with k8s is bare minimum. There are cloud-config scripts that
> automatically setup k8s cluster master and node VM’s. All CloudStack is
> doing in passing the cloud-config to the core OS VM’s as user data.
>
>     Regards
>     Murali Reddy
>
>
>
>
>
>
>
>     On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
> behalf of wstevens@cloudops.com> wrote:
>
>     >I agree that we need to be careful what we take on and own inside
>     >CloudStack.  I feel like some of the plugins or integrations which we
> have
>     >been "maintaining" may serve us better to abandon, but I feel like
> that is
>     >a whole discussion on its own.
>     >
>     >In this case, I feel like there is a minimum viable solution which
> puts
>     >CloudStack in a pretty good place to enable container orchestration.
> For
>     >example, one of the biggest challenges with K8S is the fact that it is
>     >single tenant.  CloudStack has good multi tenancy support and has the
>     >ability to orchestrate the underlying infra quite well.  We will have
> to be
>     >very careful not to try to own too deep into the K8S world though, in
> my
>     >opinion.  We only want to be responsible for providing the infra (and
> a way
>     >to bootstrap K8S ideally) and be able to scale the infra, everything
> else
>     >should be owned by the K8S on top.  That is the way I see it anyway,
> but
>     >please add your input.
>     >
>     >I think it is a liability to try to go too deep, for the same reasons
> Wido
>     >and Erik have mentioned.  But I also think we need to take it
> seriously
>     >because that train is moving and this may be a good opportunity to
> stay
>     >relevant in a rapidly changing market.
>     >
>     >*Will STEVENS*
>     >Lead Developer
>     >
>     ><https://goo.gl/NYZ8KK>
>     >
>     >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander <wi...@widodh.nl>
> wrote:
>     >
>     >>
>     >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> wstevens@cloudops.com
>     >> >:
>     >> >
>     >> >
>     >> > Hey Murali,
>     >> > How different is this proposal than what ShapeBlue already
> built.  It
>     >> looks
>     >> > pretty consistent with the functionality that you guys open
> sourced in
>     >> > Seville.
>     >> >
>     >> > I have not yet used this functionality, but I have reports that
> it works
>     >> > quite well.
>     >> >
>     >> > I believe the premise here is to only orchestrate the VM layer and
>     >> > basically expose a "group" of running VMs to the user.  The user
> is
>     >> > responsible for configuring K8S or whatever other container
> orchestrator
>     >> on
>     >> > top.  I saw mention of the "cloud-config" scripts in the FS, how
> are
>     >> those
>     >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
>     >> >
>     >> > I believe the core feature that is being requested to be added is
> the
>     >> > ability to create a group of VMs which will be kept active as a
> group if
>     >> at
>     >> > all possible.  ACS would be responsible for making sure that the
> number
>     >> of
>     >> > VMs specified for the group are in running state and it would
> spin up new
>     >> > VMs as needed in order to satisfy the group settings.  In
> general, it is
>     >> > understood that any application running on this group would have
> to be
>     >> > fault tolerant enough to be able to rediscover a new VM if one
> fails and
>     >> is
>     >> > replaced by a fresh copy.  Is that fair to say?  How is it
> expected that
>     >> > this service discovery is done, just by VMs being present on the
> network?
>     >> >
>     >> > As for some of the other people's concerns in this thread.
>     >> >
>     >> > - Regarding Wido's remarks.  I understand that there is some added
>     >> > complexity, but I don't feel like the scope of the addition is
>     >> > unrealistic.  I think the LXC integration was a lot farther out
> of the
>     >> > scope of what ACS does then this is.  This does not change the
> "things"
>     >> > which ACS orchestrates, it just adds the concept of a grouping of
> things
>     >> > which ACS already manages.  I think this is the right approach
> since it
>     >> is
>     >> > not trying to be a container orchestrator.  We will never compete
> with
>     >> K8S,
>     >> > for example, and we should not try, but K8S is here and the
> market wants
>     >> > it.  I do think we should be keeping our head up about that fact
> because
>     >> > being able to provide a the underlay for K8S is very valuable in
> the
>     >> > current marketplace.  I see this functionality as a way to enable
> K8S
>     >> > adoption on top of ACS without changing our core values.
>     >> >
>     >> > - Regarding Erik's remarks.  The container space is moving fast,
> but so
>     >> is
>     >> > the industry.  If we want to remain relevant, we need to be able
> to
>     >> adapt a
>     >> > bit.  I don't think this is a big shift in what we do, but it is
> one that
>     >> > enables people to be able to start running with something like
> K8S on top
>     >> > of their existing ACS.  This is something we are interested in
> doing and
>     >> so
>     >> > are our customers.  If we can have a thin layer in ACS which
> helps enable
>     >> > the use of K8S (or other container orchestrators) by orchestrating
>     >> > infrastructure, as we already do, and making it easier to adopt a
>     >> container
>     >> > orchestrator running on top of ACS, I think that gives us a nice
> foothold
>     >> > in the market.  I don't really feel it is fair to compare
> containers to
>     >> > IPv6.  IPv6 has been out forever and it has taken almost a decade
> to get
>     >> > anyone to adopt it.  Containers have really only been here for
> like 2
>     >> years
>     >> > and they are changing the market landscape in a very real way.
>     >> >
>     >> > Kind of on topic and kind of off topic.  I think understanding our
>     >> approach
>     >> > to containers is going to be important for the ACS community as a
> whole.
>     >> > If we don't offer that market anything, then we will not be
> considered
>     >> and
>     >> > we will lose market share we can't afford to lose.  If we try to
> hitch
>     >> our
>     >> > horse to that cart too much, we will not be able to be agile
> enough and
>     >> > will fail.  I feel like the right approach is for us to know that
> it is a
>     >> > thriving market and continue to do what we do, but to extend an
> olive
>     >> > branch to that market.  I think this sort of implementation is
> the right
>     >> > approach because we are not trying to do too much.  We are simply
> giving
>     >> a
>     >> > foundation on which the next big thing in the container
> orchestration
>     >> world
>     >> > can adopt without us having to compete directly in that space.  I
> think
>     >> we
>     >> > have to focus on what we do best, but at the same time, think
> about what
>     >> we
>     >> > can do to enable that huge market of users to adopt ACS as their
>     >> > foundation.  The ability to offer VMs and containers in the same
> data
>     >> plane
>     >> > is something we have the ability to do, especially with this
> approach,
>     >> and
>     >> > is something that most other softwares can not do.  The adoption
> of
>     >> > containers by bigger organizations will be only part of their
> workload,
>     >> > they will still be running VMs for the foreseeable future. Being
> able to
>     >> > appeal to that market is going to be important for us.
>     >> >
>     >> > Hopefully I don't have too many strong opinions here, but I do
> think we
>     >> > need to be thinking about how we move forward in a world which is
>     >> adopting
>     >> > containers in a very real way.
>     >> >
>     >>
>     >> Understood. I just want to prevent that we add more features to
> CloudStack
>     >> which are poorly maintained. Not judging Murali here at all, but
> I'd rather
>     >> see CloudStack loose code then have it added.
>     >>
>     >> Thinking about LXC, would like to see that removed together with
> various
>     >> other network plugins which I think are rarely used.
>     >>
>     >> Now, the idea of Murali was misunderstood by me. I think it would
> be worth
>     >> more if we would improve our cloud-init support and integration in
> various
>     >> tools which makes it much easier to deploy VMs running containers ON
>     >> CloudStack.
>     >>
>     >> Not so much changing CloudStack code, but rather tooling around it.
>     >>
>     >> If we have CloudStack talking to Kubernetes we suddenly have to
> maintain
>     >> that API integration. Who's going to do that. Realistically.
>     >>
>     >> That's my main worry in this regard.
>     >>
>     >> We have so much more work to do in ACS in total that I don't know
> if this
>     >> is the realistic route. I talk to many people who are not looking at
>     >> containers and are still working with VMs.
>     >>
>     >> There is not a single truth which is true, it really depends on who
> you
>     >> ask.
>     >>
>     >> Wido
>     >>
>     >> > Cheers,
>     >> >
>     >> > Will
>     >> >
>     >> > *Will STEVENS*
>     >> > Lead Developer
>     >> >
>     >> > <https://goo.gl/NYZ8KK>
>     >> >
>     >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com>
> wrote:
>     >> >
>     >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> muralimmreddy@gmail.com
>     >> >
>     >> > > wrote:
>     >> > > > All,
>     >> > > >
>     >> > > > I would like propose native functionality into CloudStack to
> provide
>     >> a
>     >> > > container service through which users out-of-the box can use to
> launch
>     >> > > container based application. Idea is to support ability to
> orchestrate
>     >> the
>     >> > > resources and automate aspects of setting up container
> orchestrator
>     >> through
>     >> > > CloudStack. Public IAAS service providers AWS with its ECS [1]
> and
>     >> google
>     >> > > with GKE [2] already provides ability container applications.
>     >> Competitive
>     >> > > cloud orchestration platforms already have native support for
> container
>     >> > > service. Users of CloudStack both as public cloud providers and
> users
>     >> with
>     >> > > private clouds will benefit with such functionality.
>     >> > > >
>     >> > > > While container orchestrator of user choice can be
> provisioned on
>     >> top of
>     >> > > CloudStack (with out CloudStack being involved) with tools like
>     >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> orchestration
>     >> is
>     >> > > giving user a nice cohesive integration. This proposal would
> like add a
>     >> > > notion of first class CloudStack entity called container
> cluster which
>     >> can
>     >> > > be used to provision resources, scale up, scale down, start and
> stop
>     >> the
>     >> > > cluster of VM’s on which containerised applications can be run.
> For
>     >> actual
>     >> > > container orchestration we will still need container
> orchestrator like
>     >> > > docker swarm, marathon, kubernetes, but CloudStack container
> service
>     >> can
>     >> > > automate setting up of control place automatically.
>     >> > > >
>     >> > >
>     >> > > To be honest I'm torn on this one.
>     >> > >
>     >> > > Containers are a rapid changing thing, and while docker swam,
>     >> > > kubernetes, rancher or whatnot is popular today, they might not
> be
>     >> > > tomorrow.
>     >> > > They might use CoreOS today, but might not tomorrow.
>     >> > >
>     >> > > We have a rather poor track record of staying up to date with
> new
>     >> > > features/versions, and adding a feature that is so rapidly
> changing
>     >> > > is, I fear, going to be hard to maintain.
>     >> > > Want an example, look at xenserver. It is one of the most used
>     >> > > hypervisors we support, yet it took 6 months or so for us to
> support
>     >> > > the latest release.
>     >> > > Or IPv6...
>     >> > >
>     >> > > I don't mean to bash at maintainers/implementers of those
> features, I
>     >> > > appreciate all the work being done in every aspect, but I
> believe we
>     >> > > should be realistic and realize that we have issues with
> keeping stuff
>     >> > > up to date.
>     >> > >
>     >> > > I'd say focus on making sure other tools can do their job well
> against
>     >> > > CloudStack (kops, rancher, ++), but that does not mean I will
> -1 the
>     >> > > idea if anyone really wants to go through with it.
>     >> > >
>     >> > > --
>     >> > > Erik
>     >> > >
>     >>
>
>     murali.reddy@shapeblue.com
>     www.shapeblue.com
>     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     @shapeblue
>
>
>
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>
>
>

Re: [PROPOSAL] add native container orchestration service

[Syed Ahmed <sa...@cloudops.com>]

Sorry for jumping into this late. I was swamped with work and finally found
some time to look at this. I like the idea of havig a native container
cluster. I think this can be very powerful to natively provision any of the
container orchestrators. I also think that orchestrating containers should
not be done by Cloudstack but should be offloaded to other orchestrators
while Cloudstack provides the low level provisioning of VMs and networks.
This could also be very powerful with baremetal provisioning(something
which I've been working on recently). Cloudstack can provision baremetal
servers which can run k8s or Mesos. This would give a lot of power and
flexibility to users who want performance.

Here are my comments:

1. Is the assumption that the container orchestrator (k8s or Mesos) are
already installed and running elsewhere? or does Cloudstack spin up a VM
and install them. Depending on what happens, there will be different
questions. For example, if CS is installing k8s, how would it do it? or if
k8s is already installed, how would the network rules configured so that it
can reach the VMs

2. What are `memory` and `cores` in the `container_cluster` table used for
given you already have service offering and number of nodes.

3. When in `Alert` state, if the recovery fails, does it give up or keeps
trying? Also, is the switching between `Alert` and `Recover` state timeout
based?


Other than the above comments it looks good from my side. Kudos Murali!

On Fri, Feb 3, 2017 at 9:20 AM, Kishan Kavala <ki...@accelerite.com>
wrote:

> Murali,
>  APIs are similar, but the name is generic - createServiceCluster instead
> of createContainerCluster
> Same with DB tables names - service_cluster instead of container_cluster
> API params or the DB columns in the spec you published don't have anything
> specific to container_clusters and can be used for any service.
>
> -----Original Message-----
> From: Murali Reddy [mailto:muralimmreddy@gmail.com]
> Sent: 02 February 2017 02:34 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
>
> Thanks for the feedback. There have been some concerns here with this
> proposal. I think we had sufficient valid arguments why we need this
> functionality in CloudStack. While this proposal is to use k8s as container
> orchestrator design should be able to adopt other container orchestrator.
>
>
> Unless anybody has any specific technical points on this, I'd like to go
> on with its implementation and open up a PR.
>
>
> Kishan, when do you think you can share further details (northbound API
> semantics etc) on the work you are doing? I am planning to get this feature
> the for 4.11.
>
>
>
> On 31/01/17, 5:54 PM, "Will Stevens" <wi...@gmail.com> wrote:
>
> >I think that is covered in this proposal. There is nothing k8s specific
> >in this integration (from what I understand), all the k8s details are
> >passed in via the cloud-init configuration after the cluster has been
> provisioned.
> >
> >On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com>
> wrote:
> >
> >
> >There are many container orchestrators. Those container orchestrators
> >are happy to run on any VMs or bare metal machines. K8s is just one of
> >them and there will be more in the future. It may not be a good idea to
> >make CloudStack to be k8s aware. IMO, the relationship between k8s and
> >cloudstack should be similar to application and os. It probably not a
> >good idea to make your OS to be aware of any specific applications so
> >IMHO I don’t think k8s should be native to CloudStack. It makes more
> >sense to provide some generic services that many applications can take
> >advantages of. Some generic resource grouping service makes sense so a
> >group of VMs, baremetal machines or network can be treated as a single
> >entity. Some life cycle management will be necessary for these entities
> >too. We can deploy k8s, swarm, dcos or even non-container specific
> >services on top of CloudStack. The k8s is changing fast. One single
> >tenant installation may need more than one VM group and may actually
> >requires more (k8s federation). It will be a struggle to be in sync if
> >we try to bring k8s specific knowledge into cloudstack.
> >
> >Regards,
> >
> >
> >--
> >Lianghwa Jou
> >VP Engineering, Accelerite
> >email: lianghwa.jou@accelerite.com
> >
> >
> >
> >
> >
> >On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
> >
> >
> >    I agree with some good views Will has shared and I also agree to
> >the concerns raised by Wido and Eric. IMO we need balance of staying
> >relevant/add new features vs stability of CloudStack and take
> >corrective action if needed. We have two good examples for both. When
> >SDN was hot technology CloudStack ended up with bunch of SDN controller
> integrations.
> >Few years later, now CloudStack is carrying baggage with no maintainers
> >for those plugins, with probably not many of CloudStack users needing
> overlays.
> >On the other hand, other than attempt to liaison with ETSI for NFV no
> >effort was done. OpenStack has become de-facto for NFV. Now for
> >OpenStack, arguably NFV has become larger use case than cloud it self.
> >I concur with Will’s point that with minimal viable solution that does
> >not change the core of CloudStack, and can keep CloudStack relevant is
> >worth to be taken in.
> >
> >    Will,
> >
> >    To your question of how different is from ShapeBlue’s container
> >service, its design, implementation and API semantics etc remain same.
> >ShapeBlue’s container service was true drop in plug-in to CloudStack,
> >with this proposal I am trying to re-work to make it a core CloudStack
> >pluggable service which is part of CloudStack.
> >
> >    Key concepts that this proposal is trying to add
> >
> >        - add notion of ‘container cluster’ as a first class entity in
> >CloudStack. Which is bacially collection of other CloudStack resources
> >(like VM’s, networks, public ip, network rules etc)
> >        - life cycle operation to manage ‘container cluster’ like
> >create, delete, start, stop, scale-up, scale-down, heal etc
> >        - orchestrate container orchestrator control plane on top of
> >provisioned resources
> >
> >    At-least for k8s (which is what this proposal is targeting),
> >integration with k8s is bare minimum. There are cloud-config scripts
> >that automatically setup k8s cluster master and node VM’s. All
> >CloudStack is doing in passing the cloud-config to the core OS VM’s as
> user data.
> >
> >    Regards
> >    Murali Reddy
> >
> >
> >
> >
> >
> >
> >
> >    On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
> >behalf of wstevens@cloudops.com> wrote:
> >
> >    >I agree that we need to be careful what we take on and own inside
> >    >CloudStack.  I feel like some of the plugins or integrations which
> >we have
> >    >been "maintaining" may serve us better to abandon, but I feel like
> >that is
> >    >a whole discussion on its own.
> >    >
> >    >In this case, I feel like there is a minimum viable solution which
> puts
> >    >CloudStack in a pretty good place to enable container orchestration.
> >For
> >    >example, one of the biggest challenges with K8S is the fact that it
> is
> >    >single tenant.  CloudStack has good multi tenancy support and has the
> >    >ability to orchestrate the underlying infra quite well.  We will
> >have to be
> >    >very careful not to try to own too deep into the K8S world though,
> >in my
> >    >opinion.  We only want to be responsible for providing the infra
> >(and a way
> >    >to bootstrap K8S ideally) and be able to scale the infra,
> >everything else
> >    >should be owned by the K8S on top.  That is the way I see it
> >anyway, but
> >    >please add your input.
> >    >
> >    >I think it is a liability to try to go too deep, for the same
> >reasons Wido
> >    >and Erik have mentioned.  But I also think we need to take it
> seriously
> >    >because that train is moving and this may be a good opportunity to
> stay
> >    >relevant in a rapidly changing market.
> >    >
> >    >*Will STEVENS*
> >    >Lead Developer
> >    >
> >    ><https://goo.gl/NYZ8KK>
> >    >
> >    >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> ><wi...@widodh.nl>
> >wrote:
> >    >
> >    >>
> >    >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> >wstevens@cloudops.com
> >    >> >:
> >    >> >
> >    >> >
> >    >> > Hey Murali,
> >    >> > How different is this proposal than what ShapeBlue already built.
> >It
> >    >> looks
> >    >> > pretty consistent with the functionality that you guys open
> >sourced in
> >    >> > Seville.
> >    >> >
> >    >> > I have not yet used this functionality, but I have reports
> >that it works
> >    >> > quite well.
> >    >> >
> >    >> > I believe the premise here is to only orchestrate the VM layer
> and
> >    >> > basically expose a "group" of running VMs to the user.  The user
> is
> >    >> > responsible for configuring K8S or whatever other container
> >orchestrator
> >    >> on
> >    >> > top.  I saw mention of the "cloud-config" scripts in the FS,
> >how are
> >    >> those
> >    >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
> >    >> >
> >    >> > I believe the core feature that is being requested to be added
> >is the
> >    >> > ability to create a group of VMs which will be kept active as
> >a group if
> >    >> at
> >    >> > all possible.  ACS would be responsible for making sure that
> >the number
> >    >> of
> >    >> > VMs specified for the group are in running state and it would
> >spin up new
> >    >> > VMs as needed in order to satisfy the group settings.  In
> >general, it is
> >    >> > understood that any application running on this group would
> >have to be
> >    >> > fault tolerant enough to be able to rediscover a new VM if one
> >fails and
> >    >> is
> >    >> > replaced by a fresh copy.  Is that fair to say?  How is it
> >expected that
> >    >> > this service discovery is done, just by VMs being present on
> >the network?
> >    >> >
> >    >> > As for some of the other people's concerns in this thread.
> >    >> >
> >    >> > - Regarding Wido's remarks.  I understand that there is some
> added
> >    >> > complexity, but I don't feel like the scope of the addition is
> >    >> > unrealistic.  I think the LXC integration was a lot farther
> >out of the
> >    >> > scope of what ACS does then this is.  This does not change the
> >"things"
> >    >> > which ACS orchestrates, it just adds the concept of a grouping
> >of things
> >    >> > which ACS already manages.  I think this is the right approach
> >since it
> >    >> is
> >    >> > not trying to be a container orchestrator.  We will never
> >compete with
> >    >> K8S,
> >    >> > for example, and we should not try, but K8S is here and the
> >market wants
> >    >> > it.  I do think we should be keeping our head up about that
> >fact because
> >    >> > being able to provide a the underlay for K8S is very valuable
> >in the
> >    >> > current marketplace.  I see this functionality as a way to
> >enable K8S
> >    >> > adoption on top of ACS without changing our core values.
> >    >> >
> >    >> > - Regarding Erik's remarks.  The container space is moving
> >fast, but so
> >    >> is
> >    >> > the industry.  If we want to remain relevant, we need to be able
> to
> >    >> adapt a
> >    >> > bit.  I don't think this is a big shift in what we do, but it
> >is one that
> >    >> > enables people to be able to start running with something like
> >K8S on top
> >    >> > of their existing ACS.  This is something we are interested in
> >doing and
> >    >> so
> >    >> > are our customers.  If we can have a thin layer in ACS which
> >helps enable
> >    >> > the use of K8S (or other container orchestrators) by
> orchestrating
> >    >> > infrastructure, as we already do, and making it easier to adopt a
> >    >> container
> >    >> > orchestrator running on top of ACS, I think that gives us a
> >nice foothold
> >    >> > in the market.  I don't really feel it is fair to compare
> >containers to
> >    >> > IPv6.  IPv6 has been out forever and it has taken almost a
> >decade to get
> >    >> > anyone to adopt it.  Containers have really only been here for
> >like 2
> >    >> years
> >    >> > and they are changing the market landscape in a very real way.
> >    >> >
> >    >> > Kind of on topic and kind of off topic.  I think understanding
> our
> >    >> approach
> >    >> > to containers is going to be important for the ACS community
> >as a whole.
> >    >> > If we don't offer that market anything, then we will not be
> >considered
> >    >> and
> >    >> > we will lose market share we can't afford to lose.  If we try
> >to hitch
> >    >> our
> >    >> > horse to that cart too much, we will not be able to be agile
> >enough and
> >    >> > will fail.  I feel like the right approach is for us to know
> >that it is a
> >    >> > thriving market and continue to do what we do, but to extend
> >an olive
> >    >> > branch to that market.  I think this sort of implementation is
> >the right
> >    >> > approach because we are not trying to do too much.  We are
> >simply giving
> >    >> a
> >    >> > foundation on which the next big thing in the container
> >orchestration
> >    >> world
> >    >> > can adopt without us having to compete directly in that space.
> >I think
> >    >> we
> >    >> > have to focus on what we do best, but at the same time, think
> >about what
> >    >> we
> >    >> > can do to enable that huge market of users to adopt ACS as their
> >    >> > foundation.  The ability to offer VMs and containers in the
> >same data
> >    >> plane
> >    >> > is something we have the ability to do, especially with this
> >approach,
> >    >> and
> >    >> > is something that most other softwares can not do.  The adoption
> of
> >    >> > containers by bigger organizations will be only part of their
> >workload,
> >    >> > they will still be running VMs for the foreseeable future.
> >Being able to
> >    >> > appeal to that market is going to be important for us.
> >    >> >
> >    >> > Hopefully I don't have too many strong opinions here, but I do
> >think we
> >    >> > need to be thinking about how we move forward in a world which is
> >    >> adopting
> >    >> > containers in a very real way.
> >    >> >
> >    >>
> >    >> Understood. I just want to prevent that we add more features to
> >CloudStack
> >    >> which are poorly maintained. Not judging Murali here at all, but
> >I'd rather
> >    >> see CloudStack loose code then have it added.
> >    >>
> >    >> Thinking about LXC, would like to see that removed together with
> >various
> >    >> other network plugins which I think are rarely used.
> >    >>
> >    >> Now, the idea of Murali was misunderstood by me. I think it
> >would be worth
> >    >> more if we would improve our cloud-init support and integration
> >in various
> >    >> tools which makes it much easier to deploy VMs running containers
> ON
> >    >> CloudStack.
> >    >>
> >    >> Not so much changing CloudStack code, but rather tooling around it.
> >    >>
> >    >> If we have CloudStack talking to Kubernetes we suddenly have to
> >maintain
> >    >> that API integration. Who's going to do that. Realistically.
> >    >>
> >    >> That's my main worry in this regard.
> >    >>
> >    >> We have so much more work to do in ACS in total that I don't
> >know if this
> >    >> is the realistic route. I talk to many people who are not looking
> at
> >    >> containers and are still working with VMs.
> >    >>
> >    >> There is not a single truth which is true, it really depends on
> >who you
> >    >> ask.
> >    >>
> >    >> Wido
> >    >>
> >    >> > Cheers,
> >    >> >
> >    >> > Will
> >    >> >
> >    >> > *Will STEVENS*
> >    >> > Lead Developer
> >    >> >
> >    >> > <https://goo.gl/NYZ8KK>
> >    >> >
> >    >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> ><te...@gmail.com>
> >wrote:
> >    >> >
> >    >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> >muralimmreddy@gmail.com
> >    >> >
> >    >> > > wrote:
> >    >> > > > All,
> >    >> > > >
> >    >> > > > I would like propose native functionality into CloudStack
> >to provide
> >    >> a
> >    >> > > container service through which users out-of-the box can use
> >to launch
> >    >> > > container based application. Idea is to support ability to
> >orchestrate
> >    >> the
> >    >> > > resources and automate aspects of setting up container
> >orchestrator
> >    >> through
> >    >> > > CloudStack. Public IAAS service providers AWS with its ECS
> >[1] and
> >    >> google
> >    >> > > with GKE [2] already provides ability container applications.
> >    >> Competitive
> >    >> > > cloud orchestration platforms already have native support
> >for container
> >    >> > > service. Users of CloudStack both as public cloud providers
> >and users
> >    >> with
> >    >> > > private clouds will benefit with such functionality.
> >    >> > > >
> >    >> > > > While container orchestrator of user choice can be
> >provisioned on
> >    >> top of
> >    >> > > CloudStack (with out CloudStack being involved) with tools like
> >    >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> >orchestration
> >    >> is
> >    >> > > giving user a nice cohesive integration. This proposal would
> >like add a
> >    >> > > notion of first class CloudStack entity called container
> >cluster which
> >    >> can
> >    >> > > be used to provision resources, scale up, scale down, start
> >and stop
> >    >> the
> >    >> > > cluster of VM’s on which containerised applications can be run.
> >For
> >    >> actual
> >    >> > > container orchestration we will still need container
> >orchestrator like
> >    >> > > docker swarm, marathon, kubernetes, but CloudStack container
> >service
> >    >> can
> >    >> > > automate setting up of control place automatically.
> >    >> > > >
> >    >> > >
> >    >> > > To be honest I'm torn on this one.
> >    >> > >
> >    >> > > Containers are a rapid changing thing, and while docker swam,
> >    >> > > kubernetes, rancher or whatnot is popular today, they might
> >not be
> >    >> > > tomorrow.
> >    >> > > They might use CoreOS today, but might not tomorrow.
> >    >> > >
> >    >> > > We have a rather poor track record of staying up to date with
> new
> >    >> > > features/versions, and adding a feature that is so rapidly
> >changing
> >    >> > > is, I fear, going to be hard to maintain.
> >    >> > > Want an example, look at xenserver. It is one of the most used
> >    >> > > hypervisors we support, yet it took 6 months or so for us to
> >support
> >    >> > > the latest release.
> >    >> > > Or IPv6...
> >    >> > >
> >    >> > > I don't mean to bash at maintainers/implementers of those
> >features, I
> >    >> > > appreciate all the work being done in every aspect, but I
> >believe we
> >    >> > > should be realistic and realize that we have issues with
> >keeping stuff
> >    >> > > up to date.
> >    >> > >
> >    >> > > I'd say focus on making sure other tools can do their job
> >well against
> >    >> > > CloudStack (kops, rancher, ++), but that does not mean I
> >will -1 the
> >    >> > > idea if anyone really wants to go through with it.
> >    >> > >
> >    >> > > --
> >    >> > > Erik
> >    >> > >
> >    >>
> >
> >    murali.reddy@shapeblue.com
> >    www.shapeblue.com
> >    53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >    @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >DISCLAIMER
> >==========
> >This e-mail may contain privileged and confidential information which
> >is the property of Accelerite, a Persistent Systems business. It is
> >intended only for the use of the individual or entity to which it is
> >addressed. If you are not the intended recipient, you are not
> >authorized to read, retain, copy, print, distribute or use this
> >message. If you have received this communication in error, please
> >notify the sender and delete all copies of this message. Accelerite, a
> >Persistent Systems business does not accept any liability for virus
> infected mails.
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>

RE: [PROPOSAL] add native container orchestration service

[Kishan Kavala <ki...@accelerite.com>]

Murali,
 APIs are similar, but the name is generic - createServiceCluster instead of createContainerCluster
Same with DB tables names - service_cluster instead of container_cluster
API params or the DB columns in the spec you published don't have anything specific to container_clusters and can be used for any service.

-----Original Message-----
From: Murali Reddy [mailto:muralimmreddy@gmail.com] 
Sent: 02 February 2017 02:34 PM
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service


Thanks for the feedback. There have been some concerns here with this proposal. I think we had sufficient valid arguments why we need this functionality in CloudStack. While this proposal is to use k8s as container orchestrator design should be able to adopt other container orchestrator.


Unless anybody has any specific technical points on this, I'd like to go on with its implementation and open up a PR.


Kishan, when do you think you can share further details (northbound API semantics etc) on the work you are doing? I am planning to get this feature the for 4.11. 



On 31/01/17, 5:54 PM, "Will Stevens" <wi...@gmail.com> wrote:

>I think that is covered in this proposal. There is nothing k8s specific 
>in this integration (from what I understand), all the k8s details are 
>passed in via the cloud-init configuration after the cluster has been provisioned.
>
>On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com> wrote:
>
>
>There are many container orchestrators. Those container orchestrators 
>are happy to run on any VMs or bare metal machines. K8s is just one of 
>them and there will be more in the future. It may not be a good idea to 
>make CloudStack to be k8s aware. IMO, the relationship between k8s and 
>cloudstack should be similar to application and os. It probably not a 
>good idea to make your OS to be aware of any specific applications so 
>IMHO I don’t think k8s should be native to CloudStack. It makes more 
>sense to provide some generic services that many applications can take 
>advantages of. Some generic resource grouping service makes sense so a 
>group of VMs, baremetal machines or network can be treated as a single 
>entity. Some life cycle management will be necessary for these entities 
>too. We can deploy k8s, swarm, dcos or even non-container specific 
>services on top of CloudStack. The k8s is changing fast. One single 
>tenant installation may need more than one VM group and may actually 
>requires more (k8s federation). It will be a struggle to be in sync if 
>we try to bring k8s specific knowledge into cloudstack.
>
>Regards,
>
>
>--
>Lianghwa Jou
>VP Engineering, Accelerite
>email: lianghwa.jou@accelerite.com
>
>
>
>
>
>On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
>
>
>    I agree with some good views Will has shared and I also agree to 
>the concerns raised by Wido and Eric. IMO we need balance of staying 
>relevant/add new features vs stability of CloudStack and take 
>corrective action if needed. We have two good examples for both. When 
>SDN was hot technology CloudStack ended up with bunch of SDN controller integrations.
>Few years later, now CloudStack is carrying baggage with no maintainers 
>for those plugins, with probably not many of CloudStack users needing overlays.
>On the other hand, other than attempt to liaison with ETSI for NFV no 
>effort was done. OpenStack has become de-facto for NFV. Now for 
>OpenStack, arguably NFV has become larger use case than cloud it self. 
>I concur with Will’s point that with minimal viable solution that does 
>not change the core of CloudStack, and can keep CloudStack relevant is 
>worth to be taken in.
>
>    Will,
>
>    To your question of how different is from ShapeBlue’s container 
>service, its design, implementation and API semantics etc remain same.
>ShapeBlue’s container service was true drop in plug-in to CloudStack, 
>with this proposal I am trying to re-work to make it a core CloudStack 
>pluggable service which is part of CloudStack.
>
>    Key concepts that this proposal is trying to add
>
>        - add notion of ‘container cluster’ as a first class entity in 
>CloudStack. Which is bacially collection of other CloudStack resources 
>(like VM’s, networks, public ip, network rules etc)
>        - life cycle operation to manage ‘container cluster’ like 
>create, delete, start, stop, scale-up, scale-down, heal etc
>        - orchestrate container orchestrator control plane on top of 
>provisioned resources
>
>    At-least for k8s (which is what this proposal is targeting), 
>integration with k8s is bare minimum. There are cloud-config scripts 
>that automatically setup k8s cluster master and node VM’s. All 
>CloudStack is doing in passing the cloud-config to the core OS VM’s as user data.
>
>    Regards
>    Murali Reddy
>
>
>
>
>
>
>
>    On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on 
>behalf of wstevens@cloudops.com> wrote:
>
>    >I agree that we need to be careful what we take on and own inside
>    >CloudStack.  I feel like some of the plugins or integrations which 
>we have
>    >been "maintaining" may serve us better to abandon, but I feel like 
>that is
>    >a whole discussion on its own.
>    >
>    >In this case, I feel like there is a minimum viable solution which puts
>    >CloudStack in a pretty good place to enable container orchestration.
>For
>    >example, one of the biggest challenges with K8S is the fact that it is
>    >single tenant.  CloudStack has good multi tenancy support and has the
>    >ability to orchestrate the underlying infra quite well.  We will 
>have to be
>    >very careful not to try to own too deep into the K8S world though, 
>in my
>    >opinion.  We only want to be responsible for providing the infra 
>(and a way
>    >to bootstrap K8S ideally) and be able to scale the infra, 
>everything else
>    >should be owned by the K8S on top.  That is the way I see it 
>anyway, but
>    >please add your input.
>    >
>    >I think it is a liability to try to go too deep, for the same 
>reasons Wido
>    >and Erik have mentioned.  But I also think we need to take it seriously
>    >because that train is moving and this may be a good opportunity to stay
>    >relevant in a rapidly changing market.
>    >
>    >*Will STEVENS*
>    >Lead Developer
>    >
>    ><https://goo.gl/NYZ8KK>
>    >
>    >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander 
><wi...@widodh.nl>
>wrote:
>    >
>    >>
>    >> > Op 27 januari 2017 om 16:08 schreef Will Stevens < 
>wstevens@cloudops.com
>    >> >:
>    >> >
>    >> >
>    >> > Hey Murali,
>    >> > How different is this proposal than what ShapeBlue already built.
>It
>    >> looks
>    >> > pretty consistent with the functionality that you guys open 
>sourced in
>    >> > Seville.
>    >> >
>    >> > I have not yet used this functionality, but I have reports 
>that it works
>    >> > quite well.
>    >> >
>    >> > I believe the premise here is to only orchestrate the VM layer and
>    >> > basically expose a "group" of running VMs to the user.  The user is
>    >> > responsible for configuring K8S or whatever other container 
>orchestrator
>    >> on
>    >> > top.  I saw mention of the "cloud-config" scripts in the FS, 
>how are
>    >> those
>    >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
>    >> >
>    >> > I believe the core feature that is being requested to be added 
>is the
>    >> > ability to create a group of VMs which will be kept active as 
>a group if
>    >> at
>    >> > all possible.  ACS would be responsible for making sure that 
>the number
>    >> of
>    >> > VMs specified for the group are in running state and it would 
>spin up new
>    >> > VMs as needed in order to satisfy the group settings.  In 
>general, it is
>    >> > understood that any application running on this group would 
>have to be
>    >> > fault tolerant enough to be able to rediscover a new VM if one 
>fails and
>    >> is
>    >> > replaced by a fresh copy.  Is that fair to say?  How is it 
>expected that
>    >> > this service discovery is done, just by VMs being present on 
>the network?
>    >> >
>    >> > As for some of the other people's concerns in this thread.
>    >> >
>    >> > - Regarding Wido's remarks.  I understand that there is some added
>    >> > complexity, but I don't feel like the scope of the addition is
>    >> > unrealistic.  I think the LXC integration was a lot farther 
>out of the
>    >> > scope of what ACS does then this is.  This does not change the 
>"things"
>    >> > which ACS orchestrates, it just adds the concept of a grouping 
>of things
>    >> > which ACS already manages.  I think this is the right approach 
>since it
>    >> is
>    >> > not trying to be a container orchestrator.  We will never 
>compete with
>    >> K8S,
>    >> > for example, and we should not try, but K8S is here and the 
>market wants
>    >> > it.  I do think we should be keeping our head up about that 
>fact because
>    >> > being able to provide a the underlay for K8S is very valuable 
>in the
>    >> > current marketplace.  I see this functionality as a way to 
>enable K8S
>    >> > adoption on top of ACS without changing our core values.
>    >> >
>    >> > - Regarding Erik's remarks.  The container space is moving 
>fast, but so
>    >> is
>    >> > the industry.  If we want to remain relevant, we need to be able to
>    >> adapt a
>    >> > bit.  I don't think this is a big shift in what we do, but it 
>is one that
>    >> > enables people to be able to start running with something like 
>K8S on top
>    >> > of their existing ACS.  This is something we are interested in 
>doing and
>    >> so
>    >> > are our customers.  If we can have a thin layer in ACS which 
>helps enable
>    >> > the use of K8S (or other container orchestrators) by orchestrating
>    >> > infrastructure, as we already do, and making it easier to adopt a
>    >> container
>    >> > orchestrator running on top of ACS, I think that gives us a 
>nice foothold
>    >> > in the market.  I don't really feel it is fair to compare 
>containers to
>    >> > IPv6.  IPv6 has been out forever and it has taken almost a 
>decade to get
>    >> > anyone to adopt it.  Containers have really only been here for 
>like 2
>    >> years
>    >> > and they are changing the market landscape in a very real way.
>    >> >
>    >> > Kind of on topic and kind of off topic.  I think understanding our
>    >> approach
>    >> > to containers is going to be important for the ACS community 
>as a whole.
>    >> > If we don't offer that market anything, then we will not be 
>considered
>    >> and
>    >> > we will lose market share we can't afford to lose.  If we try 
>to hitch
>    >> our
>    >> > horse to that cart too much, we will not be able to be agile 
>enough and
>    >> > will fail.  I feel like the right approach is for us to know 
>that it is a
>    >> > thriving market and continue to do what we do, but to extend 
>an olive
>    >> > branch to that market.  I think this sort of implementation is 
>the right
>    >> > approach because we are not trying to do too much.  We are 
>simply giving
>    >> a
>    >> > foundation on which the next big thing in the container 
>orchestration
>    >> world
>    >> > can adopt without us having to compete directly in that space.  
>I think
>    >> we
>    >> > have to focus on what we do best, but at the same time, think 
>about what
>    >> we
>    >> > can do to enable that huge market of users to adopt ACS as their
>    >> > foundation.  The ability to offer VMs and containers in the 
>same data
>    >> plane
>    >> > is something we have the ability to do, especially with this 
>approach,
>    >> and
>    >> > is something that most other softwares can not do.  The adoption of
>    >> > containers by bigger organizations will be only part of their 
>workload,
>    >> > they will still be running VMs for the foreseeable future. 
>Being able to
>    >> > appeal to that market is going to be important for us.
>    >> >
>    >> > Hopefully I don't have too many strong opinions here, but I do 
>think we
>    >> > need to be thinking about how we move forward in a world which is
>    >> adopting
>    >> > containers in a very real way.
>    >> >
>    >>
>    >> Understood. I just want to prevent that we add more features to 
>CloudStack
>    >> which are poorly maintained. Not judging Murali here at all, but 
>I'd rather
>    >> see CloudStack loose code then have it added.
>    >>
>    >> Thinking about LXC, would like to see that removed together with 
>various
>    >> other network plugins which I think are rarely used.
>    >>
>    >> Now, the idea of Murali was misunderstood by me. I think it 
>would be worth
>    >> more if we would improve our cloud-init support and integration 
>in various
>    >> tools which makes it much easier to deploy VMs running containers ON
>    >> CloudStack.
>    >>
>    >> Not so much changing CloudStack code, but rather tooling around it.
>    >>
>    >> If we have CloudStack talking to Kubernetes we suddenly have to 
>maintain
>    >> that API integration. Who's going to do that. Realistically.
>    >>
>    >> That's my main worry in this regard.
>    >>
>    >> We have so much more work to do in ACS in total that I don't 
>know if this
>    >> is the realistic route. I talk to many people who are not looking at
>    >> containers and are still working with VMs.
>    >>
>    >> There is not a single truth which is true, it really depends on 
>who you
>    >> ask.
>    >>
>    >> Wido
>    >>
>    >> > Cheers,
>    >> >
>    >> > Will
>    >> >
>    >> > *Will STEVENS*
>    >> > Lead Developer
>    >> >
>    >> > <https://goo.gl/NYZ8KK>
>    >> >
>    >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber 
><te...@gmail.com>
>wrote:
>    >> >
>    >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy < 
>muralimmreddy@gmail.com
>    >> >
>    >> > > wrote:
>    >> > > > All,
>    >> > > >
>    >> > > > I would like propose native functionality into CloudStack 
>to provide
>    >> a
>    >> > > container service through which users out-of-the box can use 
>to launch
>    >> > > container based application. Idea is to support ability to 
>orchestrate
>    >> the
>    >> > > resources and automate aspects of setting up container 
>orchestrator
>    >> through
>    >> > > CloudStack. Public IAAS service providers AWS with its ECS 
>[1] and
>    >> google
>    >> > > with GKE [2] already provides ability container applications.
>    >> Competitive
>    >> > > cloud orchestration platforms already have native support 
>for container
>    >> > > service. Users of CloudStack both as public cloud providers 
>and users
>    >> with
>    >> > > private clouds will benefit with such functionality.
>    >> > > >
>    >> > > > While container orchestrator of user choice can be 
>provisioned on
>    >> top of
>    >> > > CloudStack (with out CloudStack being involved) with tools like
>    >> > > TerraForm[3], Ansible[4] etc, advantage of having native 
>orchestration
>    >> is
>    >> > > giving user a nice cohesive integration. This proposal would 
>like add a
>    >> > > notion of first class CloudStack entity called container 
>cluster which
>    >> can
>    >> > > be used to provision resources, scale up, scale down, start 
>and stop
>    >> the
>    >> > > cluster of VM’s on which containerised applications can be run.
>For
>    >> actual
>    >> > > container orchestration we will still need container 
>orchestrator like
>    >> > > docker swarm, marathon, kubernetes, but CloudStack container 
>service
>    >> can
>    >> > > automate setting up of control place automatically.
>    >> > > >
>    >> > >
>    >> > > To be honest I'm torn on this one.
>    >> > >
>    >> > > Containers are a rapid changing thing, and while docker swam,
>    >> > > kubernetes, rancher or whatnot is popular today, they might 
>not be
>    >> > > tomorrow.
>    >> > > They might use CoreOS today, but might not tomorrow.
>    >> > >
>    >> > > We have a rather poor track record of staying up to date with new
>    >> > > features/versions, and adding a feature that is so rapidly 
>changing
>    >> > > is, I fear, going to be hard to maintain.
>    >> > > Want an example, look at xenserver. It is one of the most used
>    >> > > hypervisors we support, yet it took 6 months or so for us to 
>support
>    >> > > the latest release.
>    >> > > Or IPv6...
>    >> > >
>    >> > > I don't mean to bash at maintainers/implementers of those 
>features, I
>    >> > > appreciate all the work being done in every aspect, but I 
>believe we
>    >> > > should be realistic and realize that we have issues with 
>keeping stuff
>    >> > > up to date.
>    >> > >
>    >> > > I'd say focus on making sure other tools can do their job 
>well against
>    >> > > CloudStack (kops, rancher, ++), but that does not mean I 
>will -1 the
>    >> > > idea if anyone really wants to go through with it.
>    >> > >
>    >> > > --
>    >> > > Erik
>    >> > >
>    >>
>
>    murali.reddy@shapeblue.com
>    www.shapeblue.com
>    53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>    @shapeblue
>
>
>
>
>
>
>
>
>
>DISCLAIMER
>==========
>This e-mail may contain privileged and confidential information which 
>is the property of Accelerite, a Persistent Systems business. It is 
>intended only for the use of the individual or entity to which it is 
>addressed. If you are not the intended recipient, you are not 
>authorized to read, retain, copy, print, distribute or use this 
>message. If you have received this communication in error, please 
>notify the sender and delete all copies of this message. Accelerite, a 
>Persistent Systems business does not accept any liability for virus infected mails.




DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.

Re: [PROPOSAL] add native container orchestration service

[Murali Reddy <mu...@gmail.com>]

Thanks for the feedback. There have been some concerns here with this proposal. I think we had sufficient valid arguments why we need this functionality in CloudStack. While this proposal is to use k8s as container orchestrator design should be able to adopt other container orchestrator.


Unless anybody has any specific technical points on this, I'd like to go on with its implementation and open up a PR.


Kishan, when do you think you can share further details (northbound API semantics etc) on the work you are doing? I am planning to get this feature the for 4.11. 



On 31/01/17, 5:54 PM, "Will Stevens" <wi...@gmail.com> wrote:

>I think that is covered in this proposal. There is nothing k8s specific in
>this integration (from what I understand), all the k8s details are passed
>in via the cloud-init configuration after the cluster has been provisioned.
>
>On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com> wrote:
>
>
>There are many container orchestrators. Those container orchestrators are
>happy to run on any VMs or bare metal machines. K8s is just one of them and
>there will be more in the future. It may not be a good idea to make
>CloudStack to be k8s aware. IMO, the relationship between k8s and
>cloudstack should be similar to application and os. It probably not a good
>idea to make your OS to be aware of any specific applications so IMHO I
>don’t think k8s should be native to CloudStack. It makes more sense to
>provide some generic services that many applications can take advantages
>of. Some generic resource grouping service makes sense so a group of VMs,
>baremetal machines or network can be treated as a single entity. Some life
>cycle management will be necessary for these entities too. We can deploy
>k8s, swarm, dcos or even non-container specific services on top of
>CloudStack. The k8s is changing fast. One single tenant installation may
>need more than one VM group and may actually requires more (k8s
>federation). It will be a struggle to be in sync if we try to bring k8s
>specific knowledge into cloudstack.
>
>Regards,
>
>
>--
>Lianghwa Jou
>VP Engineering, Accelerite
>email: lianghwa.jou@accelerite.com
>
>
>
>
>
>On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
>
>
>    I agree with some good views Will has shared and I also agree to the
>concerns raised by Wido and Eric. IMO we need balance of staying
>relevant/add new features vs stability of CloudStack and take corrective
>action if needed. We have two good examples for both. When SDN was hot
>technology CloudStack ended up with bunch of SDN controller integrations.
>Few years later, now CloudStack is carrying baggage with no maintainers for
>those plugins, with probably not many of CloudStack users needing overlays.
>On the other hand, other than attempt to liaison with ETSI for NFV no
>effort was done. OpenStack has become de-facto for NFV. Now for OpenStack,
>arguably NFV has become larger use case than cloud it self. I concur with
>Will’s point that with minimal viable solution that does not change the
>core of CloudStack, and can keep CloudStack relevant is worth to be taken
>in.
>
>    Will,
>
>    To your question of how different is from ShapeBlue’s container
>service, its design, implementation and API semantics etc remain same.
>ShapeBlue’s container service was true drop in plug-in to CloudStack, with
>this proposal I am trying to re-work to make it a core CloudStack pluggable
>service which is part of CloudStack.
>
>    Key concepts that this proposal is trying to add
>
>        - add notion of ‘container cluster’ as a first class entity in
>CloudStack. Which is bacially collection of other CloudStack resources
>(like VM’s, networks, public ip, network rules etc)
>        - life cycle operation to manage ‘container cluster’ like create,
>delete, start, stop, scale-up, scale-down, heal etc
>        - orchestrate container orchestrator control plane on top of
>provisioned resources
>
>    At-least for k8s (which is what this proposal is targeting),
>integration with k8s is bare minimum. There are cloud-config scripts that
>automatically setup k8s cluster master and node VM’s. All CloudStack is
>doing in passing the cloud-config to the core OS VM’s as user data.
>
>    Regards
>    Murali Reddy
>
>
>
>
>
>
>
>    On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
>behalf of wstevens@cloudops.com> wrote:
>
>    >I agree that we need to be careful what we take on and own inside
>    >CloudStack.  I feel like some of the plugins or integrations which we
>have
>    >been "maintaining" may serve us better to abandon, but I feel like
>that is
>    >a whole discussion on its own.
>    >
>    >In this case, I feel like there is a minimum viable solution which puts
>    >CloudStack in a pretty good place to enable container orchestration.
>For
>    >example, one of the biggest challenges with K8S is the fact that it is
>    >single tenant.  CloudStack has good multi tenancy support and has the
>    >ability to orchestrate the underlying infra quite well.  We will have
>to be
>    >very careful not to try to own too deep into the K8S world though, in
>my
>    >opinion.  We only want to be responsible for providing the infra (and
>a way
>    >to bootstrap K8S ideally) and be able to scale the infra, everything
>else
>    >should be owned by the K8S on top.  That is the way I see it anyway,
>but
>    >please add your input.
>    >
>    >I think it is a liability to try to go too deep, for the same reasons
>Wido
>    >and Erik have mentioned.  But I also think we need to take it seriously
>    >because that train is moving and this may be a good opportunity to stay
>    >relevant in a rapidly changing market.
>    >
>    >*Will STEVENS*
>    >Lead Developer
>    >
>    ><https://goo.gl/NYZ8KK>
>    >
>    >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander <wi...@widodh.nl>
>wrote:
>    >
>    >>
>    >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
>wstevens@cloudops.com
>    >> >:
>    >> >
>    >> >
>    >> > Hey Murali,
>    >> > How different is this proposal than what ShapeBlue already built.
>It
>    >> looks
>    >> > pretty consistent with the functionality that you guys open
>sourced in
>    >> > Seville.
>    >> >
>    >> > I have not yet used this functionality, but I have reports that it
>works
>    >> > quite well.
>    >> >
>    >> > I believe the premise here is to only orchestrate the VM layer and
>    >> > basically expose a "group" of running VMs to the user.  The user is
>    >> > responsible for configuring K8S or whatever other container
>orchestrator
>    >> on
>    >> > top.  I saw mention of the "cloud-config" scripts in the FS, how
>are
>    >> those
>    >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
>    >> >
>    >> > I believe the core feature that is being requested to be added is
>the
>    >> > ability to create a group of VMs which will be kept active as a
>group if
>    >> at
>    >> > all possible.  ACS would be responsible for making sure that the
>number
>    >> of
>    >> > VMs specified for the group are in running state and it would spin
>up new
>    >> > VMs as needed in order to satisfy the group settings.  In general,
>it is
>    >> > understood that any application running on this group would have
>to be
>    >> > fault tolerant enough to be able to rediscover a new VM if one
>fails and
>    >> is
>    >> > replaced by a fresh copy.  Is that fair to say?  How is it
>expected that
>    >> > this service discovery is done, just by VMs being present on the
>network?
>    >> >
>    >> > As for some of the other people's concerns in this thread.
>    >> >
>    >> > - Regarding Wido's remarks.  I understand that there is some added
>    >> > complexity, but I don't feel like the scope of the addition is
>    >> > unrealistic.  I think the LXC integration was a lot farther out of
>the
>    >> > scope of what ACS does then this is.  This does not change the
>"things"
>    >> > which ACS orchestrates, it just adds the concept of a grouping of
>things
>    >> > which ACS already manages.  I think this is the right approach
>since it
>    >> is
>    >> > not trying to be a container orchestrator.  We will never compete
>with
>    >> K8S,
>    >> > for example, and we should not try, but K8S is here and the market
>wants
>    >> > it.  I do think we should be keeping our head up about that fact
>because
>    >> > being able to provide a the underlay for K8S is very valuable in
>the
>    >> > current marketplace.  I see this functionality as a way to enable
>K8S
>    >> > adoption on top of ACS without changing our core values.
>    >> >
>    >> > - Regarding Erik's remarks.  The container space is moving fast,
>but so
>    >> is
>    >> > the industry.  If we want to remain relevant, we need to be able to
>    >> adapt a
>    >> > bit.  I don't think this is a big shift in what we do, but it is
>one that
>    >> > enables people to be able to start running with something like K8S
>on top
>    >> > of their existing ACS.  This is something we are interested in
>doing and
>    >> so
>    >> > are our customers.  If we can have a thin layer in ACS which helps
>enable
>    >> > the use of K8S (or other container orchestrators) by orchestrating
>    >> > infrastructure, as we already do, and making it easier to adopt a
>    >> container
>    >> > orchestrator running on top of ACS, I think that gives us a nice
>foothold
>    >> > in the market.  I don't really feel it is fair to compare
>containers to
>    >> > IPv6.  IPv6 has been out forever and it has taken almost a decade
>to get
>    >> > anyone to adopt it.  Containers have really only been here for
>like 2
>    >> years
>    >> > and they are changing the market landscape in a very real way.
>    >> >
>    >> > Kind of on topic and kind of off topic.  I think understanding our
>    >> approach
>    >> > to containers is going to be important for the ACS community as a
>whole.
>    >> > If we don't offer that market anything, then we will not be
>considered
>    >> and
>    >> > we will lose market share we can't afford to lose.  If we try to
>hitch
>    >> our
>    >> > horse to that cart too much, we will not be able to be agile
>enough and
>    >> > will fail.  I feel like the right approach is for us to know that
>it is a
>    >> > thriving market and continue to do what we do, but to extend an
>olive
>    >> > branch to that market.  I think this sort of implementation is the
>right
>    >> > approach because we are not trying to do too much.  We are simply
>giving
>    >> a
>    >> > foundation on which the next big thing in the container
>orchestration
>    >> world
>    >> > can adopt without us having to compete directly in that space.  I
>think
>    >> we
>    >> > have to focus on what we do best, but at the same time, think
>about what
>    >> we
>    >> > can do to enable that huge market of users to adopt ACS as their
>    >> > foundation.  The ability to offer VMs and containers in the same
>data
>    >> plane
>    >> > is something we have the ability to do, especially with this
>approach,
>    >> and
>    >> > is something that most other softwares can not do.  The adoption of
>    >> > containers by bigger organizations will be only part of their
>workload,
>    >> > they will still be running VMs for the foreseeable future. Being
>able to
>    >> > appeal to that market is going to be important for us.
>    >> >
>    >> > Hopefully I don't have too many strong opinions here, but I do
>think we
>    >> > need to be thinking about how we move forward in a world which is
>    >> adopting
>    >> > containers in a very real way.
>    >> >
>    >>
>    >> Understood. I just want to prevent that we add more features to
>CloudStack
>    >> which are poorly maintained. Not judging Murali here at all, but I'd
>rather
>    >> see CloudStack loose code then have it added.
>    >>
>    >> Thinking about LXC, would like to see that removed together with
>various
>    >> other network plugins which I think are rarely used.
>    >>
>    >> Now, the idea of Murali was misunderstood by me. I think it would be
>worth
>    >> more if we would improve our cloud-init support and integration in
>various
>    >> tools which makes it much easier to deploy VMs running containers ON
>    >> CloudStack.
>    >>
>    >> Not so much changing CloudStack code, but rather tooling around it.
>    >>
>    >> If we have CloudStack talking to Kubernetes we suddenly have to
>maintain
>    >> that API integration. Who's going to do that. Realistically.
>    >>
>    >> That's my main worry in this regard.
>    >>
>    >> We have so much more work to do in ACS in total that I don't know if
>this
>    >> is the realistic route. I talk to many people who are not looking at
>    >> containers and are still working with VMs.
>    >>
>    >> There is not a single truth which is true, it really depends on who
>you
>    >> ask.
>    >>
>    >> Wido
>    >>
>    >> > Cheers,
>    >> >
>    >> > Will
>    >> >
>    >> > *Will STEVENS*
>    >> > Lead Developer
>    >> >
>    >> > <https://goo.gl/NYZ8KK>
>    >> >
>    >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com>
>wrote:
>    >> >
>    >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
>muralimmreddy@gmail.com
>    >> >
>    >> > > wrote:
>    >> > > > All,
>    >> > > >
>    >> > > > I would like propose native functionality into CloudStack to
>provide
>    >> a
>    >> > > container service through which users out-of-the box can use to
>launch
>    >> > > container based application. Idea is to support ability to
>orchestrate
>    >> the
>    >> > > resources and automate aspects of setting up container
>orchestrator
>    >> through
>    >> > > CloudStack. Public IAAS service providers AWS with its ECS [1]
>and
>    >> google
>    >> > > with GKE [2] already provides ability container applications.
>    >> Competitive
>    >> > > cloud orchestration platforms already have native support for
>container
>    >> > > service. Users of CloudStack both as public cloud providers and
>users
>    >> with
>    >> > > private clouds will benefit with such functionality.
>    >> > > >
>    >> > > > While container orchestrator of user choice can be provisioned
>on
>    >> top of
>    >> > > CloudStack (with out CloudStack being involved) with tools like
>    >> > > TerraForm[3], Ansible[4] etc, advantage of having native
>orchestration
>    >> is
>    >> > > giving user a nice cohesive integration. This proposal would
>like add a
>    >> > > notion of first class CloudStack entity called container cluster
>which
>    >> can
>    >> > > be used to provision resources, scale up, scale down, start and
>stop
>    >> the
>    >> > > cluster of VM’s on which containerised applications can be run.
>For
>    >> actual
>    >> > > container orchestration we will still need container
>orchestrator like
>    >> > > docker swarm, marathon, kubernetes, but CloudStack container
>service
>    >> can
>    >> > > automate setting up of control place automatically.
>    >> > > >
>    >> > >
>    >> > > To be honest I'm torn on this one.
>    >> > >
>    >> > > Containers are a rapid changing thing, and while docker swam,
>    >> > > kubernetes, rancher or whatnot is popular today, they might not
>be
>    >> > > tomorrow.
>    >> > > They might use CoreOS today, but might not tomorrow.
>    >> > >
>    >> > > We have a rather poor track record of staying up to date with new
>    >> > > features/versions, and adding a feature that is so rapidly
>changing
>    >> > > is, I fear, going to be hard to maintain.
>    >> > > Want an example, look at xenserver. It is one of the most used
>    >> > > hypervisors we support, yet it took 6 months or so for us to
>support
>    >> > > the latest release.
>    >> > > Or IPv6...
>    >> > >
>    >> > > I don't mean to bash at maintainers/implementers of those
>features, I
>    >> > > appreciate all the work being done in every aspect, but I
>believe we
>    >> > > should be realistic and realize that we have issues with keeping
>stuff
>    >> > > up to date.
>    >> > >
>    >> > > I'd say focus on making sure other tools can do their job well
>against
>    >> > > CloudStack (kops, rancher, ++), but that does not mean I will -1
>the
>    >> > > idea if anyone really wants to go through with it.
>    >> > >
>    >> > > --
>    >> > > Erik
>    >> > >
>    >>
>
>    murali.reddy@shapeblue.com
>    www.shapeblue.com
>    53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>    @shapeblue
>
>
>
>
>
>
>
>
>
>DISCLAIMER
>==========
>This e-mail may contain privileged and confidential information which is
>the property of Accelerite, a Persistent Systems business. It is intended
>only for the use of the individual or entity to which it is addressed. If
>you are not the intended recipient, you are not authorized to read, retain,
>copy, print, distribute or use this message. If you have received this
>communication in error, please notify the sender and delete all copies of
>this message. Accelerite, a Persistent Systems business does not accept any
>liability for virus infected mails.

Re: [PROPOSAL] add native container orchestration service

[Will Stevens <wi...@gmail.com>]

I think that is covered in this proposal. There is nothing k8s specific in
this integration (from what I understand), all the k8s details are passed
in via the cloud-init configuration after the cluster has been provisioned.

On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com> wrote:


There are many container orchestrators. Those container orchestrators are
happy to run on any VMs or bare metal machines. K8s is just one of them and
there will be more in the future. It may not be a good idea to make
CloudStack to be k8s aware. IMO, the relationship between k8s and
cloudstack should be similar to application and os. It probably not a good
idea to make your OS to be aware of any specific applications so IMHO I
don’t think k8s should be native to CloudStack. It makes more sense to
provide some generic services that many applications can take advantages
of. Some generic resource grouping service makes sense so a group of VMs,
baremetal machines or network can be treated as a single entity. Some life
cycle management will be necessary for these entities too. We can deploy
k8s, swarm, dcos or even non-container specific services on top of
CloudStack. The k8s is changing fast. One single tenant installation may
need more than one VM group and may actually requires more (k8s
federation). It will be a struggle to be in sync if we try to bring k8s
specific knowledge into cloudstack.

Regards,


--
Lianghwa Jou
VP Engineering, Accelerite
email: lianghwa.jou@accelerite.com





On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:


    I agree with some good views Will has shared and I also agree to the
concerns raised by Wido and Eric. IMO we need balance of staying
relevant/add new features vs stability of CloudStack and take corrective
action if needed. We have two good examples for both. When SDN was hot
technology CloudStack ended up with bunch of SDN controller integrations.
Few years later, now CloudStack is carrying baggage with no maintainers for
those plugins, with probably not many of CloudStack users needing overlays.
On the other hand, other than attempt to liaison with ETSI for NFV no
effort was done. OpenStack has become de-facto for NFV. Now for OpenStack,
arguably NFV has become larger use case than cloud it self. I concur with
Will’s point that with minimal viable solution that does not change the
core of CloudStack, and can keep CloudStack relevant is worth to be taken
in.

    Will,

    To your question of how different is from ShapeBlue’s container
service, its design, implementation and API semantics etc remain same.
ShapeBlue’s container service was true drop in plug-in to CloudStack, with
this proposal I am trying to re-work to make it a core CloudStack pluggable
service which is part of CloudStack.

    Key concepts that this proposal is trying to add

        - add notion of ‘container cluster’ as a first class entity in
CloudStack. Which is bacially collection of other CloudStack resources
(like VM’s, networks, public ip, network rules etc)
        - life cycle operation to manage ‘container cluster’ like create,
delete, start, stop, scale-up, scale-down, heal etc
        - orchestrate container orchestrator control plane on top of
provisioned resources

    At-least for k8s (which is what this proposal is targeting),
integration with k8s is bare minimum. There are cloud-config scripts that
automatically setup k8s cluster master and node VM’s. All CloudStack is
doing in passing the cloud-config to the core OS VM’s as user data.

    Regards
    Murali Reddy







    On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
behalf of wstevens@cloudops.com> wrote:

    >I agree that we need to be careful what we take on and own inside
    >CloudStack.  I feel like some of the plugins or integrations which we
have
    >been "maintaining" may serve us better to abandon, but I feel like
that is
    >a whole discussion on its own.
    >
    >In this case, I feel like there is a minimum viable solution which puts
    >CloudStack in a pretty good place to enable container orchestration.
For
    >example, one of the biggest challenges with K8S is the fact that it is
    >single tenant.  CloudStack has good multi tenancy support and has the
    >ability to orchestrate the underlying infra quite well.  We will have
to be
    >very careful not to try to own too deep into the K8S world though, in
my
    >opinion.  We only want to be responsible for providing the infra (and
a way
    >to bootstrap K8S ideally) and be able to scale the infra, everything
else
    >should be owned by the K8S on top.  That is the way I see it anyway,
but
    >please add your input.
    >
    >I think it is a liability to try to go too deep, for the same reasons
Wido
    >and Erik have mentioned.  But I also think we need to take it seriously
    >because that train is moving and this may be a good opportunity to stay
    >relevant in a rapidly changing market.
    >
    >*Will STEVENS*
    >Lead Developer
    >
    ><https://goo.gl/NYZ8KK>
    >
    >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander <wi...@widodh.nl>
wrote:
    >
    >>
    >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
wstevens@cloudops.com
    >> >:
    >> >
    >> >
    >> > Hey Murali,
    >> > How different is this proposal than what ShapeBlue already built.
It
    >> looks
    >> > pretty consistent with the functionality that you guys open
sourced in
    >> > Seville.
    >> >
    >> > I have not yet used this functionality, but I have reports that it
works
    >> > quite well.
    >> >
    >> > I believe the premise here is to only orchestrate the VM layer and
    >> > basically expose a "group" of running VMs to the user.  The user is
    >> > responsible for configuring K8S or whatever other container
orchestrator
    >> on
    >> > top.  I saw mention of the "cloud-config" scripts in the FS, how
are
    >> those
    >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
    >> >
    >> > I believe the core feature that is being requested to be added is
the
    >> > ability to create a group of VMs which will be kept active as a
group if
    >> at
    >> > all possible.  ACS would be responsible for making sure that the
number
    >> of
    >> > VMs specified for the group are in running state and it would spin
up new
    >> > VMs as needed in order to satisfy the group settings.  In general,
it is
    >> > understood that any application running on this group would have
to be
    >> > fault tolerant enough to be able to rediscover a new VM if one
fails and
    >> is
    >> > replaced by a fresh copy.  Is that fair to say?  How is it
expected that
    >> > this service discovery is done, just by VMs being present on the
network?
    >> >
    >> > As for some of the other people's concerns in this thread.
    >> >
    >> > - Regarding Wido's remarks.  I understand that there is some added
    >> > complexity, but I don't feel like the scope of the addition is
    >> > unrealistic.  I think the LXC integration was a lot farther out of
the
    >> > scope of what ACS does then this is.  This does not change the
"things"
    >> > which ACS orchestrates, it just adds the concept of a grouping of
things
    >> > which ACS already manages.  I think this is the right approach
since it
    >> is
    >> > not trying to be a container orchestrator.  We will never compete
with
    >> K8S,
    >> > for example, and we should not try, but K8S is here and the market
wants
    >> > it.  I do think we should be keeping our head up about that fact
because
    >> > being able to provide a the underlay for K8S is very valuable in
the
    >> > current marketplace.  I see this functionality as a way to enable
K8S
    >> > adoption on top of ACS without changing our core values.
    >> >
    >> > - Regarding Erik's remarks.  The container space is moving fast,
but so
    >> is
    >> > the industry.  If we want to remain relevant, we need to be able to
    >> adapt a
    >> > bit.  I don't think this is a big shift in what we do, but it is
one that
    >> > enables people to be able to start running with something like K8S
on top
    >> > of their existing ACS.  This is something we are interested in
doing and
    >> so
    >> > are our customers.  If we can have a thin layer in ACS which helps
enable
    >> > the use of K8S (or other container orchestrators) by orchestrating
    >> > infrastructure, as we already do, and making it easier to adopt a
    >> container
    >> > orchestrator running on top of ACS, I think that gives us a nice
foothold
    >> > in the market.  I don't really feel it is fair to compare
containers to
    >> > IPv6.  IPv6 has been out forever and it has taken almost a decade
to get
    >> > anyone to adopt it.  Containers have really only been here for
like 2
    >> years
    >> > and they are changing the market landscape in a very real way.
    >> >
    >> > Kind of on topic and kind of off topic.  I think understanding our
    >> approach
    >> > to containers is going to be important for the ACS community as a
whole.
    >> > If we don't offer that market anything, then we will not be
considered
    >> and
    >> > we will lose market share we can't afford to lose.  If we try to
hitch
    >> our
    >> > horse to that cart too much, we will not be able to be agile
enough and
    >> > will fail.  I feel like the right approach is for us to know that
it is a
    >> > thriving market and continue to do what we do, but to extend an
olive
    >> > branch to that market.  I think this sort of implementation is the
right
    >> > approach because we are not trying to do too much.  We are simply
giving
    >> a
    >> > foundation on which the next big thing in the container
orchestration
    >> world
    >> > can adopt without us having to compete directly in that space.  I
think
    >> we
    >> > have to focus on what we do best, but at the same time, think
about what
    >> we
    >> > can do to enable that huge market of users to adopt ACS as their
    >> > foundation.  The ability to offer VMs and containers in the same
data
    >> plane
    >> > is something we have the ability to do, especially with this
approach,
    >> and
    >> > is something that most other softwares can not do.  The adoption of
    >> > containers by bigger organizations will be only part of their
workload,
    >> > they will still be running VMs for the foreseeable future. Being
able to
    >> > appeal to that market is going to be important for us.
    >> >
    >> > Hopefully I don't have too many strong opinions here, but I do
think we
    >> > need to be thinking about how we move forward in a world which is
    >> adopting
    >> > containers in a very real way.
    >> >
    >>
    >> Understood. I just want to prevent that we add more features to
CloudStack
    >> which are poorly maintained. Not judging Murali here at all, but I'd
rather
    >> see CloudStack loose code then have it added.
    >>
    >> Thinking about LXC, would like to see that removed together with
various
    >> other network plugins which I think are rarely used.
    >>
    >> Now, the idea of Murali was misunderstood by me. I think it would be
worth
    >> more if we would improve our cloud-init support and integration in
various
    >> tools which makes it much easier to deploy VMs running containers ON
    >> CloudStack.
    >>
    >> Not so much changing CloudStack code, but rather tooling around it.
    >>
    >> If we have CloudStack talking to Kubernetes we suddenly have to
maintain
    >> that API integration. Who's going to do that. Realistically.
    >>
    >> That's my main worry in this regard.
    >>
    >> We have so much more work to do in ACS in total that I don't know if
this
    >> is the realistic route. I talk to many people who are not looking at
    >> containers and are still working with VMs.
    >>
    >> There is not a single truth which is true, it really depends on who
you
    >> ask.
    >>
    >> Wido
    >>
    >> > Cheers,
    >> >
    >> > Will
    >> >
    >> > *Will STEVENS*
    >> > Lead Developer
    >> >
    >> > <https://goo.gl/NYZ8KK>
    >> >
    >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com>
wrote:
    >> >
    >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
muralimmreddy@gmail.com
    >> >
    >> > > wrote:
    >> > > > All,
    >> > > >
    >> > > > I would like propose native functionality into CloudStack to
provide
    >> a
    >> > > container service through which users out-of-the box can use to
launch
    >> > > container based application. Idea is to support ability to
orchestrate
    >> the
    >> > > resources and automate aspects of setting up container
orchestrator
    >> through
    >> > > CloudStack. Public IAAS service providers AWS with its ECS [1]
and
    >> google
    >> > > with GKE [2] already provides ability container applications.
    >> Competitive
    >> > > cloud orchestration platforms already have native support for
container
    >> > > service. Users of CloudStack both as public cloud providers and
users
    >> with
    >> > > private clouds will benefit with such functionality.
    >> > > >
    >> > > > While container orchestrator of user choice can be provisioned
on
    >> top of
    >> > > CloudStack (with out CloudStack being involved) with tools like
    >> > > TerraForm[3], Ansible[4] etc, advantage of having native
orchestration
    >> is
    >> > > giving user a nice cohesive integration. This proposal would
like add a
    >> > > notion of first class CloudStack entity called container cluster
which
    >> can
    >> > > be used to provision resources, scale up, scale down, start and
stop
    >> the
    >> > > cluster of VM’s on which containerised applications can be run.
For
    >> actual
    >> > > container orchestration we will still need container
orchestrator like
    >> > > docker swarm, marathon, kubernetes, but CloudStack container
service
    >> can
    >> > > automate setting up of control place automatically.
    >> > > >
    >> > >
    >> > > To be honest I'm torn on this one.
    >> > >
    >> > > Containers are a rapid changing thing, and while docker swam,
    >> > > kubernetes, rancher or whatnot is popular today, they might not
be
    >> > > tomorrow.
    >> > > They might use CoreOS today, but might not tomorrow.
    >> > >
    >> > > We have a rather poor track record of staying up to date with new
    >> > > features/versions, and adding a feature that is so rapidly
changing
    >> > > is, I fear, going to be hard to maintain.
    >> > > Want an example, look at xenserver. It is one of the most used
    >> > > hypervisors we support, yet it took 6 months or so for us to
support
    >> > > the latest release.
    >> > > Or IPv6...
    >> > >
    >> > > I don't mean to bash at maintainers/implementers of those
features, I
    >> > > appreciate all the work being done in every aspect, but I
believe we
    >> > > should be realistic and realize that we have issues with keeping
stuff
    >> > > up to date.
    >> > >
    >> > > I'd say focus on making sure other tools can do their job well
against
    >> > > CloudStack (kops, rancher, ++), but that does not mean I will -1
the
    >> > > idea if anyone really wants to go through with it.
    >> > >
    >> > > --
    >> > > Erik
    >> > >
    >>

    murali.reddy@shapeblue.com
    www.shapeblue.com
    53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    @shapeblue









DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is
the property of Accelerite, a Persistent Systems business. It is intended
only for the use of the individual or entity to which it is addressed. If
you are not the intended recipient, you are not authorized to read, retain,
copy, print, distribute or use this message. If you have received this
communication in error, please notify the sender and delete all copies of
this message. Accelerite, a Persistent Systems business does not accept any
liability for virus infected mails.

Re: [PROPOSAL] add native container orchestration service

[Lianghwa Jou <li...@accelerite.com>]

There are many container orchestrators. Those container orchestrators are happy to run on any VMs or bare metal machines. K8s is just one of them and there will be more in the future. It may not be a good idea to make CloudStack to be k8s aware. IMO, the relationship between k8s and cloudstack should be similar to application and os. It probably not a good idea to make your OS to be aware of any specific applications so IMHO I don’t think k8s should be native to CloudStack. It makes more sense to provide some generic services that many applications can take advantages of. Some generic resource grouping service makes sense so a group of VMs, baremetal machines or network can be treated as a single entity. Some life cycle management will be necessary for these entities too. We can deploy k8s, swarm, dcos or even non-container specific services on top of CloudStack. The k8s is changing fast. One single tenant installation may need more than one VM group and may actually requires more (k8s federation). It will be a struggle to be in sync if we try to bring k8s specific knowledge into cloudstack.

Regards,


--
Lianghwa Jou
VP Engineering, Accelerite
email: lianghwa.jou@accelerite.com
 
 
 


On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:

    
    I agree with some good views Will has shared and I also agree to the concerns raised by Wido and Eric. IMO we need balance of staying relevant/add new features vs stability of CloudStack and take corrective action if needed. We have two good examples for both. When SDN was hot technology CloudStack ended up with bunch of SDN controller integrations. Few years later, now CloudStack is carrying baggage with no maintainers for those plugins, with probably not many of CloudStack users needing overlays. On the other hand, other than attempt to liaison with ETSI for NFV no effort was done. OpenStack has become de-facto for NFV. Now for OpenStack, arguably NFV has become larger use case than cloud it self. I concur with Will’s point that with minimal viable solution that does not change the core of CloudStack, and can keep CloudStack relevant is worth to be taken in. 
    
    Will,
    
    To your question of how different is from ShapeBlue’s container service, its design, implementation and API semantics etc remain same. ShapeBlue’s container service was true drop in plug-in to CloudStack, with this proposal I am trying to re-work to make it a core CloudStack pluggable service which is part of CloudStack. 
    
    Key concepts that this proposal is trying to add
    
        - add notion of ‘container cluster’ as a first class entity in CloudStack. Which is bacially collection of other CloudStack resources (like VM’s, networks, public ip, network rules etc)
        - life cycle operation to manage ‘container cluster’ like create, delete, start, stop, scale-up, scale-down, heal etc
        - orchestrate container orchestrator control plane on top of provisioned resources
    
    At-least for k8s (which is what this proposal is targeting), integration with k8s is bare minimum. There are cloud-config scripts that automatically setup k8s cluster master and node VM’s. All CloudStack is doing in passing the cloud-config to the core OS VM’s as user data.
    
    Regards
    Murali Reddy
    
    
    
    
    
    
    
    On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:
    
    >I agree that we need to be careful what we take on and own inside
    >CloudStack.  I feel like some of the plugins or integrations which we have
    >been "maintaining" may serve us better to abandon, but I feel like that is
    >a whole discussion on its own.
    >
    >In this case, I feel like there is a minimum viable solution which puts
    >CloudStack in a pretty good place to enable container orchestration.  For
    >example, one of the biggest challenges with K8S is the fact that it is
    >single tenant.  CloudStack has good multi tenancy support and has the
    >ability to orchestrate the underlying infra quite well.  We will have to be
    >very careful not to try to own too deep into the K8S world though, in my
    >opinion.  We only want to be responsible for providing the infra (and a way
    >to bootstrap K8S ideally) and be able to scale the infra, everything else
    >should be owned by the K8S on top.  That is the way I see it anyway, but
    >please add your input.
    >
    >I think it is a liability to try to go too deep, for the same reasons Wido
    >and Erik have mentioned.  But I also think we need to take it seriously
    >because that train is moving and this may be a good opportunity to stay
    >relevant in a rapidly changing market.
    >
    >*Will STEVENS*
    >Lead Developer
    >
    ><https://goo.gl/NYZ8KK>
    >
    >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander <wi...@widodh.nl> wrote:
    >
    >>
    >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <wstevens@cloudops.com
    >> >:
    >> >
    >> >
    >> > Hey Murali,
    >> > How different is this proposal than what ShapeBlue already built.  It
    >> looks
    >> > pretty consistent with the functionality that you guys open sourced in
    >> > Seville.
    >> >
    >> > I have not yet used this functionality, but I have reports that it works
    >> > quite well.
    >> >
    >> > I believe the premise here is to only orchestrate the VM layer and
    >> > basically expose a "group" of running VMs to the user.  The user is
    >> > responsible for configuring K8S or whatever other container orchestrator
    >> on
    >> > top.  I saw mention of the "cloud-config" scripts in the FS, how are
    >> those
    >> > exposed to the cluster?  Maybe the FS can expand on that a bit?
    >> >
    >> > I believe the core feature that is being requested to be added is the
    >> > ability to create a group of VMs which will be kept active as a group if
    >> at
    >> > all possible.  ACS would be responsible for making sure that the number
    >> of
    >> > VMs specified for the group are in running state and it would spin up new
    >> > VMs as needed in order to satisfy the group settings.  In general, it is
    >> > understood that any application running on this group would have to be
    >> > fault tolerant enough to be able to rediscover a new VM if one fails and
    >> is
    >> > replaced by a fresh copy.  Is that fair to say?  How is it expected that
    >> > this service discovery is done, just by VMs being present on the network?
    >> >
    >> > As for some of the other people's concerns in this thread.
    >> >
    >> > - Regarding Wido's remarks.  I understand that there is some added
    >> > complexity, but I don't feel like the scope of the addition is
    >> > unrealistic.  I think the LXC integration was a lot farther out of the
    >> > scope of what ACS does then this is.  This does not change the "things"
    >> > which ACS orchestrates, it just adds the concept of a grouping of things
    >> > which ACS already manages.  I think this is the right approach since it
    >> is
    >> > not trying to be a container orchestrator.  We will never compete with
    >> K8S,
    >> > for example, and we should not try, but K8S is here and the market wants
    >> > it.  I do think we should be keeping our head up about that fact because
    >> > being able to provide a the underlay for K8S is very valuable in the
    >> > current marketplace.  I see this functionality as a way to enable K8S
    >> > adoption on top of ACS without changing our core values.
    >> >
    >> > - Regarding Erik's remarks.  The container space is moving fast, but so
    >> is
    >> > the industry.  If we want to remain relevant, we need to be able to
    >> adapt a
    >> > bit.  I don't think this is a big shift in what we do, but it is one that
    >> > enables people to be able to start running with something like K8S on top
    >> > of their existing ACS.  This is something we are interested in doing and
    >> so
    >> > are our customers.  If we can have a thin layer in ACS which helps enable
    >> > the use of K8S (or other container orchestrators) by orchestrating
    >> > infrastructure, as we already do, and making it easier to adopt a
    >> container
    >> > orchestrator running on top of ACS, I think that gives us a nice foothold
    >> > in the market.  I don't really feel it is fair to compare containers to
    >> > IPv6.  IPv6 has been out forever and it has taken almost a decade to get
    >> > anyone to adopt it.  Containers have really only been here for like 2
    >> years
    >> > and they are changing the market landscape in a very real way.
    >> >
    >> > Kind of on topic and kind of off topic.  I think understanding our
    >> approach
    >> > to containers is going to be important for the ACS community as a whole.
    >> > If we don't offer that market anything, then we will not be considered
    >> and
    >> > we will lose market share we can't afford to lose.  If we try to hitch
    >> our
    >> > horse to that cart too much, we will not be able to be agile enough and
    >> > will fail.  I feel like the right approach is for us to know that it is a
    >> > thriving market and continue to do what we do, but to extend an olive
    >> > branch to that market.  I think this sort of implementation is the right
    >> > approach because we are not trying to do too much.  We are simply giving
    >> a
    >> > foundation on which the next big thing in the container orchestration
    >> world
    >> > can adopt without us having to compete directly in that space.  I think
    >> we
    >> > have to focus on what we do best, but at the same time, think about what
    >> we
    >> > can do to enable that huge market of users to adopt ACS as their
    >> > foundation.  The ability to offer VMs and containers in the same data
    >> plane
    >> > is something we have the ability to do, especially with this approach,
    >> and
    >> > is something that most other softwares can not do.  The adoption of
    >> > containers by bigger organizations will be only part of their workload,
    >> > they will still be running VMs for the foreseeable future. Being able to
    >> > appeal to that market is going to be important for us.
    >> >
    >> > Hopefully I don't have too many strong opinions here, but I do think we
    >> > need to be thinking about how we move forward in a world which is
    >> adopting
    >> > containers in a very real way.
    >> >
    >>
    >> Understood. I just want to prevent that we add more features to CloudStack
    >> which are poorly maintained. Not judging Murali here at all, but I'd rather
    >> see CloudStack loose code then have it added.
    >>
    >> Thinking about LXC, would like to see that removed together with various
    >> other network plugins which I think are rarely used.
    >>
    >> Now, the idea of Murali was misunderstood by me. I think it would be worth
    >> more if we would improve our cloud-init support and integration in various
    >> tools which makes it much easier to deploy VMs running containers ON
    >> CloudStack.
    >>
    >> Not so much changing CloudStack code, but rather tooling around it.
    >>
    >> If we have CloudStack talking to Kubernetes we suddenly have to maintain
    >> that API integration. Who's going to do that. Realistically.
    >>
    >> That's my main worry in this regard.
    >>
    >> We have so much more work to do in ACS in total that I don't know if this
    >> is the realistic route. I talk to many people who are not looking at
    >> containers and are still working with VMs.
    >>
    >> There is not a single truth which is true, it really depends on who you
    >> ask.
    >>
    >> Wido
    >>
    >> > Cheers,
    >> >
    >> > Will
    >> >
    >> > *Will STEVENS*
    >> > Lead Developer
    >> >
    >> > <https://goo.gl/NYZ8KK>
    >> >
    >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com> wrote:
    >> >
    >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <muralimmreddy@gmail.com
    >> >
    >> > > wrote:
    >> > > > All,
    >> > > >
    >> > > > I would like propose native functionality into CloudStack to provide
    >> a
    >> > > container service through which users out-of-the box can use to launch
    >> > > container based application. Idea is to support ability to orchestrate
    >> the
    >> > > resources and automate aspects of setting up container orchestrator
    >> through
    >> > > CloudStack. Public IAAS service providers AWS with its ECS [1] and
    >> google
    >> > > with GKE [2] already provides ability container applications.
    >> Competitive
    >> > > cloud orchestration platforms already have native support for container
    >> > > service. Users of CloudStack both as public cloud providers and users
    >> with
    >> > > private clouds will benefit with such functionality.
    >> > > >
    >> > > > While container orchestrator of user choice can be provisioned on
    >> top of
    >> > > CloudStack (with out CloudStack being involved) with tools like
    >> > > TerraForm[3], Ansible[4] etc, advantage of having native orchestration
    >> is
    >> > > giving user a nice cohesive integration. This proposal would like add a
    >> > > notion of first class CloudStack entity called container cluster which
    >> can
    >> > > be used to provision resources, scale up, scale down, start and stop
    >> the
    >> > > cluster of VM’s on which containerised applications can be run. For
    >> actual
    >> > > container orchestration we will still need container orchestrator like
    >> > > docker swarm, marathon, kubernetes, but CloudStack container service
    >> can
    >> > > automate setting up of control place automatically.
    >> > > >
    >> > >
    >> > > To be honest I'm torn on this one.
    >> > >
    >> > > Containers are a rapid changing thing, and while docker swam,
    >> > > kubernetes, rancher or whatnot is popular today, they might not be
    >> > > tomorrow.
    >> > > They might use CoreOS today, but might not tomorrow.
    >> > >
    >> > > We have a rather poor track record of staying up to date with new
    >> > > features/versions, and adding a feature that is so rapidly changing
    >> > > is, I fear, going to be hard to maintain.
    >> > > Want an example, look at xenserver. It is one of the most used
    >> > > hypervisors we support, yet it took 6 months or so for us to support
    >> > > the latest release.
    >> > > Or IPv6...
    >> > >
    >> > > I don't mean to bash at maintainers/implementers of those features, I
    >> > > appreciate all the work being done in every aspect, but I believe we
    >> > > should be realistic and realize that we have issues with keeping stuff
    >> > > up to date.
    >> > >
    >> > > I'd say focus on making sure other tools can do their job well against
    >> > > CloudStack (kops, rancher, ++), but that does not mean I will -1 the
    >> > > idea if anyone really wants to go through with it.
    >> > >
    >> > > --
    >> > > Erik
    >> > >
    >>
    
    murali.reddy@shapeblue.com 
    www.shapeblue.com
    53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    @shapeblue
      
     
    
    





DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.

RE: [PROPOSAL] add native container orchestration service

[Kishan Kavala <ki...@accelerite.com>]

Taking this abstraction one more level up, I propose we keep the cluster generic, and not just for container service. More so, because the proposal is to make it core cloudstack pluggable service.

With the proposed container orchestration service,
 - A notion of cluster is introduced, which is basically group of Vms, network rules etc
 
 Container service is not the only use-case which will benefit from this cluster. Any service which requires a bunch of Vms grouped can make use of this cluster.

 Any service which require the following can be supported by a generic framework, similar to network plugin framework.
 - Create group of Vms (ServiceCluster) and manage their life cycle 
 - Configure network to make the ServiceCluster functional (e.g. open required ports on all vms to for services to communicate)
 - Create network rules to make the control pane accessible (e.g. create PortForwarding rule to access k8s/PaaS dashboard)
 - Driven by cloud-config via userdata (e.g. config to download required binaries and start services on the Vms)
 - Support a purpose-built template (e.g. CoreOS)
 
This framework can be used by services like Container service, PaaS or DB-as-a-service .  I’ve been doing some prototypes on a similar kind of abstraction.

-----Original Message-----
From: Murali Reddy [mailto:murali.reddy@shapeblue.com] 
Sent: Monday, January 30, 2017 1:24 PM
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service


I agree with some good views Will has shared and I also agree to the concerns raised by Wido and Eric. IMO we need balance of staying relevant/add new features vs stability of CloudStack and take corrective action if needed. We have two good examples for both. When SDN was hot technology CloudStack ended up with bunch of SDN controller integrations. Few years later, now CloudStack is carrying baggage with no maintainers for those plugins, with probably not many of CloudStack users needing overlays. On the other hand, other than attempt to liaison with ETSI for NFV no effort was done. OpenStack has become de-facto for NFV. Now for OpenStack, arguably NFV has become larger use case than cloud it self. I concur with Will’s point that with minimal viable solution that does not change the core of CloudStack, and can keep CloudStack relevant is worth to be taken in. 

Will,

To your question of how different is from ShapeBlue’s container service, its design, implementation and API semantics etc remain same. ShapeBlue’s container service was true drop in plug-in to CloudStack, with this proposal I am trying to re-work to make it a core CloudStack pluggable service which is part of CloudStack. 

Key concepts that this proposal is trying to add

    - add notion of ‘container cluster’ as a first class entity in CloudStack. Which is bacially collection of other CloudStack resources (like VM’s, networks, public ip, network rules etc)
    - life cycle operation to manage ‘container cluster’ like create, delete, start, stop, scale-up, scale-down, heal etc
    - orchestrate container orchestrator control plane on top of provisioned resources

At-least for k8s (which is what this proposal is targeting), integration with k8s is bare minimum. There are cloud-config scripts that automatically setup k8s cluster master and node VM’s. All CloudStack is doing in passing the cloud-config to the core OS VM’s as user data.

Regards
Murali Reddy







On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:

>I agree that we need to be careful what we take on and own inside 
>CloudStack.  I feel like some of the plugins or integrations which we 
>have been "maintaining" may serve us better to abandon, but I feel like 
>that is a whole discussion on its own.
>
>In this case, I feel like there is a minimum viable solution which puts 
>CloudStack in a pretty good place to enable container orchestration.  
>For example, one of the biggest challenges with K8S is the fact that it 
>is single tenant.  CloudStack has good multi tenancy support and has 
>the ability to orchestrate the underlying infra quite well.  We will 
>have to be very careful not to try to own too deep into the K8S world 
>though, in my opinion.  We only want to be responsible for providing 
>the infra (and a way to bootstrap K8S ideally) and be able to scale the 
>infra, everything else should be owned by the K8S on top.  That is the 
>way I see it anyway, but please add your input.
>
>I think it is a liability to try to go too deep, for the same reasons 
>Wido and Erik have mentioned.  But I also think we need to take it 
>seriously because that train is moving and this may be a good 
>opportunity to stay relevant in a rapidly changing market.
>
>*Will STEVENS*
>Lead Developer
>
><https://goo.gl/NYZ8KK>
>
>On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>
>>
>> > Op 27 januari 2017 om 16:08 schreef Will Stevens 
>> ><wstevens@cloudops.com
>> >:
>> >
>> >
>> > Hey Murali,
>> > How different is this proposal than what ShapeBlue already built.  
>> > It
>> looks
>> > pretty consistent with the functionality that you guys open sourced 
>> > in Seville.
>> >
>> > I have not yet used this functionality, but I have reports that it 
>> > works quite well.
>> >
>> > I believe the premise here is to only orchestrate the VM layer and 
>> > basically expose a "group" of running VMs to the user.  The user is 
>> > responsible for configuring K8S or whatever other container 
>> > orchestrator
>> on
>> > top.  I saw mention of the "cloud-config" scripts in the FS, how 
>> > are
>> those
>> > exposed to the cluster?  Maybe the FS can expand on that a bit?
>> >
>> > I believe the core feature that is being requested to be added is 
>> > the ability to create a group of VMs which will be kept active as a 
>> > group if
>> at
>> > all possible.  ACS would be responsible for making sure that the 
>> > number
>> of
>> > VMs specified for the group are in running state and it would spin 
>> > up new VMs as needed in order to satisfy the group settings.  In 
>> > general, it is understood that any application running on this 
>> > group would have to be fault tolerant enough to be able to 
>> > rediscover a new VM if one fails and
>> is
>> > replaced by a fresh copy.  Is that fair to say?  How is it expected 
>> > that this service discovery is done, just by VMs being present on the network?
>> >
>> > As for some of the other people's concerns in this thread.
>> >
>> > - Regarding Wido's remarks.  I understand that there is some added 
>> > complexity, but I don't feel like the scope of the addition is 
>> > unrealistic.  I think the LXC integration was a lot farther out of 
>> > the scope of what ACS does then this is.  This does not change the "things"
>> > which ACS orchestrates, it just adds the concept of a grouping of 
>> > things which ACS already manages.  I think this is the right 
>> > approach since it
>> is
>> > not trying to be a container orchestrator.  We will never compete 
>> > with
>> K8S,
>> > for example, and we should not try, but K8S is here and the market 
>> > wants it.  I do think we should be keeping our head up about that 
>> > fact because being able to provide a the underlay for K8S is very 
>> > valuable in the current marketplace.  I see this functionality as a 
>> > way to enable K8S adoption on top of ACS without changing our core values.
>> >
>> > - Regarding Erik's remarks.  The container space is moving fast, 
>> > but so
>> is
>> > the industry.  If we want to remain relevant, we need to be able to
>> adapt a
>> > bit.  I don't think this is a big shift in what we do, but it is 
>> > one that enables people to be able to start running with something 
>> > like K8S on top of their existing ACS.  This is something we are 
>> > interested in doing and
>> so
>> > are our customers.  If we can have a thin layer in ACS which helps 
>> > enable the use of K8S (or other container orchestrators) by 
>> > orchestrating infrastructure, as we already do, and making it 
>> > easier to adopt a
>> container
>> > orchestrator running on top of ACS, I think that gives us a nice 
>> > foothold in the market.  I don't really feel it is fair to compare 
>> > containers to IPv6.  IPv6 has been out forever and it has taken 
>> > almost a decade to get anyone to adopt it.  Containers have really 
>> > only been here for like 2
>> years
>> > and they are changing the market landscape in a very real way.
>> >
>> > Kind of on topic and kind of off topic.  I think understanding our
>> approach
>> > to containers is going to be important for the ACS community as a whole.
>> > If we don't offer that market anything, then we will not be 
>> > considered
>> and
>> > we will lose market share we can't afford to lose.  If we try to 
>> > hitch
>> our
>> > horse to that cart too much, we will not be able to be agile enough 
>> > and will fail.  I feel like the right approach is for us to know 
>> > that it is a thriving market and continue to do what we do, but to 
>> > extend an olive branch to that market.  I think this sort of 
>> > implementation is the right approach because we are not trying to 
>> > do too much.  We are simply giving
>> a
>> > foundation on which the next big thing in the container 
>> > orchestration
>> world
>> > can adopt without us having to compete directly in that space.  I 
>> > think
>> we
>> > have to focus on what we do best, but at the same time, think about 
>> > what
>> we
>> > can do to enable that huge market of users to adopt ACS as their 
>> > foundation.  The ability to offer VMs and containers in the same 
>> > data
>> plane
>> > is something we have the ability to do, especially with this 
>> > approach,
>> and
>> > is something that most other softwares can not do.  The adoption of 
>> > containers by bigger organizations will be only part of their 
>> > workload, they will still be running VMs for the foreseeable 
>> > future. Being able to appeal to that market is going to be important for us.
>> >
>> > Hopefully I don't have too many strong opinions here, but I do 
>> > think we need to be thinking about how we move forward in a world 
>> > which is
>> adopting
>> > containers in a very real way.
>> >
>>
>> Understood. I just want to prevent that we add more features to 
>> CloudStack which are poorly maintained. Not judging Murali here at 
>> all, but I'd rather see CloudStack loose code then have it added.
>>
>> Thinking about LXC, would like to see that removed together with 
>> various other network plugins which I think are rarely used.
>>
>> Now, the idea of Murali was misunderstood by me. I think it would be 
>> worth more if we would improve our cloud-init support and integration 
>> in various tools which makes it much easier to deploy VMs running 
>> containers ON CloudStack.
>>
>> Not so much changing CloudStack code, but rather tooling around it.
>>
>> If we have CloudStack talking to Kubernetes we suddenly have to 
>> maintain that API integration. Who's going to do that. Realistically.
>>
>> That's my main worry in this regard.
>>
>> We have so much more work to do in ACS in total that I don't know if 
>> this is the realistic route. I talk to many people who are not 
>> looking at containers and are still working with VMs.
>>
>> There is not a single truth which is true, it really depends on who 
>> you ask.
>>
>> Wido
>>
>> > Cheers,
>> >
>> > Will
>> >
>> > *Will STEVENS*
>> > Lead Developer
>> >
>> > <https://goo.gl/NYZ8KK>
>> >
>> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com> wrote:
>> >
>> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy 
>> > > <muralimmreddy@gmail.com
>> >
>> > > wrote:
>> > > > All,
>> > > >
>> > > > I would like propose native functionality into CloudStack to 
>> > > > provide
>> a
>> > > container service through which users out-of-the box can use to 
>> > > launch container based application. Idea is to support ability to 
>> > > orchestrate
>> the
>> > > resources and automate aspects of setting up container 
>> > > orchestrator
>> through
>> > > CloudStack. Public IAAS service providers AWS with its ECS [1] 
>> > > and
>> google
>> > > with GKE [2] already provides ability container applications.
>> Competitive
>> > > cloud orchestration platforms already have native support for 
>> > > container service. Users of CloudStack both as public cloud 
>> > > providers and users
>> with
>> > > private clouds will benefit with such functionality.
>> > > >
>> > > > While container orchestrator of user choice can be provisioned 
>> > > > on
>> top of
>> > > CloudStack (with out CloudStack being involved) with tools like 
>> > > TerraForm[3], Ansible[4] etc, advantage of having native 
>> > > orchestration
>> is
>> > > giving user a nice cohesive integration. This proposal would like 
>> > > add a notion of first class CloudStack entity called container 
>> > > cluster which
>> can
>> > > be used to provision resources, scale up, scale down, start and 
>> > > stop
>> the
>> > > cluster of VM’s on which containerised applications can be run. 
>> > > For
>> actual
>> > > container orchestration we will still need container orchestrator 
>> > > like docker swarm, marathon, kubernetes, but CloudStack container 
>> > > service
>> can
>> > > automate setting up of control place automatically.
>> > > >
>> > >
>> > > To be honest I'm torn on this one.
>> > >
>> > > Containers are a rapid changing thing, and while docker swam, 
>> > > kubernetes, rancher or whatnot is popular today, they might not 
>> > > be tomorrow.
>> > > They might use CoreOS today, but might not tomorrow.
>> > >
>> > > We have a rather poor track record of staying up to date with new 
>> > > features/versions, and adding a feature that is so rapidly 
>> > > changing is, I fear, going to be hard to maintain.
>> > > Want an example, look at xenserver. It is one of the most used 
>> > > hypervisors we support, yet it took 6 months or so for us to 
>> > > support the latest release.
>> > > Or IPv6...
>> > >
>> > > I don't mean to bash at maintainers/implementers of those 
>> > > features, I appreciate all the work being done in every aspect, 
>> > > but I believe we should be realistic and realize that we have 
>> > > issues with keeping stuff up to date.
>> > >
>> > > I'd say focus on making sure other tools can do their job well 
>> > > against CloudStack (kops, rancher, ++), but that does not mean I 
>> > > will -1 the idea if anyone really wants to go through with it.
>> > >
>> > > --
>> > > Erik
>> > >
>>

murali.reddy@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
  
 




DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.

Re: [PROPOSAL] add native container orchestration service

[Murali Reddy <mu...@shapeblue.com>]

I agree with some good views Will has shared and I also agree to the concerns raised by Wido and Eric. IMO we need balance of staying relevant/add new features vs stability of CloudStack and take corrective action if needed. We have two good examples for both. When SDN was hot technology CloudStack ended up with bunch of SDN controller integrations. Few years later, now CloudStack is carrying baggage with no maintainers for those plugins, with probably not many of CloudStack users needing overlays. On the other hand, other than attempt to liaison with ETSI for NFV no effort was done. OpenStack has become de-facto for NFV. Now for OpenStack, arguably NFV has become larger use case than cloud it self. I concur with Will’s point that with minimal viable solution that does not change the core of CloudStack, and can keep CloudStack relevant is worth to be taken in. 

Will,

To your question of how different is from ShapeBlue’s container service, its design, implementation and API semantics etc remain same. ShapeBlue’s container service was true drop in plug-in to CloudStack, with this proposal I am trying to re-work to make it a core CloudStack pluggable service which is part of CloudStack. 

Key concepts that this proposal is trying to add

    - add notion of ‘container cluster’ as a first class entity in CloudStack. Which is bacially collection of other CloudStack resources (like VM’s, networks, public ip, network rules etc)
    - life cycle operation to manage ‘container cluster’ like create, delete, start, stop, scale-up, scale-down, heal etc
    - orchestrate container orchestrator control plane on top of provisioned resources

At-least for k8s (which is what this proposal is targeting), integration with k8s is bare minimum. There are cloud-config scripts that automatically setup k8s cluster master and node VM’s. All CloudStack is doing in passing the cloud-config to the core OS VM’s as user data.

Regards
Murali Reddy







On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on behalf of wstevens@cloudops.com> wrote:

>I agree that we need to be careful what we take on and own inside
>CloudStack.  I feel like some of the plugins or integrations which we have
>been "maintaining" may serve us better to abandon, but I feel like that is
>a whole discussion on its own.
>
>In this case, I feel like there is a minimum viable solution which puts
>CloudStack in a pretty good place to enable container orchestration.  For
>example, one of the biggest challenges with K8S is the fact that it is
>single tenant.  CloudStack has good multi tenancy support and has the
>ability to orchestrate the underlying infra quite well.  We will have to be
>very careful not to try to own too deep into the K8S world though, in my
>opinion.  We only want to be responsible for providing the infra (and a way
>to bootstrap K8S ideally) and be able to scale the infra, everything else
>should be owned by the K8S on top.  That is the way I see it anyway, but
>please add your input.
>
>I think it is a liability to try to go too deep, for the same reasons Wido
>and Erik have mentioned.  But I also think we need to take it seriously
>because that train is moving and this may be a good opportunity to stay
>relevant in a rapidly changing market.
>
>*Will STEVENS*
>Lead Developer
>
><https://goo.gl/NYZ8KK>
>
>On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>
>>
>> > Op 27 januari 2017 om 16:08 schreef Will Stevens <wstevens@cloudops.com
>> >:
>> >
>> >
>> > Hey Murali,
>> > How different is this proposal than what ShapeBlue already built.  It
>> looks
>> > pretty consistent with the functionality that you guys open sourced in
>> > Seville.
>> >
>> > I have not yet used this functionality, but I have reports that it works
>> > quite well.
>> >
>> > I believe the premise here is to only orchestrate the VM layer and
>> > basically expose a "group" of running VMs to the user.  The user is
>> > responsible for configuring K8S or whatever other container orchestrator
>> on
>> > top.  I saw mention of the "cloud-config" scripts in the FS, how are
>> those
>> > exposed to the cluster?  Maybe the FS can expand on that a bit?
>> >
>> > I believe the core feature that is being requested to be added is the
>> > ability to create a group of VMs which will be kept active as a group if
>> at
>> > all possible.  ACS would be responsible for making sure that the number
>> of
>> > VMs specified for the group are in running state and it would spin up new
>> > VMs as needed in order to satisfy the group settings.  In general, it is
>> > understood that any application running on this group would have to be
>> > fault tolerant enough to be able to rediscover a new VM if one fails and
>> is
>> > replaced by a fresh copy.  Is that fair to say?  How is it expected that
>> > this service discovery is done, just by VMs being present on the network?
>> >
>> > As for some of the other people's concerns in this thread.
>> >
>> > - Regarding Wido's remarks.  I understand that there is some added
>> > complexity, but I don't feel like the scope of the addition is
>> > unrealistic.  I think the LXC integration was a lot farther out of the
>> > scope of what ACS does then this is.  This does not change the "things"
>> > which ACS orchestrates, it just adds the concept of a grouping of things
>> > which ACS already manages.  I think this is the right approach since it
>> is
>> > not trying to be a container orchestrator.  We will never compete with
>> K8S,
>> > for example, and we should not try, but K8S is here and the market wants
>> > it.  I do think we should be keeping our head up about that fact because
>> > being able to provide a the underlay for K8S is very valuable in the
>> > current marketplace.  I see this functionality as a way to enable K8S
>> > adoption on top of ACS without changing our core values.
>> >
>> > - Regarding Erik's remarks.  The container space is moving fast, but so
>> is
>> > the industry.  If we want to remain relevant, we need to be able to
>> adapt a
>> > bit.  I don't think this is a big shift in what we do, but it is one that
>> > enables people to be able to start running with something like K8S on top
>> > of their existing ACS.  This is something we are interested in doing and
>> so
>> > are our customers.  If we can have a thin layer in ACS which helps enable
>> > the use of K8S (or other container orchestrators) by orchestrating
>> > infrastructure, as we already do, and making it easier to adopt a
>> container
>> > orchestrator running on top of ACS, I think that gives us a nice foothold
>> > in the market.  I don't really feel it is fair to compare containers to
>> > IPv6.  IPv6 has been out forever and it has taken almost a decade to get
>> > anyone to adopt it.  Containers have really only been here for like 2
>> years
>> > and they are changing the market landscape in a very real way.
>> >
>> > Kind of on topic and kind of off topic.  I think understanding our
>> approach
>> > to containers is going to be important for the ACS community as a whole.
>> > If we don't offer that market anything, then we will not be considered
>> and
>> > we will lose market share we can't afford to lose.  If we try to hitch
>> our
>> > horse to that cart too much, we will not be able to be agile enough and
>> > will fail.  I feel like the right approach is for us to know that it is a
>> > thriving market and continue to do what we do, but to extend an olive
>> > branch to that market.  I think this sort of implementation is the right
>> > approach because we are not trying to do too much.  We are simply giving
>> a
>> > foundation on which the next big thing in the container orchestration
>> world
>> > can adopt without us having to compete directly in that space.  I think
>> we
>> > have to focus on what we do best, but at the same time, think about what
>> we
>> > can do to enable that huge market of users to adopt ACS as their
>> > foundation.  The ability to offer VMs and containers in the same data
>> plane
>> > is something we have the ability to do, especially with this approach,
>> and
>> > is something that most other softwares can not do.  The adoption of
>> > containers by bigger organizations will be only part of their workload,
>> > they will still be running VMs for the foreseeable future. Being able to
>> > appeal to that market is going to be important for us.
>> >
>> > Hopefully I don't have too many strong opinions here, but I do think we
>> > need to be thinking about how we move forward in a world which is
>> adopting
>> > containers in a very real way.
>> >
>>
>> Understood. I just want to prevent that we add more features to CloudStack
>> which are poorly maintained. Not judging Murali here at all, but I'd rather
>> see CloudStack loose code then have it added.
>>
>> Thinking about LXC, would like to see that removed together with various
>> other network plugins which I think are rarely used.
>>
>> Now, the idea of Murali was misunderstood by me. I think it would be worth
>> more if we would improve our cloud-init support and integration in various
>> tools which makes it much easier to deploy VMs running containers ON
>> CloudStack.
>>
>> Not so much changing CloudStack code, but rather tooling around it.
>>
>> If we have CloudStack talking to Kubernetes we suddenly have to maintain
>> that API integration. Who's going to do that. Realistically.
>>
>> That's my main worry in this regard.
>>
>> We have so much more work to do in ACS in total that I don't know if this
>> is the realistic route. I talk to many people who are not looking at
>> containers and are still working with VMs.
>>
>> There is not a single truth which is true, it really depends on who you
>> ask.
>>
>> Wido
>>
>> > Cheers,
>> >
>> > Will
>> >
>> > *Will STEVENS*
>> > Lead Developer
>> >
>> > <https://goo.gl/NYZ8KK>
>> >
>> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com> wrote:
>> >
>> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <muralimmreddy@gmail.com
>> >
>> > > wrote:
>> > > > All,
>> > > >
>> > > > I would like propose native functionality into CloudStack to provide
>> a
>> > > container service through which users out-of-the box can use to launch
>> > > container based application. Idea is to support ability to orchestrate
>> the
>> > > resources and automate aspects of setting up container orchestrator
>> through
>> > > CloudStack. Public IAAS service providers AWS with its ECS [1] and
>> google
>> > > with GKE [2] already provides ability container applications.
>> Competitive
>> > > cloud orchestration platforms already have native support for container
>> > > service. Users of CloudStack both as public cloud providers and users
>> with
>> > > private clouds will benefit with such functionality.
>> > > >
>> > > > While container orchestrator of user choice can be provisioned on
>> top of
>> > > CloudStack (with out CloudStack being involved) with tools like
>> > > TerraForm[3], Ansible[4] etc, advantage of having native orchestration
>> is
>> > > giving user a nice cohesive integration. This proposal would like add a
>> > > notion of first class CloudStack entity called container cluster which
>> can
>> > > be used to provision resources, scale up, scale down, start and stop
>> the
>> > > cluster of VM’s on which containerised applications can be run. For
>> actual
>> > > container orchestration we will still need container orchestrator like
>> > > docker swarm, marathon, kubernetes, but CloudStack container service
>> can
>> > > automate setting up of control place automatically.
>> > > >
>> > >
>> > > To be honest I'm torn on this one.
>> > >
>> > > Containers are a rapid changing thing, and while docker swam,
>> > > kubernetes, rancher or whatnot is popular today, they might not be
>> > > tomorrow.
>> > > They might use CoreOS today, but might not tomorrow.
>> > >
>> > > We have a rather poor track record of staying up to date with new
>> > > features/versions, and adding a feature that is so rapidly changing
>> > > is, I fear, going to be hard to maintain.
>> > > Want an example, look at xenserver. It is one of the most used
>> > > hypervisors we support, yet it took 6 months or so for us to support
>> > > the latest release.
>> > > Or IPv6...
>> > >
>> > > I don't mean to bash at maintainers/implementers of those features, I
>> > > appreciate all the work being done in every aspect, but I believe we
>> > > should be realistic and realize that we have issues with keeping stuff
>> > > up to date.
>> > >
>> > > I'd say focus on making sure other tools can do their job well against
>> > > CloudStack (kops, rancher, ++), but that does not mean I will -1 the
>> > > idea if anyone really wants to go through with it.
>> > >
>> > > --
>> > > Erik
>> > >
>>

murali.reddy@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: [PROPOSAL] add native container orchestration service

[Will Stevens <ws...@cloudops.com>]

I agree that we need to be careful what we take on and own inside
CloudStack.  I feel like some of the plugins or integrations which we have
been "maintaining" may serve us better to abandon, but I feel like that is
a whole discussion on its own.

In this case, I feel like there is a minimum viable solution which puts
CloudStack in a pretty good place to enable container orchestration.  For
example, one of the biggest challenges with K8S is the fact that it is
single tenant.  CloudStack has good multi tenancy support and has the
ability to orchestrate the underlying infra quite well.  We will have to be
very careful not to try to own too deep into the K8S world though, in my
opinion.  We only want to be responsible for providing the infra (and a way
to bootstrap K8S ideally) and be able to scale the infra, everything else
should be owned by the K8S on top.  That is the way I see it anyway, but
please add your input.

I think it is a liability to try to go too deep, for the same reasons Wido
and Erik have mentioned.  But I also think we need to take it seriously
because that train is moving and this may be a good opportunity to stay
relevant in a rapidly changing market.

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander <wi...@widodh.nl> wrote:

>
> > Op 27 januari 2017 om 16:08 schreef Will Stevens <wstevens@cloudops.com
> >:
> >
> >
> > Hey Murali,
> > How different is this proposal than what ShapeBlue already built.  It
> looks
> > pretty consistent with the functionality that you guys open sourced in
> > Seville.
> >
> > I have not yet used this functionality, but I have reports that it works
> > quite well.
> >
> > I believe the premise here is to only orchestrate the VM layer and
> > basically expose a "group" of running VMs to the user.  The user is
> > responsible for configuring K8S or whatever other container orchestrator
> on
> > top.  I saw mention of the "cloud-config" scripts in the FS, how are
> those
> > exposed to the cluster?  Maybe the FS can expand on that a bit?
> >
> > I believe the core feature that is being requested to be added is the
> > ability to create a group of VMs which will be kept active as a group if
> at
> > all possible.  ACS would be responsible for making sure that the number
> of
> > VMs specified for the group are in running state and it would spin up new
> > VMs as needed in order to satisfy the group settings.  In general, it is
> > understood that any application running on this group would have to be
> > fault tolerant enough to be able to rediscover a new VM if one fails and
> is
> > replaced by a fresh copy.  Is that fair to say?  How is it expected that
> > this service discovery is done, just by VMs being present on the network?
> >
> > As for some of the other people's concerns in this thread.
> >
> > - Regarding Wido's remarks.  I understand that there is some added
> > complexity, but I don't feel like the scope of the addition is
> > unrealistic.  I think the LXC integration was a lot farther out of the
> > scope of what ACS does then this is.  This does not change the "things"
> > which ACS orchestrates, it just adds the concept of a grouping of things
> > which ACS already manages.  I think this is the right approach since it
> is
> > not trying to be a container orchestrator.  We will never compete with
> K8S,
> > for example, and we should not try, but K8S is here and the market wants
> > it.  I do think we should be keeping our head up about that fact because
> > being able to provide a the underlay for K8S is very valuable in the
> > current marketplace.  I see this functionality as a way to enable K8S
> > adoption on top of ACS without changing our core values.
> >
> > - Regarding Erik's remarks.  The container space is moving fast, but so
> is
> > the industry.  If we want to remain relevant, we need to be able to
> adapt a
> > bit.  I don't think this is a big shift in what we do, but it is one that
> > enables people to be able to start running with something like K8S on top
> > of their existing ACS.  This is something we are interested in doing and
> so
> > are our customers.  If we can have a thin layer in ACS which helps enable
> > the use of K8S (or other container orchestrators) by orchestrating
> > infrastructure, as we already do, and making it easier to adopt a
> container
> > orchestrator running on top of ACS, I think that gives us a nice foothold
> > in the market.  I don't really feel it is fair to compare containers to
> > IPv6.  IPv6 has been out forever and it has taken almost a decade to get
> > anyone to adopt it.  Containers have really only been here for like 2
> years
> > and they are changing the market landscape in a very real way.
> >
> > Kind of on topic and kind of off topic.  I think understanding our
> approach
> > to containers is going to be important for the ACS community as a whole.
> > If we don't offer that market anything, then we will not be considered
> and
> > we will lose market share we can't afford to lose.  If we try to hitch
> our
> > horse to that cart too much, we will not be able to be agile enough and
> > will fail.  I feel like the right approach is for us to know that it is a
> > thriving market and continue to do what we do, but to extend an olive
> > branch to that market.  I think this sort of implementation is the right
> > approach because we are not trying to do too much.  We are simply giving
> a
> > foundation on which the next big thing in the container orchestration
> world
> > can adopt without us having to compete directly in that space.  I think
> we
> > have to focus on what we do best, but at the same time, think about what
> we
> > can do to enable that huge market of users to adopt ACS as their
> > foundation.  The ability to offer VMs and containers in the same data
> plane
> > is something we have the ability to do, especially with this approach,
> and
> > is something that most other softwares can not do.  The adoption of
> > containers by bigger organizations will be only part of their workload,
> > they will still be running VMs for the foreseeable future. Being able to
> > appeal to that market is going to be important for us.
> >
> > Hopefully I don't have too many strong opinions here, but I do think we
> > need to be thinking about how we move forward in a world which is
> adopting
> > containers in a very real way.
> >
>
> Understood. I just want to prevent that we add more features to CloudStack
> which are poorly maintained. Not judging Murali here at all, but I'd rather
> see CloudStack loose code then have it added.
>
> Thinking about LXC, would like to see that removed together with various
> other network plugins which I think are rarely used.
>
> Now, the idea of Murali was misunderstood by me. I think it would be worth
> more if we would improve our cloud-init support and integration in various
> tools which makes it much easier to deploy VMs running containers ON
> CloudStack.
>
> Not so much changing CloudStack code, but rather tooling around it.
>
> If we have CloudStack talking to Kubernetes we suddenly have to maintain
> that API integration. Who's going to do that. Realistically.
>
> That's my main worry in this regard.
>
> We have so much more work to do in ACS in total that I don't know if this
> is the realistic route. I talk to many people who are not looking at
> containers and are still working with VMs.
>
> There is not a single truth which is true, it really depends on who you
> ask.
>
> Wido
>
> > Cheers,
> >
> > Will
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com> wrote:
> >
> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <muralimmreddy@gmail.com
> >
> > > wrote:
> > > > All,
> > > >
> > > > I would like propose native functionality into CloudStack to provide
> a
> > > container service through which users out-of-the box can use to launch
> > > container based application. Idea is to support ability to orchestrate
> the
> > > resources and automate aspects of setting up container orchestrator
> through
> > > CloudStack. Public IAAS service providers AWS with its ECS [1] and
> google
> > > with GKE [2] already provides ability container applications.
> Competitive
> > > cloud orchestration platforms already have native support for container
> > > service. Users of CloudStack both as public cloud providers and users
> with
> > > private clouds will benefit with such functionality.
> > > >
> > > > While container orchestrator of user choice can be provisioned on
> top of
> > > CloudStack (with out CloudStack being involved) with tools like
> > > TerraForm[3], Ansible[4] etc, advantage of having native orchestration
> is
> > > giving user a nice cohesive integration. This proposal would like add a
> > > notion of first class CloudStack entity called container cluster which
> can
> > > be used to provision resources, scale up, scale down, start and stop
> the
> > > cluster of VM’s on which containerised applications can be run. For
> actual
> > > container orchestration we will still need container orchestrator like
> > > docker swarm, marathon, kubernetes, but CloudStack container service
> can
> > > automate setting up of control place automatically.
> > > >
> > >
> > > To be honest I'm torn on this one.
> > >
> > > Containers are a rapid changing thing, and while docker swam,
> > > kubernetes, rancher or whatnot is popular today, they might not be
> > > tomorrow.
> > > They might use CoreOS today, but might not tomorrow.
> > >
> > > We have a rather poor track record of staying up to date with new
> > > features/versions, and adding a feature that is so rapidly changing
> > > is, I fear, going to be hard to maintain.
> > > Want an example, look at xenserver. It is one of the most used
> > > hypervisors we support, yet it took 6 months or so for us to support
> > > the latest release.
> > > Or IPv6...
> > >
> > > I don't mean to bash at maintainers/implementers of those features, I
> > > appreciate all the work being done in every aspect, but I believe we
> > > should be realistic and realize that we have issues with keeping stuff
> > > up to date.
> > >
> > > I'd say focus on making sure other tools can do their job well against
> > > CloudStack (kops, rancher, ++), but that does not mean I will -1 the
> > > idea if anyone really wants to go through with it.
> > >
> > > --
> > > Erik
> > >
>

Re: [PROPOSAL] add native container orchestration service

[Wido den Hollander <wi...@widodh.nl>]

> Op 27 januari 2017 om 16:08 schreef Will Stevens <ws...@cloudops.com>:
> 
> 
> Hey Murali,
> How different is this proposal than what ShapeBlue already built.  It looks
> pretty consistent with the functionality that you guys open sourced in
> Seville.
> 
> I have not yet used this functionality, but I have reports that it works
> quite well.
> 
> I believe the premise here is to only orchestrate the VM layer and
> basically expose a "group" of running VMs to the user.  The user is
> responsible for configuring K8S or whatever other container orchestrator on
> top.  I saw mention of the "cloud-config" scripts in the FS, how are those
> exposed to the cluster?  Maybe the FS can expand on that a bit?
> 
> I believe the core feature that is being requested to be added is the
> ability to create a group of VMs which will be kept active as a group if at
> all possible.  ACS would be responsible for making sure that the number of
> VMs specified for the group are in running state and it would spin up new
> VMs as needed in order to satisfy the group settings.  In general, it is
> understood that any application running on this group would have to be
> fault tolerant enough to be able to rediscover a new VM if one fails and is
> replaced by a fresh copy.  Is that fair to say?  How is it expected that
> this service discovery is done, just by VMs being present on the network?
> 
> As for some of the other people's concerns in this thread.
> 
> - Regarding Wido's remarks.  I understand that there is some added
> complexity, but I don't feel like the scope of the addition is
> unrealistic.  I think the LXC integration was a lot farther out of the
> scope of what ACS does then this is.  This does not change the "things"
> which ACS orchestrates, it just adds the concept of a grouping of things
> which ACS already manages.  I think this is the right approach since it is
> not trying to be a container orchestrator.  We will never compete with K8S,
> for example, and we should not try, but K8S is here and the market wants
> it.  I do think we should be keeping our head up about that fact because
> being able to provide a the underlay for K8S is very valuable in the
> current marketplace.  I see this functionality as a way to enable K8S
> adoption on top of ACS without changing our core values.
> 
> - Regarding Erik's remarks.  The container space is moving fast, but so is
> the industry.  If we want to remain relevant, we need to be able to adapt a
> bit.  I don't think this is a big shift in what we do, but it is one that
> enables people to be able to start running with something like K8S on top
> of their existing ACS.  This is something we are interested in doing and so
> are our customers.  If we can have a thin layer in ACS which helps enable
> the use of K8S (or other container orchestrators) by orchestrating
> infrastructure, as we already do, and making it easier to adopt a container
> orchestrator running on top of ACS, I think that gives us a nice foothold
> in the market.  I don't really feel it is fair to compare containers to
> IPv6.  IPv6 has been out forever and it has taken almost a decade to get
> anyone to adopt it.  Containers have really only been here for like 2 years
> and they are changing the market landscape in a very real way.
> 
> Kind of on topic and kind of off topic.  I think understanding our approach
> to containers is going to be important for the ACS community as a whole.
> If we don't offer that market anything, then we will not be considered and
> we will lose market share we can't afford to lose.  If we try to hitch our
> horse to that cart too much, we will not be able to be agile enough and
> will fail.  I feel like the right approach is for us to know that it is a
> thriving market and continue to do what we do, but to extend an olive
> branch to that market.  I think this sort of implementation is the right
> approach because we are not trying to do too much.  We are simply giving a
> foundation on which the next big thing in the container orchestration world
> can adopt without us having to compete directly in that space.  I think we
> have to focus on what we do best, but at the same time, think about what we
> can do to enable that huge market of users to adopt ACS as their
> foundation.  The ability to offer VMs and containers in the same data plane
> is something we have the ability to do, especially with this approach, and
> is something that most other softwares can not do.  The adoption of
> containers by bigger organizations will be only part of their workload,
> they will still be running VMs for the foreseeable future. Being able to
> appeal to that market is going to be important for us.
> 
> Hopefully I don't have too many strong opinions here, but I do think we
> need to be thinking about how we move forward in a world which is adopting
> containers in a very real way.
> 

Understood. I just want to prevent that we add more features to CloudStack which are poorly maintained. Not judging Murali here at all, but I'd rather see CloudStack loose code then have it added.

Thinking about LXC, would like to see that removed together with various other network plugins which I think are rarely used.

Now, the idea of Murali was misunderstood by me. I think it would be worth more if we would improve our cloud-init support and integration in various tools which makes it much easier to deploy VMs running containers ON CloudStack.

Not so much changing CloudStack code, but rather tooling around it.

If we have CloudStack talking to Kubernetes we suddenly have to maintain that API integration. Who's going to do that. Realistically.

That's my main worry in this regard.

We have so much more work to do in ACS in total that I don't know if this is the realistic route. I talk to many people who are not looking at containers and are still working with VMs.

There is not a single truth which is true, it really depends on who you ask.

Wido

> Cheers,
> 
> Will
> 
> *Will STEVENS*
> Lead Developer
> 
> <https://goo.gl/NYZ8KK>
> 
> On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com> wrote:
> 
> > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <mu...@gmail.com>
> > wrote:
> > > All,
> > >
> > > I would like propose native functionality into CloudStack to provide a
> > container service through which users out-of-the box can use to launch
> > container based application. Idea is to support ability to orchestrate the
> > resources and automate aspects of setting up container orchestrator through
> > CloudStack. Public IAAS service providers AWS with its ECS [1] and google
> > with GKE [2] already provides ability container applications.  Competitive
> > cloud orchestration platforms already have native support for container
> > service. Users of CloudStack both as public cloud providers and users with
> > private clouds will benefit with such functionality.
> > >
> > > While container orchestrator of user choice can be provisioned on top of
> > CloudStack (with out CloudStack being involved) with tools like
> > TerraForm[3], Ansible[4] etc, advantage of having native orchestration is
> > giving user a nice cohesive integration. This proposal would like add a
> > notion of first class CloudStack entity called container cluster which can
> > be used to provision resources, scale up, scale down, start and stop the
> > cluster of VM’s on which containerised applications can be run. For actual
> > container orchestration we will still need container orchestrator like
> > docker swarm, marathon, kubernetes, but CloudStack container service can
> > automate setting up of control place automatically.
> > >
> >
> > To be honest I'm torn on this one.
> >
> > Containers are a rapid changing thing, and while docker swam,
> > kubernetes, rancher or whatnot is popular today, they might not be
> > tomorrow.
> > They might use CoreOS today, but might not tomorrow.
> >
> > We have a rather poor track record of staying up to date with new
> > features/versions, and adding a feature that is so rapidly changing
> > is, I fear, going to be hard to maintain.
> > Want an example, look at xenserver. It is one of the most used
> > hypervisors we support, yet it took 6 months or so for us to support
> > the latest release.
> > Or IPv6...
> >
> > I don't mean to bash at maintainers/implementers of those features, I
> > appreciate all the work being done in every aspect, but I believe we
> > should be realistic and realize that we have issues with keeping stuff
> > up to date.
> >
> > I'd say focus on making sure other tools can do their job well against
> > CloudStack (kops, rancher, ++), but that does not mean I will -1 the
> > idea if anyone really wants to go through with it.
> >
> > --
> > Erik
> >

Re: [PROPOSAL] add native container orchestration service

[Will Stevens <ws...@cloudops.com>]

Hey Murali,
How different is this proposal than what ShapeBlue already built.  It looks
pretty consistent with the functionality that you guys open sourced in
Seville.

I have not yet used this functionality, but I have reports that it works
quite well.

I believe the premise here is to only orchestrate the VM layer and
basically expose a "group" of running VMs to the user.  The user is
responsible for configuring K8S or whatever other container orchestrator on
top.  I saw mention of the "cloud-config" scripts in the FS, how are those
exposed to the cluster?  Maybe the FS can expand on that a bit?

I believe the core feature that is being requested to be added is the
ability to create a group of VMs which will be kept active as a group if at
all possible.  ACS would be responsible for making sure that the number of
VMs specified for the group are in running state and it would spin up new
VMs as needed in order to satisfy the group settings.  In general, it is
understood that any application running on this group would have to be
fault tolerant enough to be able to rediscover a new VM if one fails and is
replaced by a fresh copy.  Is that fair to say?  How is it expected that
this service discovery is done, just by VMs being present on the network?

As for some of the other people's concerns in this thread.

- Regarding Wido's remarks.  I understand that there is some added
complexity, but I don't feel like the scope of the addition is
unrealistic.  I think the LXC integration was a lot farther out of the
scope of what ACS does then this is.  This does not change the "things"
which ACS orchestrates, it just adds the concept of a grouping of things
which ACS already manages.  I think this is the right approach since it is
not trying to be a container orchestrator.  We will never compete with K8S,
for example, and we should not try, but K8S is here and the market wants
it.  I do think we should be keeping our head up about that fact because
being able to provide a the underlay for K8S is very valuable in the
current marketplace.  I see this functionality as a way to enable K8S
adoption on top of ACS without changing our core values.

- Regarding Erik's remarks.  The container space is moving fast, but so is
the industry.  If we want to remain relevant, we need to be able to adapt a
bit.  I don't think this is a big shift in what we do, but it is one that
enables people to be able to start running with something like K8S on top
of their existing ACS.  This is something we are interested in doing and so
are our customers.  If we can have a thin layer in ACS which helps enable
the use of K8S (or other container orchestrators) by orchestrating
infrastructure, as we already do, and making it easier to adopt a container
orchestrator running on top of ACS, I think that gives us a nice foothold
in the market.  I don't really feel it is fair to compare containers to
IPv6.  IPv6 has been out forever and it has taken almost a decade to get
anyone to adopt it.  Containers have really only been here for like 2 years
and they are changing the market landscape in a very real way.

Kind of on topic and kind of off topic.  I think understanding our approach
to containers is going to be important for the ACS community as a whole.
If we don't offer that market anything, then we will not be considered and
we will lose market share we can't afford to lose.  If we try to hitch our
horse to that cart too much, we will not be able to be agile enough and
will fail.  I feel like the right approach is for us to know that it is a
thriving market and continue to do what we do, but to extend an olive
branch to that market.  I think this sort of implementation is the right
approach because we are not trying to do too much.  We are simply giving a
foundation on which the next big thing in the container orchestration world
can adopt without us having to compete directly in that space.  I think we
have to focus on what we do best, but at the same time, think about what we
can do to enable that huge market of users to adopt ACS as their
foundation.  The ability to offer VMs and containers in the same data plane
is something we have the ability to do, especially with this approach, and
is something that most other softwares can not do.  The adoption of
containers by bigger organizations will be only part of their workload,
they will still be running VMs for the foreseeable future. Being able to
appeal to that market is going to be important for us.

Hopefully I don't have too many strong opinions here, but I do think we
need to be thinking about how we move forward in a world which is adopting
containers in a very real way.

Cheers,

Will

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber <te...@gmail.com> wrote:

> On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <mu...@gmail.com>
> wrote:
> > All,
> >
> > I would like propose native functionality into CloudStack to provide a
> container service through which users out-of-the box can use to launch
> container based application. Idea is to support ability to orchestrate the
> resources and automate aspects of setting up container orchestrator through
> CloudStack. Public IAAS service providers AWS with its ECS [1] and google
> with GKE [2] already provides ability container applications.  Competitive
> cloud orchestration platforms already have native support for container
> service. Users of CloudStack both as public cloud providers and users with
> private clouds will benefit with such functionality.
> >
> > While container orchestrator of user choice can be provisioned on top of
> CloudStack (with out CloudStack being involved) with tools like
> TerraForm[3], Ansible[4] etc, advantage of having native orchestration is
> giving user a nice cohesive integration. This proposal would like add a
> notion of first class CloudStack entity called container cluster which can
> be used to provision resources, scale up, scale down, start and stop the
> cluster of VM’s on which containerised applications can be run. For actual
> container orchestration we will still need container orchestrator like
> docker swarm, marathon, kubernetes, but CloudStack container service can
> automate setting up of control place automatically.
> >
>
> To be honest I'm torn on this one.
>
> Containers are a rapid changing thing, and while docker swam,
> kubernetes, rancher or whatnot is popular today, they might not be
> tomorrow.
> They might use CoreOS today, but might not tomorrow.
>
> We have a rather poor track record of staying up to date with new
> features/versions, and adding a feature that is so rapidly changing
> is, I fear, going to be hard to maintain.
> Want an example, look at xenserver. It is one of the most used
> hypervisors we support, yet it took 6 months or so for us to support
> the latest release.
> Or IPv6...
>
> I don't mean to bash at maintainers/implementers of those features, I
> appreciate all the work being done in every aspect, but I believe we
> should be realistic and realize that we have issues with keeping stuff
> up to date.
>
> I'd say focus on making sure other tools can do their job well against
> CloudStack (kops, rancher, ++), but that does not mean I will -1 the
> idea if anyone really wants to go through with it.
>
> --
> Erik
>

Re: [PROPOSAL] add native container orchestration service

[Erik Weber <te...@gmail.com>]

On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <mu...@gmail.com> wrote:
> All,
>
> I would like propose native functionality into CloudStack to provide a container service through which users out-of-the box can use to launch container based application. Idea is to support ability to orchestrate the resources and automate aspects of setting up container orchestrator through CloudStack. Public IAAS service providers AWS with its ECS [1] and google with GKE [2] already provides ability container applications.  Competitive cloud orchestration platforms already have native support for container service. Users of CloudStack both as public cloud providers and users with private clouds will benefit with such functionality.
>
> While container orchestrator of user choice can be provisioned on top of CloudStack (with out CloudStack being involved) with tools like TerraForm[3], Ansible[4] etc, advantage of having native orchestration is giving user a nice cohesive integration. This proposal would like add a notion of first class CloudStack entity called container cluster which can be used to provision resources, scale up, scale down, start and stop the cluster of VM’s on which containerised applications can be run. For actual container orchestration we will still need container orchestrator like docker swarm, marathon, kubernetes, but CloudStack container service can automate setting up of control place automatically.
>

To be honest I'm torn on this one.

Containers are a rapid changing thing, and while docker swam,
kubernetes, rancher or whatnot is popular today, they might not be
tomorrow.
They might use CoreOS today, but might not tomorrow.

We have a rather poor track record of staying up to date with new
features/versions, and adding a feature that is so rapidly changing
is, I fear, going to be hard to maintain.
Want an example, look at xenserver. It is one of the most used
hypervisors we support, yet it took 6 months or so for us to support
the latest release.
Or IPv6...

I don't mean to bash at maintainers/implementers of those features, I
appreciate all the work being done in every aspect, but I believe we
should be realistic and realize that we have issues with keeping stuff
up to date.

I'd say focus on making sure other tools can do their job well against
CloudStack (kops, rancher, ++), but that does not mean I will -1 the
idea if anyone really wants to go through with it.

-- 
Erik