You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Daan Hoogland <da...@shapeblue.com> on 2017/02/20 13:56:55 UTC
[PROPOSAL] add native vm-cluster orchestration service
(was: [PROPOSAL] add native container orchestration service)
So, being very late in the discussion but havingread the whole thread before editting the title of this thread,
Can we agree that we want a generic vm-cluster service and leave the container bits to containers? Kishan can you share your design? Shapeblue wants to rebase their k8 service on top of this and I would like yours and Murali's work to not conflict.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands
@shapeblue
-----Original Message-----
From: Paul Angus [mailto:paul.angus@shapeblue.com]
Sent: dinsdag 7 februari 2017 08:14
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
Will is 100% correct. As I mentioned the Title is misleading. However, Murali did clarify in his explanation; this is really about vm cluster orchestration.
________________________________
From: Will Stevens <ws...@cloudops.com>
Sent: 6 Feb 2017 22:54
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
My understanding is that what Paul is talking about is what is already built and IS what the thread is talking about.
*Will STEVENS*
Lead Developer
<https://goo.gl/NYZ8KK>
On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani < rajesh.ramchandani@accelerite.com> wrote:
> Hi Paul - I think this is different from what the thread was about.
> The conversation was specifically about adding support for container
> orchestrators. You are talking about provisioning a group of VMs.
> Although, this is something I think several Cloudstack users have
> requested before and we should propose a solution to this.
>
> Raj
>
> ________________________________
> From: Paul Angus <pa...@shapeblue.com>
> Sent: Monday, February 6, 2017 11:16:41 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] add native container orchestration service
>
> #WhatHeSaid
>
> The title is misleading. The proposal is purely to add the notion of
> Clusters of VMs to CloudStack. These may be for databases, containers
> or anything else that needs 'clusters' of compute. Self-healing and
> autoscaling are logical next steps to be added.
>
> Those guys at ShapeBlue have open-sourced their whole k8s container
> service piece. If/when the 'cluster' part of that work is added into
> CloudStack, the k8s specific pieces can be used by anyone who wants
> to, alternatively they could be used for reference in order to create
> another types of cluster. (or ignored completely).
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
> -----Original Message-----
> From: Will Stevens [mailto:williamstevens@gmail.com]
> Sent: 31 January 2017 13:26
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> s/cloud-init/cloud-config/
>
> On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:
>
> > I think that is covered in this proposal. There is nothing k8s
> > specific in this integration (from what I understand), all the k8s
> > details are passed in via the cloud-init configuration after the cluster
> has been provisioned.
> >
> > On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com>
> > wrote:
> >
> >
> > There are many container orchestrators. Those container orchestrators
> > are happy to run on any VMs or bare metal machines. K8s is just one of
> > them and there will be more in the future. It may not be a good idea
> > to make CloudStack to be k8s aware. IMO, the relationship between k8s
> > and cloudstack should be similar to application and os. It probably
> > not a good idea to make your OS to be aware of any specific
> > applications so IMHO I don’t think k8s should be native to CloudStack.
> > It makes more sense to provide some generic services that many
> > applications can take advantages of. Some generic resource grouping
> > service makes sense so a group of VMs, baremetal machines or network
> > can be treated as a single entity. Some life cycle management will be
> > necessary for these entities too. We can deploy k8s, swarm, dcos or
> > even non-container specific services on top of CloudStack. The k8s is
> > changing fast. One single tenant installation may need more than one
> > VM group and may actually requires more (k8s federation). It will be a
> > struggle to be in sync if we try to bring k8s specific knowledge into
> cloudstack.
> >
> > Regards,
> >
> >
> > --
> > Lianghwa Jou
> > VP Engineering, Accelerite
> > email: lianghwa.jou@accelerite.com
> >
> >
> >
> >
> >
> > On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
> >
> >
> > I agree with some good views Will has shared and I also agree to
> > the concerns raised by Wido and Eric. IMO we need balance of staying
> > relevant/add new features vs stability of CloudStack and take
> > corrective action if needed. We have two good examples for both. When
> > SDN was hot technology CloudStack ended up with bunch of SDN controller
> integrations.
> > Few years later, now CloudStack is carrying baggage with no
> > maintainers for those plugins, with probably not many of CloudStack
> users needing overlays.
> > On the other hand, other than attempt to liaison with ETSI for NFV no
> > effort was done. OpenStack has become de-facto for NFV. Now for
> > OpenStack, arguably NFV has become larger use case than cloud it self.
> > I concur with Will’s point that with minimal viable solution that does
> > not change the core of CloudStack, and can keep CloudStack relevant is
> > worth to be taken in.
> >
> > Will,
> >
> > To your question of how different is from ShapeBlue’s container
> > service, its design, implementation and API semantics etc remain same.
> > ShapeBlue’s container service was true drop in plug-in to CloudStack,
> > with this proposal I am trying to re-work to make it a core CloudStack
> > pluggable service which is part of CloudStack.
> >
> > Key concepts that this proposal is trying to add
> >
> > - add notion of ‘container cluster’ as a first class entity in
> > CloudStack. Which is bacially collection of other CloudStack resources
> > (like VM’s, networks, public ip, network rules etc)
> > - life cycle operation to manage ‘container cluster’ like
> > create, delete, start, stop, scale-up, scale-down, heal etc
> > - orchestrate container orchestrator control plane on top of
> > provisioned resources
> >
> > At-least for k8s (which is what this proposal is targeting),
> > integration with k8s is bare minimum. There are cloud-config scripts
> > that automatically setup k8s cluster master and node VM’s. All
> > CloudStack is doing in passing the cloud-config to the core OS VM’s as
> user data.
> >
> > Regards
> > Murali Reddy
> >
> >
> >
> >
> >
> >
> >
> > On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
> > behalf of wstevens@cloudops.com> wrote:
> >
> > >I agree that we need to be careful what we take on and own inside
> > >CloudStack. I feel like some of the plugins or integrations
> > which we have
> > >been "maintaining" may serve us better to abandon, but I feel
> > like that is
> > >a whole discussion on its own.
> > >
> > >In this case, I feel like there is a minimum viable solution
> > which puts
> > >CloudStack in a pretty good place to enable container orchestration.
> > For
> > >example, one of the biggest challenges with K8S is the fact that it
> is
> > >single tenant. CloudStack has good multi tenancy support and has
> the
> > >ability to orchestrate the underlying infra quite well. We will
> > have to be
> > >very careful not to try to own too deep into the K8S world
> > though, in my
> > >opinion. We only want to be responsible for providing the infra
> > (and a way
> > >to bootstrap K8S ideally) and be able to scale the infra,
> > everything else
> > >should be owned by the K8S on top. That is the way I see it
> > anyway, but
> > >please add your input.
> > >
> > >I think it is a liability to try to go too deep, for the same
> > reasons Wido
> > >and Erik have mentioned. But I also think we need to take it
> > seriously
> > >because that train is moving and this may be a good opportunity
> > to stay
> > >relevant in a rapidly changing market.
> > >
> > >*Will STEVENS*
> > >Lead Developer
> > >
> > ><https://goo.gl/NYZ8KK>
> > >
> > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > <wi...@widodh.nl>
> > wrote:
> > >
> > >>
> > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > wstevens@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > Hey Murali,
> > >> > How different is this proposal than what ShapeBlue already
> > built. It
> > >> looks
> > >> > pretty consistent with the functionality that you guys open
> > sourced in
> > >> > Seville.
> > >> >
> > >> > I have not yet used this functionality, but I have reports
> > that it works
> > >> > quite well.
> > >> >
> > >> > I believe the premise here is to only orchestrate the VM layer
> and
> > >> > basically expose a "group" of running VMs to the user. The
> > user is
> > >> > responsible for configuring K8S or whatever other container
> > orchestrator
> > >> on
> > >> > top. I saw mention of the "cloud-config" scripts in the FS,
> > how are
> > >> those
> > >> > exposed to the cluster? Maybe the FS can expand on that a bit?
> > >> >
> > >> > I believe the core feature that is being requested to be
> > added is the
> > >> > ability to create a group of VMs which will be kept active as
> > a group if
> > >> at
> > >> > all possible. ACS would be responsible for making sure that
> > the number
> > >> of
> > >> > VMs specified for the group are in running state and it would
> > spin up new
> > >> > VMs as needed in order to satisfy the group settings. In
> > general, it is
> > >> > understood that any application running on this group would
> > have to be
> > >> > fault tolerant enough to be able to rediscover a new VM if
> > one fails and
> > >> is
> > >> > replaced by a fresh copy. Is that fair to say? How is it
> > expected that
> > >> > this service discovery is done, just by VMs being present on
> > the network?
> > >> >
> > >> > As for some of the other people's concerns in this thread.
> > >> >
> > >> > - Regarding Wido's remarks. I understand that there is some
> added
> > >> > complexity, but I don't feel like the scope of the addition is
> > >> > unrealistic. I think the LXC integration was a lot farther
> > out of the
> > >> > scope of what ACS does then this is. This does not change
> > the "things"
> > >> > which ACS orchestrates, it just adds the concept of a
> > grouping of things
> > >> > which ACS already manages. I think this is the right
> > approach since it
> > >> is
> > >> > not trying to be a container orchestrator. We will never
> > compete with
> > >> K8S,
> > >> > for example, and we should not try, but K8S is here and the
> > market wants
> > >> > it. I do think we should be keeping our head up about that
> > fact because
> > >> > being able to provide a the underlay for K8S is very valuable
> > in the
> > >> > current marketplace. I see this functionality as a way to
> > enable K8S
> > >> > adoption on top of ACS without changing our core values.
> > >> >
> > >> > - Regarding Erik's remarks. The container space is moving
> > fast, but so
> > >> is
> > >> > the industry. If we want to remain relevant, we need to be
> > able to
> > >> adapt a
> > >> > bit. I don't think this is a big shift in what we do, but it
> > is one that
> > >> > enables people to be able to start running with something
> > like K8S on top
> > >> > of their existing ACS. This is something we are interested
> > in doing and
> > >> so
> > >> > are our customers. If we can have a thin layer in ACS which
> > helps enable
> > >> > the use of K8S (or other container orchestrators) by
> orchestrating
> > >> > infrastructure, as we already do, and making it easier to adopt
> a
> > >> container
> > >> > orchestrator running on top of ACS, I think that gives us a
> > nice foothold
> > >> > in the market. I don't really feel it is fair to compare
> > containers to
> > >> > IPv6. IPv6 has been out forever and it has taken almost a
> > decade to get
> > >> > anyone to adopt it. Containers have really only been here
> > for like 2
> > >> years
> > >> > and they are changing the market landscape in a very real way.
> > >> >
> > >> > Kind of on topic and kind of off topic. I think understanding
> our
> > >> approach
> > >> > to containers is going to be important for the ACS community
> > as a whole.
> > >> > If we don't offer that market anything, then we will not be
> > considered
> > >> and
> > >> > we will lose market share we can't afford to lose. If we try
> > to hitch
> > >> our
> > >> > horse to that cart too much, we will not be able to be agile
> > enough and
> > >> > will fail. I feel like the right approach is for us to know
> > that it is a
> > >> > thriving market and continue to do what we do, but to extend
> > an olive
> > >> > branch to that market. I think this sort of implementation
> > is the right
> > >> > approach because we are not trying to do too much. We are
> > simply giving
> > >> a
> > >> > foundation on which the next big thing in the container
> > orchestration
> > >> world
> > >> > can adopt without us having to compete directly in that
> > space. I think
> > >> we
> > >> > have to focus on what we do best, but at the same time, think
> > about what
> > >> we
> > >> > can do to enable that huge market of users to adopt ACS as their
> > >> > foundation. The ability to offer VMs and containers in the
> > same data
> > >> plane
> > >> > is something we have the ability to do, especially with this
> > approach,
> > >> and
> > >> > is something that most other softwares can not do. The
> > adoption of
> > >> > containers by bigger organizations will be only part of their
> > workload,
> > >> > they will still be running VMs for the foreseeable future.
> > Being able to
> > >> > appeal to that market is going to be important for us.
> > >> >
> > >> > Hopefully I don't have too many strong opinions here, but I
> > do think we
> > >> > need to be thinking about how we move forward in a world which
> is
> > >> adopting
> > >> > containers in a very real way.
> > >> >
> > >>
> > >> Understood. I just want to prevent that we add more features to
> > CloudStack
> > >> which are poorly maintained. Not judging Murali here at all,
> > but I'd rather
> > >> see CloudStack loose code then have it added.
> > >>
> > >> Thinking about LXC, would like to see that removed together
> > with various
> > >> other network plugins which I think are rarely used.
> > >>
> > >> Now, the idea of Murali was misunderstood by me. I think it
> > would be worth
> > >> more if we would improve our cloud-init support and integration
> > in various
> > >> tools which makes it much easier to deploy VMs running containers
> ON
> > >> CloudStack.
> > >>
> > >> Not so much changing CloudStack code, but rather tooling around
> it.
> > >>
> > >> If we have CloudStack talking to Kubernetes we suddenly have to
> > maintain
> > >> that API integration. Who's going to do that. Realistically.
> > >>
> > >> That's my main worry in this regard.
> > >>
> > >> We have so much more work to do in ACS in total that I don't
> > know if this
> > >> is the realistic route. I talk to many people who are not looking
> at
> > >> containers and are still working with VMs.
> > >>
> > >> There is not a single truth which is true, it really depends on
> > who you
> > >> ask.
> > >>
> > >> Wido
> > >>
> > >> > Cheers,
> > >> >
> > >> > Will
> > >> >
> > >> > *Will STEVENS*
> > >> > Lead Developer
> > >> >
> > >> > <https://goo.gl/NYZ8KK>
> > >> >
> > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > <te...@gmail.com>
> > wrote:
> > >> >
> > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > muralimmreddy@gmail.com
> > >> >
> > >> > > wrote:
> > >> > > > All,
> > >> > > >
> > >> > > > I would like propose native functionality into CloudStack
> > to provide
> > >> a
> > >> > > container service through which users out-of-the box can
> > use to launch
> > >> > > container based application. Idea is to support ability to
> > orchestrate
> > >> the
> > >> > > resources and automate aspects of setting up container
> > orchestrator
> > >> through
> > >> > > CloudStack. Public IAAS service providers AWS with its ECS
> > [1] and
> > >> google
> > >> > > with GKE [2] already provides ability container applications.
> > >> Competitive
> > >> > > cloud orchestration platforms already have native support
> > for container
> > >> > > service. Users of CloudStack both as public cloud providers
> > and users
> > >> with
> > >> > > private clouds will benefit with such functionality.
> > >> > > >
> > >> > > > While container orchestrator of user choice can be
> > provisioned on
> > >> top of
> > >> > > CloudStack (with out CloudStack being involved) with tools
> like
> > >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> > orchestration
> > >> is
> > >> > > giving user a nice cohesive integration. This proposal
> > would like add a
> > >> > > notion of first class CloudStack entity called container
> > cluster which
> > >> can
> > >> > > be used to provision resources, scale up, scale down, start
> > and stop
> > >> the
> > >> > > cluster of VM’s on which containerised applications can be
> run.
> > For
> > >> actual
> > >> > > container orchestration we will still need container
> > orchestrator like
> > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > container service
> > >> can
> > >> > > automate setting up of control place automatically.
> > >> > > >
> > >> > >
> > >> > > To be honest I'm torn on this one.
> > >> > >
> > >> > > Containers are a rapid changing thing, and while docker swam,
> > >> > > kubernetes, rancher or whatnot is popular today, they might
> > not be
> > >> > > tomorrow.
> > >> > > They might use CoreOS today, but might not tomorrow.
> > >> > >
> > >> > > We have a rather poor track record of staying up to date
> > with new
> > >> > > features/versions, and adding a feature that is so rapidly
> > changing
> > >> > > is, I fear, going to be hard to maintain.
> > >> > > Want an example, look at xenserver. It is one of the most used
> > >> > > hypervisors we support, yet it took 6 months or so for us
> > to support
> > >> > > the latest release.
> > >> > > Or IPv6...
> > >> > >
> > >> > > I don't mean to bash at maintainers/implementers of those
> > features, I
> > >> > > appreciate all the work being done in every aspect, but I
> > believe we
> > >> > > should be realistic and realize that we have issues with
> > keeping stuff
> > >> > > up to date.
> > >> > >
> > >> > > I'd say focus on making sure other tools can do their job
> > well against
> > >> > > CloudStack (kops, rancher, ++), but that does not mean I
> > will
> > -1 the
> > >> > > idea if anyone really wants to go through with it.
> > >> > >
> > >> > > --
> > >> > > Erik
> > >> > >
> > >>
> >
> > murali.reddy@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which
> > is the property of Accelerite, a Persistent Systems business. It is
> > intended only for the use of the individual or entity to which it is
> > addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Accelerite, a
> > Persistent Systems business does not accept any liability for virus
> infected mails.
> >
> >
> >
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: [PROPOSAL] add native vm-cluster orchestration service (was:
[PROPOSAL] add native container orchestration service)
Posted by Rene Moser <ma...@renemoser.net>.
On 02/20/2017 02:56 PM, Daan Hoogland wrote:
> So, being very late in the discussion but havingread the whole thread before editting the title of this thread,
>
> Can we agree that we want a generic vm-cluster service and leave the container bits to containers? Kishan can you share your design? Shapeblue wants to rebase their k8 service on top of this and I would like yours and Murali's work to not conflict.
+1
Re: [PROPOSAL] add native vm-cluster orchestration service (was:
[PROPOSAL] add native container orchestration service)
Posted by Syed Ahmed <sa...@cloudops.com>.
+1 for ApplicationCluster. I think it is generic and not tied to a specific
concept (machine,VM, service etc)
On Wed, Mar 1, 2017 at 1:54 AM, Daan Hoogland <da...@shapeblue.com>
wrote:
> Syed, Kilham, et. al.
>
> After some thought I came up with ApplicationCluster. It is not purely
> machines or instances but includes network and storage resources and maybe
> more. Next to that these are meant for running application like k8, mesos
> or DBaaS. I don't like service as prefix because it implies the cluster is
> for servicing the cloud or VMs that may be broken or need a kind of extra
> feature while from user perspective they are an addition, hence application
> to cloudstack.
>
> Any push back?
>
> Sent from Nine<http://www.9folders.com/>
> ________________________________
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
> From: Daan Hoogland
> Sent: 28 Feb 2017 6:49 pm
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native vm-cluster orchestration service (was:
> [PROPOSAL] add native container orchestration service)
>
> Syed, I chose machine as they might be bare metal in some cases.
>
> Sent from Nine<http://www.9folders.com/>
> ________________________________
> From: Syed Ahmed <sa...@cloudops.com>
> Sent: 28 Feb 2017 4:22 pm
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native vm-cluster orchestration service (was:
> [PROPOSAL] add native container orchestration service)
>
> We already call the VMs as Instances. So, InstanceCluster would be a better
> name imo.
>
> On Tue, Feb 28, 2017 at 10:05 AM, Daan Hoogland <
> daan.hoogland@shapeblue.com
> > wrote:
>
> > Kishan, I see some sensible additions but also some unnecessary
> omissions.
> > Most of it seems to be Murali’s text so I’ll c&p your improvements back
> and
> > rename the page to the more sensible title of “MachineCluster service”
> and
> > delete the other.
> >
> > About naming, I was thinking MachineCluster instead of ServiceCluster,
> > makes sense? Or even GuestMachineCluster. ServiceCluster could mean a
> > supporting cluster that delivers e.g. backup as a service to guests, or
> > maybe some build, artifact or networking service. For this ambiguity im
> am
> > :-1: on the name ServiceCluster.
> >
> >
> > On 28/02/17 11:16, "Kishan Kavala" <ki...@accelerite.com> wrote:
> >
> > Daan,
> > I've updated the earlier spec to support any Vm cluster. Please let
> > me know your thoughts on this.
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> > Service+Cluster+Functional+Specification
> >
> > regards,
> > Kishan
> >
> > -----Original Message-----
> > From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
> > Sent: 27 February 2017 04:02 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] add native vm-cluster orchestration service
> > (was: [PROPOSAL] add native container orchestration service)
> >
> > Any follow up Koushik? I want to refactor our proof of concept and
> > integrate it in master.
> >
> > On 21/02/17 10:42, "Kishan Kavala" <ki...@accelerite.com>
> > wrote:
> >
> > Sure Daan. I'll publish the design on cwiki and share the link.
> >
> > -----Original Message-----
> > From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
> > Sent: Monday, February 20, 2017 7:27 PM
> > To: dev@cloudstack.apache.org
> > Subject: [PROPOSAL] add native vm-cluster orchestration service
> > (was: [PROPOSAL] add native container orchestration service)
> >
> > So, being very late in the discussion but havingread the whole
> > thread before editting the title of this thread,
> >
> > Can we agree that we want a generic vm-cluster service and leave
> > the container bits to containers? Kishan can you share your design?
> > Shapeblue wants to rebase their k8 service on top of this and I would
> like
> > yours and Murali's work to not conflict.
> >
> > daan.hoogland@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531
> > VENetherlands @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Paul Angus [mailto:paul.angus@shapeblue.com]
> > Sent: dinsdag 7 februari 2017 08:14
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] add native container orchestration
> service
> >
> > Will is 100% correct. As I mentioned the Title is misleading.
> > However, Murali did clarify in his explanation; this is really about vm
> > cluster orchestration.
> >
> >
> >
> > ________________________________
> > From: Will Stevens <ws...@cloudops.com>
> > Sent: 6 Feb 2017 22:54
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] add native container orchestration
> service
> >
> > My understanding is that what Paul is talking about is what is
> > already built and IS what the thread is talking about.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani <
> > rajesh.ramchandani@accelerite.com> wrote:
> >
> > > Hi Paul - I think this is different from what the thread was
> > about.
> > > The conversation was specifically about adding support for
> > container
> > > orchestrators. You are talking about provisioning a group of
> VMs.
> > > Although, this is something I think several Cloudstack users
> have
> > > requested before and we should propose a solution to this.
> > >
> > > Raj
> > >
> > > ________________________________
> > > From: Paul Angus <pa...@shapeblue.com>
> > > Sent: Monday, February 6, 2017 11:16:41 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: RE: [PROPOSAL] add native container orchestration
> > service
> > >
> > > #WhatHeSaid
> > >
> > > The title is misleading. The proposal is purely to add the
> > notion of
> > > Clusters of VMs to CloudStack. These may be for databases,
> > containers
> > > or anything else that needs 'clusters' of compute. Self-healing
> > and
> > > autoscaling are logical next steps to be added.
> > >
> > > Those guys at ShapeBlue have open-sourced their whole k8s
> > container
> > > service piece. If/when the 'cluster' part of that work is
> added
> > into
> > > CloudStack, the k8s specific pieces can be used by anyone who
> > wants
> > > to, alternatively they could be used for reference in order to
> > create
> > > another types of cluster. (or ignored completely).
> > >
> > >
> > >
> > >
> > > paul.angus@shapeblue.com
> > > www.shapeblue.com<http://www.shapeblue.com>
> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Will Stevens [mailto:williamstevens@gmail.com]
> > > Sent: 31 January 2017 13:26
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [PROPOSAL] add native container orchestration
> > service
> > >
> > > s/cloud-init/cloud-config/
> > >
> > > On Jan 31, 2017 7:24 AM, "Will Stevens" <
> > williamstevens@gmail.com> wrote:
> > >
> > > > I think that is covered in this proposal. There is nothing
> k8s
> > > > specific in this integration (from what I understand), all
> the
> > k8s
> > > > details are passed in via the cloud-init configuration after
> > the
> > > > cluster
> > > has been provisioned.
> > > >
> > > > On Jan 31, 2017 3:06 AM, "Lianghwa Jou"
> > > > <li...@accelerite.com>
> > > > wrote:
> > > >
> > > >
> > > > There are many container orchestrators. Those container
> > > > orchestrators are happy to run on any VMs or bare metal
> > machines.
> > > > K8s is just one of them and there will be more in the future.
> > It may
> > > > not be a good idea to make CloudStack to be k8s aware. IMO,
> the
> > > > relationship between k8s and cloudstack should be similar to
> > > > application and os. It probably not a good idea to make your
> > OS to
> > > > be aware of any specific applications so IMHO I don’t think
> > k8s should be native to CloudStack.
> > > > It makes more sense to provide some generic services that
> many
> > > > applications can take advantages of. Some generic resource
> > grouping
> > > > service makes sense so a group of VMs, baremetal machines or
> > network
> > > > can be treated as a single entity. Some life cycle management
> > will
> > > > be necessary for these entities too. We can deploy k8s,
> swarm,
> > dcos
> > > > or even non-container specific services on top of CloudStack.
> > The
> > > > k8s is changing fast. One single tenant installation may need
> > more
> > > > than one VM group and may actually requires more (k8s
> > federation).
> > > > It will be a struggle to be in sync if we try to bring k8s
> > specific
> > > > knowledge into
> > > cloudstack.
> > > >
> > > > Regards,
> > > >
> > > >
> > > > --
> > > > Lianghwa Jou
> > > > VP Engineering, Accelerite
> > > > email: lianghwa.jou@accelerite.com
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 1/29/17, 11:54 PM, "Murali Reddy" <
> > murali.reddy@shapeblue.com> wrote:
> > > >
> > > >
> > > > I agree with some good views Will has shared and I also
> > agree to
> > > > the concerns raised by Wido and Eric. IMO we need balance of
> > staying
> > > > relevant/add new features vs stability of CloudStack and take
> > > > corrective action if needed. We have two good examples for
> > both.
> > > > When SDN was hot technology CloudStack ended up with bunch of
> > SDN
> > > > controller
> > > integrations.
> > > > Few years later, now CloudStack is carrying baggage with no
> > > > maintainers for those plugins, with probably not many of
> > CloudStack
> > > users needing overlays.
> > > > On the other hand, other than attempt to liaison with ETSI
> for
> > NFV
> > > > no effort was done. OpenStack has become de-facto for NFV.
> Now
> > for
> > > > OpenStack, arguably NFV has become larger use case than cloud
> > it self.
> > > > I concur with Will’s point that with minimal viable solution
> > that
> > > > does not change the core of CloudStack, and can keep
> CloudStack
> > > > relevant is worth to be taken in.
> > > >
> > > > Will,
> > > >
> > > > To your question of how different is from ShapeBlue’s
> > container
> > > > service, its design, implementation and API semantics etc
> > remain same.
> > > > ShapeBlue’s container service was true drop in plug-in to
> > > > CloudStack, with this proposal I am trying to re-work to make
> > it a
> > > > core CloudStack pluggable service which is part of
> CloudStack.
> > > >
> > > > Key concepts that this proposal is trying to add
> > > >
> > > > - add notion of ‘container cluster’ as a first class
> > entity
> > > > in CloudStack. Which is bacially collection of other
> CloudStack
> > > > resources (like VM’s, networks, public ip, network rules etc)
> > > > - life cycle operation to manage ‘container cluster’
> > like
> > > > create, delete, start, stop, scale-up, scale-down, heal etc
> > > > - orchestrate container orchestrator control plane on
> > top of
> > > > provisioned resources
> > > >
> > > > At-least for k8s (which is what this proposal is
> > targeting),
> > > > integration with k8s is bare minimum. There are cloud-config
> > scripts
> > > > that automatically setup k8s cluster master and node VM’s.
> All
> > > > CloudStack is doing in passing the cloud-config to the core
> OS
> > VM’s
> > > > as
> > > user data.
> > > >
> > > > Regards
> > > > Murali Reddy
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 29/01/17, 8:54 AM, "Will Stevens" <
> > williamstevens@gmail.com
> > > > on behalf of wstevens@cloudops.com> wrote:
> > > >
> > > > >I agree that we need to be careful what we take on and
> > own inside
> > > > >CloudStack. I feel like some of the plugins or
> > integrations
> > > > which we have
> > > > >been "maintaining" may serve us better to abandon, but I
> > feel
> > > > like that is
> > > > >a whole discussion on its own.
> > > > >
> > > > >In this case, I feel like there is a minimum viable
> > solution
> > > > which puts
> > > > >CloudStack in a pretty good place to enable container
> > orchestration.
> > > > For
> > > > >example, one of the biggest challenges with K8S is the
> > fact
> > > > that it
> > > is
> > > > >single tenant. CloudStack has good multi tenancy
> support
> > and
> > > > has
> > > the
> > > > >ability to orchestrate the underlying infra quite well.
> > We
> > > > will have to be
> > > > >very careful not to try to own too deep into the K8S
> world
> > > > though, in my
> > > > >opinion. We only want to be responsible for providing
> the
> > > > infra (and a way
> > > > >to bootstrap K8S ideally) and be able to scale the
> infra,
> > > > everything else
> > > > >should be owned by the K8S on top. That is the way I
> see
> > it
> > > > anyway, but
> > > > >please add your input.
> > > > >
> > > > >I think it is a liability to try to go too deep, for the
> > same
> > > > reasons Wido
> > > > >and Erik have mentioned. But I also think we need to
> > take it
> > > > seriously
> > > > >because that train is moving and this may be a good
> > opportunity
> > > > to stay
> > > > >relevant in a rapidly changing market.
> > > > >
> > > > >*Will STEVENS*
> > > > >Lead Developer
> > > > >
> > > > ><https://goo.gl/NYZ8KK>
> > > > >
> > > > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > > > <wi...@widodh.nl>
> > > > wrote:
> > > > >
> > > > >>
> > > > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > > > wstevens@cloudops.com
> > > > >> >:
> > > > >> >
> > > > >> >
> > > > >> > Hey Murali,
> > > > >> > How different is this proposal than what ShapeBlue
> > already
> > > > built. It
> > > > >> looks
> > > > >> > pretty consistent with the functionality that you
> > guys open
> > > > sourced in
> > > > >> > Seville.
> > > > >> >
> > > > >> > I have not yet used this functionality, but I have
> > reports
> > > > that it works
> > > > >> > quite well.
> > > > >> >
> > > > >> > I believe the premise here is to only orchestrate
> the
> > VM
> > > > layer
> > > and
> > > > >> > basically expose a "group" of running VMs to the
> > user. The
> > > > user is
> > > > >> > responsible for configuring K8S or whatever other
> > container
> > > > orchestrator
> > > > >> on
> > > > >> > top. I saw mention of the "cloud-config" scripts in
> > the
> > > > FS, how are
> > > > >> those
> > > > >> > exposed to the cluster? Maybe the FS can expand on
> > that a bit?
> > > > >> >
> > > > >> > I believe the core feature that is being requested
> to
> > be
> > > > added is the
> > > > >> > ability to create a group of VMs which will be kept
> > active
> > > > as a group if
> > > > >> at
> > > > >> > all possible. ACS would be responsible for making
> > sure
> > > > that the number
> > > > >> of
> > > > >> > VMs specified for the group are in running state and
> > it
> > > > would spin up new
> > > > >> > VMs as needed in order to satisfy the group
> > settings. In
> > > > general, it is
> > > > >> > understood that any application running on this
> group
> > would
> > > > have to be
> > > > >> > fault tolerant enough to be able to rediscover a new
> > VM if
> > > > one fails and
> > > > >> is
> > > > >> > replaced by a fresh copy. Is that fair to say? How
> > is it
> > > > expected that
> > > > >> > this service discovery is done, just by VMs being
> > present
> > > > on the network?
> > > > >> >
> > > > >> > As for some of the other people's concerns in this
> > thread.
> > > > >> >
> > > > >> > - Regarding Wido's remarks. I understand that there
> > is
> > > > some
> > > added
> > > > >> > complexity, but I don't feel like the scope of the
> > addition is
> > > > >> > unrealistic. I think the LXC integration was a lot
> > farther
> > > > out of the
> > > > >> > scope of what ACS does then this is. This does not
> > change
> > > > the "things"
> > > > >> > which ACS orchestrates, it just adds the concept of
> a
> > > > grouping of things
> > > > >> > which ACS already manages. I think this is the
> right
> > > > approach since it
> > > > >> is
> > > > >> > not trying to be a container orchestrator. We will
> > never
> > > > compete with
> > > > >> K8S,
> > > > >> > for example, and we should not try, but K8S is here
> > and the
> > > > market wants
> > > > >> > it. I do think we should be keeping our head up
> > about that
> > > > fact because
> > > > >> > being able to provide a the underlay for K8S is very
> > > > valuable in the
> > > > >> > current marketplace. I see this functionality as a
> > way to
> > > > enable K8S
> > > > >> > adoption on top of ACS without changing our core
> > values.
> > > > >> >
> > > > >> > - Regarding Erik's remarks. The container space is
> > moving
> > > > fast, but so
> > > > >> is
> > > > >> > the industry. If we want to remain relevant, we
> need
> > to be
> > > > able to
> > > > >> adapt a
> > > > >> > bit. I don't think this is a big shift in what we
> > do, but
> > > > it is one that
> > > > >> > enables people to be able to start running with
> > something
> > > > like K8S on top
> > > > >> > of their existing ACS. This is something we are
> > interested
> > > > in doing and
> > > > >> so
> > > > >> > are our customers. If we can have a thin layer in
> ACS
> > > > which helps enable
> > > > >> > the use of K8S (or other container orchestrators) by
> > > orchestrating
> > > > >> > infrastructure, as we already do, and making it
> > easier to
> > > > adopt
> > > a
> > > > >> container
> > > > >> > orchestrator running on top of ACS, I think that
> > gives us a
> > > > nice foothold
> > > > >> > in the market. I don't really feel it is fair to
> > compare
> > > > containers to
> > > > >> > IPv6. IPv6 has been out forever and it has taken
> > almost a
> > > > decade to get
> > > > >> > anyone to adopt it. Containers have really only
> been
> > here
> > > > for like 2
> > > > >> years
> > > > >> > and they are changing the market landscape in a very
> > real way.
> > > > >> >
> > > > >> > Kind of on topic and kind of off topic. I think
> > > > understanding
> > > our
> > > > >> approach
> > > > >> > to containers is going to be important for the ACS
> > > > community as a whole.
> > > > >> > If we don't offer that market anything, then we will
> > not be
> > > > considered
> > > > >> and
> > > > >> > we will lose market share we can't afford to lose.
> > If we
> > > > try to hitch
> > > > >> our
> > > > >> > horse to that cart too much, we will not be able to
> be
> > > > agile enough and
> > > > >> > will fail. I feel like the right approach is for us
> > to
> > > > know that it is a
> > > > >> > thriving market and continue to do what we do, but
> to
> > > > extend an olive
> > > > >> > branch to that market. I think this sort of
> > implementation
> > > > is the right
> > > > >> > approach because we are not trying to do too much.
> > We are
> > > > simply giving
> > > > >> a
> > > > >> > foundation on which the next big thing in the
> > container
> > > > orchestration
> > > > >> world
> > > > >> > can adopt without us having to compete directly in
> > that
> > > > space. I think
> > > > >> we
> > > > >> > have to focus on what we do best, but at the same
> > time,
> > > > think about what
> > > > >> we
> > > > >> > can do to enable that huge market of users to adopt
> > ACS as their
> > > > >> > foundation. The ability to offer VMs and containers
> > in the
> > > > same data
> > > > >> plane
> > > > >> > is something we have the ability to do, especially
> > with
> > > > this approach,
> > > > >> and
> > > > >> > is something that most other softwares can not do.
> > The
> > > > adoption of
> > > > >> > containers by bigger organizations will be only part
> > of
> > > > their workload,
> > > > >> > they will still be running VMs for the foreseeable
> > future.
> > > > Being able to
> > > > >> > appeal to that market is going to be important for
> us.
> > > > >> >
> > > > >> > Hopefully I don't have too many strong opinions
> here,
> > but I
> > > > do think we
> > > > >> > need to be thinking about how we move forward in a
> > world
> > > > which
> > > is
> > > > >> adopting
> > > > >> > containers in a very real way.
> > > > >> >
> > > > >>
> > > > >> Understood. I just want to prevent that we add more
> > features
> > > > to CloudStack
> > > > >> which are poorly maintained. Not judging Murali here
> at
> > all,
> > > > but I'd rather
> > > > >> see CloudStack loose code then have it added.
> > > > >>
> > > > >> Thinking about LXC, would like to see that removed
> > together
> > > > with various
> > > > >> other network plugins which I think are rarely used.
> > > > >>
> > > > >> Now, the idea of Murali was misunderstood by me. I
> > think it
> > > > would be worth
> > > > >> more if we would improve our cloud-init support and
> > > > integration in various
> > > > >> tools which makes it much easier to deploy VMs running
> > > > containers
> > > ON
> > > > >> CloudStack.
> > > > >>
> > > > >> Not so much changing CloudStack code, but rather
> tooling
> > > > around
> > > it.
> > > > >>
> > > > >> If we have CloudStack talking to Kubernetes we
> suddenly
> > have
> > > > to maintain
> > > > >> that API integration. Who's going to do that.
> > Realistically.
> > > > >>
> > > > >> That's my main worry in this regard.
> > > > >>
> > > > >> We have so much more work to do in ACS in total that I
> > don't
> > > > know if this
> > > > >> is the realistic route. I talk to many people who are
> > not
> > > > looking
> > > at
> > > > >> containers and are still working with VMs.
> > > > >>
> > > > >> There is not a single truth which is true, it really
> > depends
> > > > on who you
> > > > >> ask.
> > > > >>
> > > > >> Wido
> > > > >>
> > > > >> > Cheers,
> > > > >> >
> > > > >> > Will
> > > > >> >
> > > > >> > *Will STEVENS*
> > > > >> > Lead Developer
> > > > >> >
> > > > >> > <https://goo.gl/NYZ8KK>
> > > > >> >
> > > > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > > > <te...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > > > muralimmreddy@gmail.com
> > > > >> >
> > > > >> > > wrote:
> > > > >> > > > All,
> > > > >> > > >
> > > > >> > > > I would like propose native functionality into
> > > > CloudStack to provide
> > > > >> a
> > > > >> > > container service through which users out-of-the
> > box can
> > > > use to launch
> > > > >> > > container based application. Idea is to support
> > ability
> > > > to orchestrate
> > > > >> the
> > > > >> > > resources and automate aspects of setting up
> > container
> > > > orchestrator
> > > > >> through
> > > > >> > > CloudStack. Public IAAS service providers AWS with
> > its
> > > > ECS [1] and
> > > > >> google
> > > > >> > > with GKE [2] already provides ability container
> > applications.
> > > > >> Competitive
> > > > >> > > cloud orchestration platforms already have native
> > support
> > > > for container
> > > > >> > > service. Users of CloudStack both as public cloud
> > > > providers and users
> > > > >> with
> > > > >> > > private clouds will benefit with such
> functionality.
> > > > >> > > >
> > > > >> > > > While container orchestrator of user choice can
> be
> > > > provisioned on
> > > > >> top of
> > > > >> > > CloudStack (with out CloudStack being involved)
> with
> > > > tools
> > > like
> > > > >> > > TerraForm[3], Ansible[4] etc, advantage of having
> > native
> > > > orchestration
> > > > >> is
> > > > >> > > giving user a nice cohesive integration. This
> > proposal
> > > > would like add a
> > > > >> > > notion of first class CloudStack entity called
> > container
> > > > cluster which
> > > > >> can
> > > > >> > > be used to provision resources, scale up, scale
> > down,
> > > > start and stop
> > > > >> the
> > > > >> > > cluster of VM’s on which containerised
> applications
> > can
> > > > be
> > > run.
> > > > For
> > > > >> actual
> > > > >> > > container orchestration we will still need
> container
> > > > orchestrator like
> > > > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > > > container service
> > > > >> can
> > > > >> > > automate setting up of control place
> automatically.
> > > > >> > > >
> > > > >> > >
> > > > >> > > To be honest I'm torn on this one.
> > > > >> > >
> > > > >> > > Containers are a rapid changing thing, and while
> > docker swam,
> > > > >> > > kubernetes, rancher or whatnot is popular today,
> > they
> > > > might not be
> > > > >> > > tomorrow.
> > > > >> > > They might use CoreOS today, but might not
> tomorrow.
> > > > >> > >
> > > > >> > > We have a rather poor track record of staying up
> to
> > date
> > > > with new
> > > > >> > > features/versions, and adding a feature that is so
> > > > rapidly changing
> > > > >> > > is, I fear, going to be hard to maintain.
> > > > >> > > Want an example, look at xenserver. It is one of
> > the most used
> > > > >> > > hypervisors we support, yet it took 6 months or so
> > for us
> > > > to support
> > > > >> > > the latest release.
> > > > >> > > Or IPv6...
> > > > >> > >
> > > > >> > > I don't mean to bash at maintainers/implementers
> of
> > those
> > > > features, I
> > > > >> > > appreciate all the work being done in every
> aspect,
> > but I
> > > > believe we
> > > > >> > > should be realistic and realize that we have
> issues
> > with
> > > > keeping stuff
> > > > >> > > up to date.
> > > > >> > >
> > > > >> > > I'd say focus on making sure other tools can do
> > their job
> > > > well against
> > > > >> > > CloudStack (kops, rancher, ++), but that does not
> > mean I
> > > > will
> > > > -1 the
> > > > >> > > idea if anyone really wants to go through with it.
> > > > >> > >
> > > > >> > > --
> > > > >> > > Erik
> > > > >> > >
> > > > >>
> > > >
> > > > murali.reddy@shapeblue.com
> > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > DISCLAIMER
> > > > ==========
> > > > This e-mail may contain privileged and confidential
> information
> > > > which is the property of Accelerite, a Persistent Systems
> > business.
> > > > It is intended only for the use of the individual or entity
> to
> > which
> > > > it is addressed. If you are not the intended recipient, you
> > are not
> > > > authorized to read, retain, copy, print, distribute or use
> this
> > > > message. If you have received this communication in error,
> > please
> > > > notify the sender and delete all copies of this message.
> > Accelerite,
> > > > a Persistent Systems business does not accept any liability
> for
> > > > virus
> > > infected mails.
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information
> > which
> > > is the property of Accelerite, a Persistent Systems business.
> It
> > is
> > > intended only for the use of the individual or entity to which
> > it is
> > > addressed. If you are not the intended recipient, you are not
> > > authorized to read, retain, copy, print, distribute or use this
> > > message. If you have received this communication in error,
> please
> > > notify the sender and delete all copies of this message.
> > Accelerite, a
> > > Persistent Systems business does not accept any liability for
> > virus infected mails.
> > >
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> > which is the property of Accelerite, a Persistent Systems business. It is
> > intended only for the use of the individual or entity to which it is
> > addressed. If you are not the intended recipient, you are not authorized
> to
> > read, retain, copy, print, distribute or use this message. If you have
> > received this communication in error, please notify the sender and delete
> > all copies of this message. Accelerite, a Persistent Systems business
> does
> > not accept any liability for virus infected mails.
> >
> >
> >
> > daan.hoogland@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands
> > @shapeblue
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which
> > is the property of Accelerite, a Persistent Systems business. It is
> > intended only for the use of the individual or entity to which it is
> > addressed. If you are not the intended recipient, you are not authorized
> to
> > read, retain, copy, print, distribute or use this message. If you have
> > received this communication in error, please notify the sender and delete
> > all copies of this message. Accelerite, a Persistent Systems business
> does
> > not accept any liability for virus infected mails.
> >
> >
> >
> > daan.hoogland@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
Re: [PROPOSAL] add native vm-cluster orchestration service
(was: [PROPOSAL] add native container orchestration service)
Posted by Daan Hoogland <da...@shapeblue.com>.
Syed, Kilham, et. al.
After some thought I came up with ApplicationCluster. It is not purely machines or instances but includes network and storage resources and maybe more. Next to that these are meant for running application like k8, mesos or DBaaS. I don't like service as prefix because it implies the cluster is for servicing the cloud or VMs that may be broken or need a kind of extra feature while from user perspective they are an addition, hence application to cloudstack.
Any push back?
Sent from Nine<http://www.9folders.com/>
________________________________
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
From: Daan Hoogland
Sent: 28 Feb 2017 6:49 pm
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native vm-cluster orchestration service (was: [PROPOSAL] add native container orchestration service)
Syed, I chose machine as they might be bare metal in some cases.
Sent from Nine<http://www.9folders.com/>
________________________________
From: Syed Ahmed <sa...@cloudops.com>
Sent: 28 Feb 2017 4:22 pm
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native vm-cluster orchestration service (was: [PROPOSAL] add native container orchestration service)
We already call the VMs as Instances. So, InstanceCluster would be a better
name imo.
On Tue, Feb 28, 2017 at 10:05 AM, Daan Hoogland <daan.hoogland@shapeblue.com
> wrote:
> Kishan, I see some sensible additions but also some unnecessary omissions.
> Most of it seems to be Murali’s text so I’ll c&p your improvements back and
> rename the page to the more sensible title of “MachineCluster service” and
> delete the other.
>
> About naming, I was thinking MachineCluster instead of ServiceCluster,
> makes sense? Or even GuestMachineCluster. ServiceCluster could mean a
> supporting cluster that delivers e.g. backup as a service to guests, or
> maybe some build, artifact or networking service. For this ambiguity im am
> :-1: on the name ServiceCluster.
>
>
> On 28/02/17 11:16, "Kishan Kavala" <ki...@accelerite.com> wrote:
>
> Daan,
> I've updated the earlier spec to support any Vm cluster. Please let
> me know your thoughts on this.
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> Service+Cluster+Functional+Specification
>
> regards,
> Kishan
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
> Sent: 27 February 2017 04:02 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native vm-cluster orchestration service
> (was: [PROPOSAL] add native container orchestration service)
>
> Any follow up Koushik? I want to refactor our proof of concept and
> integrate it in master.
>
> On 21/02/17 10:42, "Kishan Kavala" <ki...@accelerite.com>
> wrote:
>
> Sure Daan. I'll publish the design on cwiki and share the link.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
> Sent: Monday, February 20, 2017 7:27 PM
> To: dev@cloudstack.apache.org
> Subject: [PROPOSAL] add native vm-cluster orchestration service
> (was: [PROPOSAL] add native container orchestration service)
>
> So, being very late in the discussion but havingread the whole
> thread before editting the title of this thread,
>
> Can we agree that we want a generic vm-cluster service and leave
> the container bits to containers? Kishan can you share your design?
> Shapeblue wants to rebase their k8 service on top of this and I would like
> yours and Murali's work to not conflict.
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531
> VENetherlands @shapeblue
>
>
>
>
> -----Original Message-----
> From: Paul Angus [mailto:paul.angus@shapeblue.com]
> Sent: dinsdag 7 februari 2017 08:14
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> Will is 100% correct. As I mentioned the Title is misleading.
> However, Murali did clarify in his explanation; this is really about vm
> cluster orchestration.
>
>
>
> ________________________________
> From: Will Stevens <ws...@cloudops.com>
> Sent: 6 Feb 2017 22:54
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> My understanding is that what Paul is talking about is what is
> already built and IS what the thread is talking about.
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani <
> rajesh.ramchandani@accelerite.com> wrote:
>
> > Hi Paul - I think this is different from what the thread was
> about.
> > The conversation was specifically about adding support for
> container
> > orchestrators. You are talking about provisioning a group of VMs.
> > Although, this is something I think several Cloudstack users have
> > requested before and we should propose a solution to this.
> >
> > Raj
> >
> > ________________________________
> > From: Paul Angus <pa...@shapeblue.com>
> > Sent: Monday, February 6, 2017 11:16:41 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: [PROPOSAL] add native container orchestration
> service
> >
> > #WhatHeSaid
> >
> > The title is misleading. The proposal is purely to add the
> notion of
> > Clusters of VMs to CloudStack. These may be for databases,
> containers
> > or anything else that needs 'clusters' of compute. Self-healing
> and
> > autoscaling are logical next steps to be added.
> >
> > Those guys at ShapeBlue have open-sourced their whole k8s
> container
> > service piece. If/when the 'cluster' part of that work is added
> into
> > CloudStack, the k8s specific pieces can be used by anyone who
> wants
> > to, alternatively they could be used for reference in order to
> create
> > another types of cluster. (or ignored completely).
> >
> >
> >
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Will Stevens [mailto:williamstevens@gmail.com]
> > Sent: 31 January 2017 13:26
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] add native container orchestration
> service
> >
> > s/cloud-init/cloud-config/
> >
> > On Jan 31, 2017 7:24 AM, "Will Stevens" <
> williamstevens@gmail.com> wrote:
> >
> > > I think that is covered in this proposal. There is nothing k8s
> > > specific in this integration (from what I understand), all the
> k8s
> > > details are passed in via the cloud-init configuration after
> the
> > > cluster
> > has been provisioned.
> > >
> > > On Jan 31, 2017 3:06 AM, "Lianghwa Jou"
> > > <li...@accelerite.com>
> > > wrote:
> > >
> > >
> > > There are many container orchestrators. Those container
> > > orchestrators are happy to run on any VMs or bare metal
> machines.
> > > K8s is just one of them and there will be more in the future.
> It may
> > > not be a good idea to make CloudStack to be k8s aware. IMO, the
> > > relationship between k8s and cloudstack should be similar to
> > > application and os. It probably not a good idea to make your
> OS to
> > > be aware of any specific applications so IMHO I don’t think
> k8s should be native to CloudStack.
> > > It makes more sense to provide some generic services that many
> > > applications can take advantages of. Some generic resource
> grouping
> > > service makes sense so a group of VMs, baremetal machines or
> network
> > > can be treated as a single entity. Some life cycle management
> will
> > > be necessary for these entities too. We can deploy k8s, swarm,
> dcos
> > > or even non-container specific services on top of CloudStack.
> The
> > > k8s is changing fast. One single tenant installation may need
> more
> > > than one VM group and may actually requires more (k8s
> federation).
> > > It will be a struggle to be in sync if we try to bring k8s
> specific
> > > knowledge into
> > cloudstack.
> > >
> > > Regards,
> > >
> > >
> > > --
> > > Lianghwa Jou
> > > VP Engineering, Accelerite
> > > email: lianghwa.jou@accelerite.com
> > >
> > >
> > >
> > >
> > >
> > > On 1/29/17, 11:54 PM, "Murali Reddy" <
> murali.reddy@shapeblue.com> wrote:
> > >
> > >
> > > I agree with some good views Will has shared and I also
> agree to
> > > the concerns raised by Wido and Eric. IMO we need balance of
> staying
> > > relevant/add new features vs stability of CloudStack and take
> > > corrective action if needed. We have two good examples for
> both.
> > > When SDN was hot technology CloudStack ended up with bunch of
> SDN
> > > controller
> > integrations.
> > > Few years later, now CloudStack is carrying baggage with no
> > > maintainers for those plugins, with probably not many of
> CloudStack
> > users needing overlays.
> > > On the other hand, other than attempt to liaison with ETSI for
> NFV
> > > no effort was done. OpenStack has become de-facto for NFV. Now
> for
> > > OpenStack, arguably NFV has become larger use case than cloud
> it self.
> > > I concur with Will’s point that with minimal viable solution
> that
> > > does not change the core of CloudStack, and can keep CloudStack
> > > relevant is worth to be taken in.
> > >
> > > Will,
> > >
> > > To your question of how different is from ShapeBlue’s
> container
> > > service, its design, implementation and API semantics etc
> remain same.
> > > ShapeBlue’s container service was true drop in plug-in to
> > > CloudStack, with this proposal I am trying to re-work to make
> it a
> > > core CloudStack pluggable service which is part of CloudStack.
> > >
> > > Key concepts that this proposal is trying to add
> > >
> > > - add notion of ‘container cluster’ as a first class
> entity
> > > in CloudStack. Which is bacially collection of other CloudStack
> > > resources (like VM’s, networks, public ip, network rules etc)
> > > - life cycle operation to manage ‘container cluster’
> like
> > > create, delete, start, stop, scale-up, scale-down, heal etc
> > > - orchestrate container orchestrator control plane on
> top of
> > > provisioned resources
> > >
> > > At-least for k8s (which is what this proposal is
> targeting),
> > > integration with k8s is bare minimum. There are cloud-config
> scripts
> > > that automatically setup k8s cluster master and node VM’s. All
> > > CloudStack is doing in passing the cloud-config to the core OS
> VM’s
> > > as
> > user data.
> > >
> > > Regards
> > > Murali Reddy
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 29/01/17, 8:54 AM, "Will Stevens" <
> williamstevens@gmail.com
> > > on behalf of wstevens@cloudops.com> wrote:
> > >
> > > >I agree that we need to be careful what we take on and
> own inside
> > > >CloudStack. I feel like some of the plugins or
> integrations
> > > which we have
> > > >been "maintaining" may serve us better to abandon, but I
> feel
> > > like that is
> > > >a whole discussion on its own.
> > > >
> > > >In this case, I feel like there is a minimum viable
> solution
> > > which puts
> > > >CloudStack in a pretty good place to enable container
> orchestration.
> > > For
> > > >example, one of the biggest challenges with K8S is the
> fact
> > > that it
> > is
> > > >single tenant. CloudStack has good multi tenancy support
> and
> > > has
> > the
> > > >ability to orchestrate the underlying infra quite well.
> We
> > > will have to be
> > > >very careful not to try to own too deep into the K8S world
> > > though, in my
> > > >opinion. We only want to be responsible for providing the
> > > infra (and a way
> > > >to bootstrap K8S ideally) and be able to scale the infra,
> > > everything else
> > > >should be owned by the K8S on top. That is the way I see
> it
> > > anyway, but
> > > >please add your input.
> > > >
> > > >I think it is a liability to try to go too deep, for the
> same
> > > reasons Wido
> > > >and Erik have mentioned. But I also think we need to
> take it
> > > seriously
> > > >because that train is moving and this may be a good
> opportunity
> > > to stay
> > > >relevant in a rapidly changing market.
> > > >
> > > >*Will STEVENS*
> > > >Lead Developer
> > > >
> > > ><https://goo.gl/NYZ8KK>
> > > >
> > > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > > <wi...@widodh.nl>
> > > wrote:
> > > >
> > > >>
> > > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > > wstevens@cloudops.com
> > > >> >:
> > > >> >
> > > >> >
> > > >> > Hey Murali,
> > > >> > How different is this proposal than what ShapeBlue
> already
> > > built. It
> > > >> looks
> > > >> > pretty consistent with the functionality that you
> guys open
> > > sourced in
> > > >> > Seville.
> > > >> >
> > > >> > I have not yet used this functionality, but I have
> reports
> > > that it works
> > > >> > quite well.
> > > >> >
> > > >> > I believe the premise here is to only orchestrate the
> VM
> > > layer
> > and
> > > >> > basically expose a "group" of running VMs to the
> user. The
> > > user is
> > > >> > responsible for configuring K8S or whatever other
> container
> > > orchestrator
> > > >> on
> > > >> > top. I saw mention of the "cloud-config" scripts in
> the
> > > FS, how are
> > > >> those
> > > >> > exposed to the cluster? Maybe the FS can expand on
> that a bit?
> > > >> >
> > > >> > I believe the core feature that is being requested to
> be
> > > added is the
> > > >> > ability to create a group of VMs which will be kept
> active
> > > as a group if
> > > >> at
> > > >> > all possible. ACS would be responsible for making
> sure
> > > that the number
> > > >> of
> > > >> > VMs specified for the group are in running state and
> it
> > > would spin up new
> > > >> > VMs as needed in order to satisfy the group
> settings. In
> > > general, it is
> > > >> > understood that any application running on this group
> would
> > > have to be
> > > >> > fault tolerant enough to be able to rediscover a new
> VM if
> > > one fails and
> > > >> is
> > > >> > replaced by a fresh copy. Is that fair to say? How
> is it
> > > expected that
> > > >> > this service discovery is done, just by VMs being
> present
> > > on the network?
> > > >> >
> > > >> > As for some of the other people's concerns in this
> thread.
> > > >> >
> > > >> > - Regarding Wido's remarks. I understand that there
> is
> > > some
> > added
> > > >> > complexity, but I don't feel like the scope of the
> addition is
> > > >> > unrealistic. I think the LXC integration was a lot
> farther
> > > out of the
> > > >> > scope of what ACS does then this is. This does not
> change
> > > the "things"
> > > >> > which ACS orchestrates, it just adds the concept of a
> > > grouping of things
> > > >> > which ACS already manages. I think this is the right
> > > approach since it
> > > >> is
> > > >> > not trying to be a container orchestrator. We will
> never
> > > compete with
> > > >> K8S,
> > > >> > for example, and we should not try, but K8S is here
> and the
> > > market wants
> > > >> > it. I do think we should be keeping our head up
> about that
> > > fact because
> > > >> > being able to provide a the underlay for K8S is very
> > > valuable in the
> > > >> > current marketplace. I see this functionality as a
> way to
> > > enable K8S
> > > >> > adoption on top of ACS without changing our core
> values.
> > > >> >
> > > >> > - Regarding Erik's remarks. The container space is
> moving
> > > fast, but so
> > > >> is
> > > >> > the industry. If we want to remain relevant, we need
> to be
> > > able to
> > > >> adapt a
> > > >> > bit. I don't think this is a big shift in what we
> do, but
> > > it is one that
> > > >> > enables people to be able to start running with
> something
> > > like K8S on top
> > > >> > of their existing ACS. This is something we are
> interested
> > > in doing and
> > > >> so
> > > >> > are our customers. If we can have a thin layer in ACS
> > > which helps enable
> > > >> > the use of K8S (or other container orchestrators) by
> > orchestrating
> > > >> > infrastructure, as we already do, and making it
> easier to
> > > adopt
> > a
> > > >> container
> > > >> > orchestrator running on top of ACS, I think that
> gives us a
> > > nice foothold
> > > >> > in the market. I don't really feel it is fair to
> compare
> > > containers to
> > > >> > IPv6. IPv6 has been out forever and it has taken
> almost a
> > > decade to get
> > > >> > anyone to adopt it. Containers have really only been
> here
> > > for like 2
> > > >> years
> > > >> > and they are changing the market landscape in a very
> real way.
> > > >> >
> > > >> > Kind of on topic and kind of off topic. I think
> > > understanding
> > our
> > > >> approach
> > > >> > to containers is going to be important for the ACS
> > > community as a whole.
> > > >> > If we don't offer that market anything, then we will
> not be
> > > considered
> > > >> and
> > > >> > we will lose market share we can't afford to lose.
> If we
> > > try to hitch
> > > >> our
> > > >> > horse to that cart too much, we will not be able to be
> > > agile enough and
> > > >> > will fail. I feel like the right approach is for us
> to
> > > know that it is a
> > > >> > thriving market and continue to do what we do, but to
> > > extend an olive
> > > >> > branch to that market. I think this sort of
> implementation
> > > is the right
> > > >> > approach because we are not trying to do too much.
> We are
> > > simply giving
> > > >> a
> > > >> > foundation on which the next big thing in the
> container
> > > orchestration
> > > >> world
> > > >> > can adopt without us having to compete directly in
> that
> > > space. I think
> > > >> we
> > > >> > have to focus on what we do best, but at the same
> time,
> > > think about what
> > > >> we
> > > >> > can do to enable that huge market of users to adopt
> ACS as their
> > > >> > foundation. The ability to offer VMs and containers
> in the
> > > same data
> > > >> plane
> > > >> > is something we have the ability to do, especially
> with
> > > this approach,
> > > >> and
> > > >> > is something that most other softwares can not do.
> The
> > > adoption of
> > > >> > containers by bigger organizations will be only part
> of
> > > their workload,
> > > >> > they will still be running VMs for the foreseeable
> future.
> > > Being able to
> > > >> > appeal to that market is going to be important for us.
> > > >> >
> > > >> > Hopefully I don't have too many strong opinions here,
> but I
> > > do think we
> > > >> > need to be thinking about how we move forward in a
> world
> > > which
> > is
> > > >> adopting
> > > >> > containers in a very real way.
> > > >> >
> > > >>
> > > >> Understood. I just want to prevent that we add more
> features
> > > to CloudStack
> > > >> which are poorly maintained. Not judging Murali here at
> all,
> > > but I'd rather
> > > >> see CloudStack loose code then have it added.
> > > >>
> > > >> Thinking about LXC, would like to see that removed
> together
> > > with various
> > > >> other network plugins which I think are rarely used.
> > > >>
> > > >> Now, the idea of Murali was misunderstood by me. I
> think it
> > > would be worth
> > > >> more if we would improve our cloud-init support and
> > > integration in various
> > > >> tools which makes it much easier to deploy VMs running
> > > containers
> > ON
> > > >> CloudStack.
> > > >>
> > > >> Not so much changing CloudStack code, but rather tooling
> > > around
> > it.
> > > >>
> > > >> If we have CloudStack talking to Kubernetes we suddenly
> have
> > > to maintain
> > > >> that API integration. Who's going to do that.
> Realistically.
> > > >>
> > > >> That's my main worry in this regard.
> > > >>
> > > >> We have so much more work to do in ACS in total that I
> don't
> > > know if this
> > > >> is the realistic route. I talk to many people who are
> not
> > > looking
> > at
> > > >> containers and are still working with VMs.
> > > >>
> > > >> There is not a single truth which is true, it really
> depends
> > > on who you
> > > >> ask.
> > > >>
> > > >> Wido
> > > >>
> > > >> > Cheers,
> > > >> >
> > > >> > Will
> > > >> >
> > > >> > *Will STEVENS*
> > > >> > Lead Developer
> > > >> >
> > > >> > <https://goo.gl/NYZ8KK>
> > > >> >
> > > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > > <te...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > > muralimmreddy@gmail.com
> > > >> >
> > > >> > > wrote:
> > > >> > > > All,
> > > >> > > >
> > > >> > > > I would like propose native functionality into
> > > CloudStack to provide
> > > >> a
> > > >> > > container service through which users out-of-the
> box can
> > > use to launch
> > > >> > > container based application. Idea is to support
> ability
> > > to orchestrate
> > > >> the
> > > >> > > resources and automate aspects of setting up
> container
> > > orchestrator
> > > >> through
> > > >> > > CloudStack. Public IAAS service providers AWS with
> its
> > > ECS [1] and
> > > >> google
> > > >> > > with GKE [2] already provides ability container
> applications.
> > > >> Competitive
> > > >> > > cloud orchestration platforms already have native
> support
> > > for container
> > > >> > > service. Users of CloudStack both as public cloud
> > > providers and users
> > > >> with
> > > >> > > private clouds will benefit with such functionality.
> > > >> > > >
> > > >> > > > While container orchestrator of user choice can be
> > > provisioned on
> > > >> top of
> > > >> > > CloudStack (with out CloudStack being involved) with
> > > tools
> > like
> > > >> > > TerraForm[3], Ansible[4] etc, advantage of having
> native
> > > orchestration
> > > >> is
> > > >> > > giving user a nice cohesive integration. This
> proposal
> > > would like add a
> > > >> > > notion of first class CloudStack entity called
> container
> > > cluster which
> > > >> can
> > > >> > > be used to provision resources, scale up, scale
> down,
> > > start and stop
> > > >> the
> > > >> > > cluster of VM’s on which containerised applications
> can
> > > be
> > run.
> > > For
> > > >> actual
> > > >> > > container orchestration we will still need container
> > > orchestrator like
> > > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > > container service
> > > >> can
> > > >> > > automate setting up of control place automatically.
> > > >> > > >
> > > >> > >
> > > >> > > To be honest I'm torn on this one.
> > > >> > >
> > > >> > > Containers are a rapid changing thing, and while
> docker swam,
> > > >> > > kubernetes, rancher or whatnot is popular today,
> they
> > > might not be
> > > >> > > tomorrow.
> > > >> > > They might use CoreOS today, but might not tomorrow.
> > > >> > >
> > > >> > > We have a rather poor track record of staying up to
> date
> > > with new
> > > >> > > features/versions, and adding a feature that is so
> > > rapidly changing
> > > >> > > is, I fear, going to be hard to maintain.
> > > >> > > Want an example, look at xenserver. It is one of
> the most used
> > > >> > > hypervisors we support, yet it took 6 months or so
> for us
> > > to support
> > > >> > > the latest release.
> > > >> > > Or IPv6...
> > > >> > >
> > > >> > > I don't mean to bash at maintainers/implementers of
> those
> > > features, I
> > > >> > > appreciate all the work being done in every aspect,
> but I
> > > believe we
> > > >> > > should be realistic and realize that we have issues
> with
> > > keeping stuff
> > > >> > > up to date.
> > > >> > >
> > > >> > > I'd say focus on making sure other tools can do
> their job
> > > well against
> > > >> > > CloudStack (kops, rancher, ++), but that does not
> mean I
> > > will
> > > -1 the
> > > >> > > idea if anyone really wants to go through with it.
> > > >> > >
> > > >> > > --
> > > >> > > Erik
> > > >> > >
> > > >>
> > >
> > > murali.reddy@shapeblue.com
> > > www.shapeblue.com<http://www.shapeblue.com>
> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information
> > > which is the property of Accelerite, a Persistent Systems
> business.
> > > It is intended only for the use of the individual or entity to
> which
> > > it is addressed. If you are not the intended recipient, you
> are not
> > > authorized to read, retain, copy, print, distribute or use this
> > > message. If you have received this communication in error,
> please
> > > notify the sender and delete all copies of this message.
> Accelerite,
> > > a Persistent Systems business does not accept any liability for
> > > virus
> > infected mails.
> > >
> > >
> > >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> which
> > is the property of Accelerite, a Persistent Systems business. It
> is
> > intended only for the use of the individual or entity to which
> it is
> > addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message.
> Accelerite, a
> > Persistent Systems business does not accept any liability for
> virus infected mails.
> >
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information
> which is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Accelerite, a Persistent Systems business does
> not accept any liability for virus infected mails.
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands
> @shapeblue
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Accelerite, a Persistent Systems business does
> not accept any liability for virus infected mails.
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
daan.hoogland@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: [PROPOSAL] add native vm-cluster orchestration service
(was: [PROPOSAL] add native container orchestration service)
Posted by Daan Hoogland <da...@shapeblue.com>.
Syed, I chose machine as they might be bare metal in some cases.
Sent from Nine<http://www.9folders.com/>
________________________________
From: Syed Ahmed <sa...@cloudops.com>
Sent: 28 Feb 2017 4:22 pm
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native vm-cluster orchestration service (was: [PROPOSAL] add native container orchestration service)
We already call the VMs as Instances. So, InstanceCluster would be a better
name imo.
On Tue, Feb 28, 2017 at 10:05 AM, Daan Hoogland <daan.hoogland@shapeblue.com
> wrote:
> Kishan, I see some sensible additions but also some unnecessary omissions.
> Most of it seems to be Murali’s text so I’ll c&p your improvements back and
> rename the page to the more sensible title of “MachineCluster service” and
> delete the other.
>
> About naming, I was thinking MachineCluster instead of ServiceCluster,
> makes sense? Or even GuestMachineCluster. ServiceCluster could mean a
> supporting cluster that delivers e.g. backup as a service to guests, or
> maybe some build, artifact or networking service. For this ambiguity im am
> :-1: on the name ServiceCluster.
>
>
> On 28/02/17 11:16, "Kishan Kavala" <ki...@accelerite.com> wrote:
>
> Daan,
> I've updated the earlier spec to support any Vm cluster. Please let
> me know your thoughts on this.
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> Service+Cluster+Functional+Specification
>
> regards,
> Kishan
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
> Sent: 27 February 2017 04:02 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native vm-cluster orchestration service
> (was: [PROPOSAL] add native container orchestration service)
>
> Any follow up Koushik? I want to refactor our proof of concept and
> integrate it in master.
>
> On 21/02/17 10:42, "Kishan Kavala" <ki...@accelerite.com>
> wrote:
>
> Sure Daan. I'll publish the design on cwiki and share the link.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
> Sent: Monday, February 20, 2017 7:27 PM
> To: dev@cloudstack.apache.org
> Subject: [PROPOSAL] add native vm-cluster orchestration service
> (was: [PROPOSAL] add native container orchestration service)
>
> So, being very late in the discussion but havingread the whole
> thread before editting the title of this thread,
>
> Can we agree that we want a generic vm-cluster service and leave
> the container bits to containers? Kishan can you share your design?
> Shapeblue wants to rebase their k8 service on top of this and I would like
> yours and Murali's work to not conflict.
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531
> VENetherlands @shapeblue
>
>
>
>
> -----Original Message-----
> From: Paul Angus [mailto:paul.angus@shapeblue.com]
> Sent: dinsdag 7 februari 2017 08:14
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> Will is 100% correct. As I mentioned the Title is misleading.
> However, Murali did clarify in his explanation; this is really about vm
> cluster orchestration.
>
>
>
> ________________________________
> From: Will Stevens <ws...@cloudops.com>
> Sent: 6 Feb 2017 22:54
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> My understanding is that what Paul is talking about is what is
> already built and IS what the thread is talking about.
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani <
> rajesh.ramchandani@accelerite.com> wrote:
>
> > Hi Paul - I think this is different from what the thread was
> about.
> > The conversation was specifically about adding support for
> container
> > orchestrators. You are talking about provisioning a group of VMs.
> > Although, this is something I think several Cloudstack users have
> > requested before and we should propose a solution to this.
> >
> > Raj
> >
> > ________________________________
> > From: Paul Angus <pa...@shapeblue.com>
> > Sent: Monday, February 6, 2017 11:16:41 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: [PROPOSAL] add native container orchestration
> service
> >
> > #WhatHeSaid
> >
> > The title is misleading. The proposal is purely to add the
> notion of
> > Clusters of VMs to CloudStack. These may be for databases,
> containers
> > or anything else that needs 'clusters' of compute. Self-healing
> and
> > autoscaling are logical next steps to be added.
> >
> > Those guys at ShapeBlue have open-sourced their whole k8s
> container
> > service piece. If/when the 'cluster' part of that work is added
> into
> > CloudStack, the k8s specific pieces can be used by anyone who
> wants
> > to, alternatively they could be used for reference in order to
> create
> > another types of cluster. (or ignored completely).
> >
> >
> >
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Will Stevens [mailto:williamstevens@gmail.com]
> > Sent: 31 January 2017 13:26
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] add native container orchestration
> service
> >
> > s/cloud-init/cloud-config/
> >
> > On Jan 31, 2017 7:24 AM, "Will Stevens" <
> williamstevens@gmail.com> wrote:
> >
> > > I think that is covered in this proposal. There is nothing k8s
> > > specific in this integration (from what I understand), all the
> k8s
> > > details are passed in via the cloud-init configuration after
> the
> > > cluster
> > has been provisioned.
> > >
> > > On Jan 31, 2017 3:06 AM, "Lianghwa Jou"
> > > <li...@accelerite.com>
> > > wrote:
> > >
> > >
> > > There are many container orchestrators. Those container
> > > orchestrators are happy to run on any VMs or bare metal
> machines.
> > > K8s is just one of them and there will be more in the future.
> It may
> > > not be a good idea to make CloudStack to be k8s aware. IMO, the
> > > relationship between k8s and cloudstack should be similar to
> > > application and os. It probably not a good idea to make your
> OS to
> > > be aware of any specific applications so IMHO I don’t think
> k8s should be native to CloudStack.
> > > It makes more sense to provide some generic services that many
> > > applications can take advantages of. Some generic resource
> grouping
> > > service makes sense so a group of VMs, baremetal machines or
> network
> > > can be treated as a single entity. Some life cycle management
> will
> > > be necessary for these entities too. We can deploy k8s, swarm,
> dcos
> > > or even non-container specific services on top of CloudStack.
> The
> > > k8s is changing fast. One single tenant installation may need
> more
> > > than one VM group and may actually requires more (k8s
> federation).
> > > It will be a struggle to be in sync if we try to bring k8s
> specific
> > > knowledge into
> > cloudstack.
> > >
> > > Regards,
> > >
> > >
> > > --
> > > Lianghwa Jou
> > > VP Engineering, Accelerite
> > > email: lianghwa.jou@accelerite.com
> > >
> > >
> > >
> > >
> > >
> > > On 1/29/17, 11:54 PM, "Murali Reddy" <
> murali.reddy@shapeblue.com> wrote:
> > >
> > >
> > > I agree with some good views Will has shared and I also
> agree to
> > > the concerns raised by Wido and Eric. IMO we need balance of
> staying
> > > relevant/add new features vs stability of CloudStack and take
> > > corrective action if needed. We have two good examples for
> both.
> > > When SDN was hot technology CloudStack ended up with bunch of
> SDN
> > > controller
> > integrations.
> > > Few years later, now CloudStack is carrying baggage with no
> > > maintainers for those plugins, with probably not many of
> CloudStack
> > users needing overlays.
> > > On the other hand, other than attempt to liaison with ETSI for
> NFV
> > > no effort was done. OpenStack has become de-facto for NFV. Now
> for
> > > OpenStack, arguably NFV has become larger use case than cloud
> it self.
> > > I concur with Will’s point that with minimal viable solution
> that
> > > does not change the core of CloudStack, and can keep CloudStack
> > > relevant is worth to be taken in.
> > >
> > > Will,
> > >
> > > To your question of how different is from ShapeBlue’s
> container
> > > service, its design, implementation and API semantics etc
> remain same.
> > > ShapeBlue’s container service was true drop in plug-in to
> > > CloudStack, with this proposal I am trying to re-work to make
> it a
> > > core CloudStack pluggable service which is part of CloudStack.
> > >
> > > Key concepts that this proposal is trying to add
> > >
> > > - add notion of ‘container cluster’ as a first class
> entity
> > > in CloudStack. Which is bacially collection of other CloudStack
> > > resources (like VM’s, networks, public ip, network rules etc)
> > > - life cycle operation to manage ‘container cluster’
> like
> > > create, delete, start, stop, scale-up, scale-down, heal etc
> > > - orchestrate container orchestrator control plane on
> top of
> > > provisioned resources
> > >
> > > At-least for k8s (which is what this proposal is
> targeting),
> > > integration with k8s is bare minimum. There are cloud-config
> scripts
> > > that automatically setup k8s cluster master and node VM’s. All
> > > CloudStack is doing in passing the cloud-config to the core OS
> VM’s
> > > as
> > user data.
> > >
> > > Regards
> > > Murali Reddy
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 29/01/17, 8:54 AM, "Will Stevens" <
> williamstevens@gmail.com
> > > on behalf of wstevens@cloudops.com> wrote:
> > >
> > > >I agree that we need to be careful what we take on and
> own inside
> > > >CloudStack. I feel like some of the plugins or
> integrations
> > > which we have
> > > >been "maintaining" may serve us better to abandon, but I
> feel
> > > like that is
> > > >a whole discussion on its own.
> > > >
> > > >In this case, I feel like there is a minimum viable
> solution
> > > which puts
> > > >CloudStack in a pretty good place to enable container
> orchestration.
> > > For
> > > >example, one of the biggest challenges with K8S is the
> fact
> > > that it
> > is
> > > >single tenant. CloudStack has good multi tenancy support
> and
> > > has
> > the
> > > >ability to orchestrate the underlying infra quite well.
> We
> > > will have to be
> > > >very careful not to try to own too deep into the K8S world
> > > though, in my
> > > >opinion. We only want to be responsible for providing the
> > > infra (and a way
> > > >to bootstrap K8S ideally) and be able to scale the infra,
> > > everything else
> > > >should be owned by the K8S on top. That is the way I see
> it
> > > anyway, but
> > > >please add your input.
> > > >
> > > >I think it is a liability to try to go too deep, for the
> same
> > > reasons Wido
> > > >and Erik have mentioned. But I also think we need to
> take it
> > > seriously
> > > >because that train is moving and this may be a good
> opportunity
> > > to stay
> > > >relevant in a rapidly changing market.
> > > >
> > > >*Will STEVENS*
> > > >Lead Developer
> > > >
> > > ><https://goo.gl/NYZ8KK>
> > > >
> > > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > > <wi...@widodh.nl>
> > > wrote:
> > > >
> > > >>
> > > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > > wstevens@cloudops.com
> > > >> >:
> > > >> >
> > > >> >
> > > >> > Hey Murali,
> > > >> > How different is this proposal than what ShapeBlue
> already
> > > built. It
> > > >> looks
> > > >> > pretty consistent with the functionality that you
> guys open
> > > sourced in
> > > >> > Seville.
> > > >> >
> > > >> > I have not yet used this functionality, but I have
> reports
> > > that it works
> > > >> > quite well.
> > > >> >
> > > >> > I believe the premise here is to only orchestrate the
> VM
> > > layer
> > and
> > > >> > basically expose a "group" of running VMs to the
> user. The
> > > user is
> > > >> > responsible for configuring K8S or whatever other
> container
> > > orchestrator
> > > >> on
> > > >> > top. I saw mention of the "cloud-config" scripts in
> the
> > > FS, how are
> > > >> those
> > > >> > exposed to the cluster? Maybe the FS can expand on
> that a bit?
> > > >> >
> > > >> > I believe the core feature that is being requested to
> be
> > > added is the
> > > >> > ability to create a group of VMs which will be kept
> active
> > > as a group if
> > > >> at
> > > >> > all possible. ACS would be responsible for making
> sure
> > > that the number
> > > >> of
> > > >> > VMs specified for the group are in running state and
> it
> > > would spin up new
> > > >> > VMs as needed in order to satisfy the group
> settings. In
> > > general, it is
> > > >> > understood that any application running on this group
> would
> > > have to be
> > > >> > fault tolerant enough to be able to rediscover a new
> VM if
> > > one fails and
> > > >> is
> > > >> > replaced by a fresh copy. Is that fair to say? How
> is it
> > > expected that
> > > >> > this service discovery is done, just by VMs being
> present
> > > on the network?
> > > >> >
> > > >> > As for some of the other people's concerns in this
> thread.
> > > >> >
> > > >> > - Regarding Wido's remarks. I understand that there
> is
> > > some
> > added
> > > >> > complexity, but I don't feel like the scope of the
> addition is
> > > >> > unrealistic. I think the LXC integration was a lot
> farther
> > > out of the
> > > >> > scope of what ACS does then this is. This does not
> change
> > > the "things"
> > > >> > which ACS orchestrates, it just adds the concept of a
> > > grouping of things
> > > >> > which ACS already manages. I think this is the right
> > > approach since it
> > > >> is
> > > >> > not trying to be a container orchestrator. We will
> never
> > > compete with
> > > >> K8S,
> > > >> > for example, and we should not try, but K8S is here
> and the
> > > market wants
> > > >> > it. I do think we should be keeping our head up
> about that
> > > fact because
> > > >> > being able to provide a the underlay for K8S is very
> > > valuable in the
> > > >> > current marketplace. I see this functionality as a
> way to
> > > enable K8S
> > > >> > adoption on top of ACS without changing our core
> values.
> > > >> >
> > > >> > - Regarding Erik's remarks. The container space is
> moving
> > > fast, but so
> > > >> is
> > > >> > the industry. If we want to remain relevant, we need
> to be
> > > able to
> > > >> adapt a
> > > >> > bit. I don't think this is a big shift in what we
> do, but
> > > it is one that
> > > >> > enables people to be able to start running with
> something
> > > like K8S on top
> > > >> > of their existing ACS. This is something we are
> interested
> > > in doing and
> > > >> so
> > > >> > are our customers. If we can have a thin layer in ACS
> > > which helps enable
> > > >> > the use of K8S (or other container orchestrators) by
> > orchestrating
> > > >> > infrastructure, as we already do, and making it
> easier to
> > > adopt
> > a
> > > >> container
> > > >> > orchestrator running on top of ACS, I think that
> gives us a
> > > nice foothold
> > > >> > in the market. I don't really feel it is fair to
> compare
> > > containers to
> > > >> > IPv6. IPv6 has been out forever and it has taken
> almost a
> > > decade to get
> > > >> > anyone to adopt it. Containers have really only been
> here
> > > for like 2
> > > >> years
> > > >> > and they are changing the market landscape in a very
> real way.
> > > >> >
> > > >> > Kind of on topic and kind of off topic. I think
> > > understanding
> > our
> > > >> approach
> > > >> > to containers is going to be important for the ACS
> > > community as a whole.
> > > >> > If we don't offer that market anything, then we will
> not be
> > > considered
> > > >> and
> > > >> > we will lose market share we can't afford to lose.
> If we
> > > try to hitch
> > > >> our
> > > >> > horse to that cart too much, we will not be able to be
> > > agile enough and
> > > >> > will fail. I feel like the right approach is for us
> to
> > > know that it is a
> > > >> > thriving market and continue to do what we do, but to
> > > extend an olive
> > > >> > branch to that market. I think this sort of
> implementation
> > > is the right
> > > >> > approach because we are not trying to do too much.
> We are
> > > simply giving
> > > >> a
> > > >> > foundation on which the next big thing in the
> container
> > > orchestration
> > > >> world
> > > >> > can adopt without us having to compete directly in
> that
> > > space. I think
> > > >> we
> > > >> > have to focus on what we do best, but at the same
> time,
> > > think about what
> > > >> we
> > > >> > can do to enable that huge market of users to adopt
> ACS as their
> > > >> > foundation. The ability to offer VMs and containers
> in the
> > > same data
> > > >> plane
> > > >> > is something we have the ability to do, especially
> with
> > > this approach,
> > > >> and
> > > >> > is something that most other softwares can not do.
> The
> > > adoption of
> > > >> > containers by bigger organizations will be only part
> of
> > > their workload,
> > > >> > they will still be running VMs for the foreseeable
> future.
> > > Being able to
> > > >> > appeal to that market is going to be important for us.
> > > >> >
> > > >> > Hopefully I don't have too many strong opinions here,
> but I
> > > do think we
> > > >> > need to be thinking about how we move forward in a
> world
> > > which
> > is
> > > >> adopting
> > > >> > containers in a very real way.
> > > >> >
> > > >>
> > > >> Understood. I just want to prevent that we add more
> features
> > > to CloudStack
> > > >> which are poorly maintained. Not judging Murali here at
> all,
> > > but I'd rather
> > > >> see CloudStack loose code then have it added.
> > > >>
> > > >> Thinking about LXC, would like to see that removed
> together
> > > with various
> > > >> other network plugins which I think are rarely used.
> > > >>
> > > >> Now, the idea of Murali was misunderstood by me. I
> think it
> > > would be worth
> > > >> more if we would improve our cloud-init support and
> > > integration in various
> > > >> tools which makes it much easier to deploy VMs running
> > > containers
> > ON
> > > >> CloudStack.
> > > >>
> > > >> Not so much changing CloudStack code, but rather tooling
> > > around
> > it.
> > > >>
> > > >> If we have CloudStack talking to Kubernetes we suddenly
> have
> > > to maintain
> > > >> that API integration. Who's going to do that.
> Realistically.
> > > >>
> > > >> That's my main worry in this regard.
> > > >>
> > > >> We have so much more work to do in ACS in total that I
> don't
> > > know if this
> > > >> is the realistic route. I talk to many people who are
> not
> > > looking
> > at
> > > >> containers and are still working with VMs.
> > > >>
> > > >> There is not a single truth which is true, it really
> depends
> > > on who you
> > > >> ask.
> > > >>
> > > >> Wido
> > > >>
> > > >> > Cheers,
> > > >> >
> > > >> > Will
> > > >> >
> > > >> > *Will STEVENS*
> > > >> > Lead Developer
> > > >> >
> > > >> > <https://goo.gl/NYZ8KK>
> > > >> >
> > > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > > <te...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > > muralimmreddy@gmail.com
> > > >> >
> > > >> > > wrote:
> > > >> > > > All,
> > > >> > > >
> > > >> > > > I would like propose native functionality into
> > > CloudStack to provide
> > > >> a
> > > >> > > container service through which users out-of-the
> box can
> > > use to launch
> > > >> > > container based application. Idea is to support
> ability
> > > to orchestrate
> > > >> the
> > > >> > > resources and automate aspects of setting up
> container
> > > orchestrator
> > > >> through
> > > >> > > CloudStack. Public IAAS service providers AWS with
> its
> > > ECS [1] and
> > > >> google
> > > >> > > with GKE [2] already provides ability container
> applications.
> > > >> Competitive
> > > >> > > cloud orchestration platforms already have native
> support
> > > for container
> > > >> > > service. Users of CloudStack both as public cloud
> > > providers and users
> > > >> with
> > > >> > > private clouds will benefit with such functionality.
> > > >> > > >
> > > >> > > > While container orchestrator of user choice can be
> > > provisioned on
> > > >> top of
> > > >> > > CloudStack (with out CloudStack being involved) with
> > > tools
> > like
> > > >> > > TerraForm[3], Ansible[4] etc, advantage of having
> native
> > > orchestration
> > > >> is
> > > >> > > giving user a nice cohesive integration. This
> proposal
> > > would like add a
> > > >> > > notion of first class CloudStack entity called
> container
> > > cluster which
> > > >> can
> > > >> > > be used to provision resources, scale up, scale
> down,
> > > start and stop
> > > >> the
> > > >> > > cluster of VM’s on which containerised applications
> can
> > > be
> > run.
> > > For
> > > >> actual
> > > >> > > container orchestration we will still need container
> > > orchestrator like
> > > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > > container service
> > > >> can
> > > >> > > automate setting up of control place automatically.
> > > >> > > >
> > > >> > >
> > > >> > > To be honest I'm torn on this one.
> > > >> > >
> > > >> > > Containers are a rapid changing thing, and while
> docker swam,
> > > >> > > kubernetes, rancher or whatnot is popular today,
> they
> > > might not be
> > > >> > > tomorrow.
> > > >> > > They might use CoreOS today, but might not tomorrow.
> > > >> > >
> > > >> > > We have a rather poor track record of staying up to
> date
> > > with new
> > > >> > > features/versions, and adding a feature that is so
> > > rapidly changing
> > > >> > > is, I fear, going to be hard to maintain.
> > > >> > > Want an example, look at xenserver. It is one of
> the most used
> > > >> > > hypervisors we support, yet it took 6 months or so
> for us
> > > to support
> > > >> > > the latest release.
> > > >> > > Or IPv6...
> > > >> > >
> > > >> > > I don't mean to bash at maintainers/implementers of
> those
> > > features, I
> > > >> > > appreciate all the work being done in every aspect,
> but I
> > > believe we
> > > >> > > should be realistic and realize that we have issues
> with
> > > keeping stuff
> > > >> > > up to date.
> > > >> > >
> > > >> > > I'd say focus on making sure other tools can do
> their job
> > > well against
> > > >> > > CloudStack (kops, rancher, ++), but that does not
> mean I
> > > will
> > > -1 the
> > > >> > > idea if anyone really wants to go through with it.
> > > >> > >
> > > >> > > --
> > > >> > > Erik
> > > >> > >
> > > >>
> > >
> > > murali.reddy@shapeblue.com
> > > www.shapeblue.com<http://www.shapeblue.com>
> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information
> > > which is the property of Accelerite, a Persistent Systems
> business.
> > > It is intended only for the use of the individual or entity to
> which
> > > it is addressed. If you are not the intended recipient, you
> are not
> > > authorized to read, retain, copy, print, distribute or use this
> > > message. If you have received this communication in error,
> please
> > > notify the sender and delete all copies of this message.
> Accelerite,
> > > a Persistent Systems business does not accept any liability for
> > > virus
> > infected mails.
> > >
> > >
> > >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> which
> > is the property of Accelerite, a Persistent Systems business. It
> is
> > intended only for the use of the individual or entity to which
> it is
> > addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message.
> Accelerite, a
> > Persistent Systems business does not accept any liability for
> virus infected mails.
> >
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information
> which is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Accelerite, a Persistent Systems business does
> not accept any liability for virus infected mails.
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands
> @shapeblue
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Accelerite, a Persistent Systems business does
> not accept any liability for virus infected mails.
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: [PROPOSAL] add native vm-cluster orchestration service (was:
[PROPOSAL] add native container orchestration service)
Posted by Syed Ahmed <sa...@cloudops.com>.
We already call the VMs as Instances. So, InstanceCluster would be a better
name imo.
On Tue, Feb 28, 2017 at 10:05 AM, Daan Hoogland <daan.hoogland@shapeblue.com
> wrote:
> Kishan, I see some sensible additions but also some unnecessary omissions.
> Most of it seems to be Murali’s text so I’ll c&p your improvements back and
> rename the page to the more sensible title of “MachineCluster service” and
> delete the other.
>
> About naming, I was thinking MachineCluster instead of ServiceCluster,
> makes sense? Or even GuestMachineCluster. ServiceCluster could mean a
> supporting cluster that delivers e.g. backup as a service to guests, or
> maybe some build, artifact or networking service. For this ambiguity im am
> :-1: on the name ServiceCluster.
>
>
> On 28/02/17 11:16, "Kishan Kavala" <ki...@accelerite.com> wrote:
>
> Daan,
> I've updated the earlier spec to support any Vm cluster. Please let
> me know your thoughts on this.
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/
> Service+Cluster+Functional+Specification
>
> regards,
> Kishan
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
> Sent: 27 February 2017 04:02 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native vm-cluster orchestration service
> (was: [PROPOSAL] add native container orchestration service)
>
> Any follow up Koushik? I want to refactor our proof of concept and
> integrate it in master.
>
> On 21/02/17 10:42, "Kishan Kavala" <ki...@accelerite.com>
> wrote:
>
> Sure Daan. I'll publish the design on cwiki and share the link.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
> Sent: Monday, February 20, 2017 7:27 PM
> To: dev@cloudstack.apache.org
> Subject: [PROPOSAL] add native vm-cluster orchestration service
> (was: [PROPOSAL] add native container orchestration service)
>
> So, being very late in the discussion but havingread the whole
> thread before editting the title of this thread,
>
> Can we agree that we want a generic vm-cluster service and leave
> the container bits to containers? Kishan can you share your design?
> Shapeblue wants to rebase their k8 service on top of this and I would like
> yours and Murali's work to not conflict.
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531
> VENetherlands @shapeblue
>
>
>
>
> -----Original Message-----
> From: Paul Angus [mailto:paul.angus@shapeblue.com]
> Sent: dinsdag 7 februari 2017 08:14
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> Will is 100% correct. As I mentioned the Title is misleading.
> However, Murali did clarify in his explanation; this is really about vm
> cluster orchestration.
>
>
>
> ________________________________
> From: Will Stevens <ws...@cloudops.com>
> Sent: 6 Feb 2017 22:54
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> My understanding is that what Paul is talking about is what is
> already built and IS what the thread is talking about.
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani <
> rajesh.ramchandani@accelerite.com> wrote:
>
> > Hi Paul - I think this is different from what the thread was
> about.
> > The conversation was specifically about adding support for
> container
> > orchestrators. You are talking about provisioning a group of VMs.
> > Although, this is something I think several Cloudstack users have
> > requested before and we should propose a solution to this.
> >
> > Raj
> >
> > ________________________________
> > From: Paul Angus <pa...@shapeblue.com>
> > Sent: Monday, February 6, 2017 11:16:41 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: [PROPOSAL] add native container orchestration
> service
> >
> > #WhatHeSaid
> >
> > The title is misleading. The proposal is purely to add the
> notion of
> > Clusters of VMs to CloudStack. These may be for databases,
> containers
> > or anything else that needs 'clusters' of compute. Self-healing
> and
> > autoscaling are logical next steps to be added.
> >
> > Those guys at ShapeBlue have open-sourced their whole k8s
> container
> > service piece. If/when the 'cluster' part of that work is added
> into
> > CloudStack, the k8s specific pieces can be used by anyone who
> wants
> > to, alternatively they could be used for reference in order to
> create
> > another types of cluster. (or ignored completely).
> >
> >
> >
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Will Stevens [mailto:williamstevens@gmail.com]
> > Sent: 31 January 2017 13:26
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] add native container orchestration
> service
> >
> > s/cloud-init/cloud-config/
> >
> > On Jan 31, 2017 7:24 AM, "Will Stevens" <
> williamstevens@gmail.com> wrote:
> >
> > > I think that is covered in this proposal. There is nothing k8s
> > > specific in this integration (from what I understand), all the
> k8s
> > > details are passed in via the cloud-init configuration after
> the
> > > cluster
> > has been provisioned.
> > >
> > > On Jan 31, 2017 3:06 AM, "Lianghwa Jou"
> > > <li...@accelerite.com>
> > > wrote:
> > >
> > >
> > > There are many container orchestrators. Those container
> > > orchestrators are happy to run on any VMs or bare metal
> machines.
> > > K8s is just one of them and there will be more in the future.
> It may
> > > not be a good idea to make CloudStack to be k8s aware. IMO, the
> > > relationship between k8s and cloudstack should be similar to
> > > application and os. It probably not a good idea to make your
> OS to
> > > be aware of any specific applications so IMHO I don’t think
> k8s should be native to CloudStack.
> > > It makes more sense to provide some generic services that many
> > > applications can take advantages of. Some generic resource
> grouping
> > > service makes sense so a group of VMs, baremetal machines or
> network
> > > can be treated as a single entity. Some life cycle management
> will
> > > be necessary for these entities too. We can deploy k8s, swarm,
> dcos
> > > or even non-container specific services on top of CloudStack.
> The
> > > k8s is changing fast. One single tenant installation may need
> more
> > > than one VM group and may actually requires more (k8s
> federation).
> > > It will be a struggle to be in sync if we try to bring k8s
> specific
> > > knowledge into
> > cloudstack.
> > >
> > > Regards,
> > >
> > >
> > > --
> > > Lianghwa Jou
> > > VP Engineering, Accelerite
> > > email: lianghwa.jou@accelerite.com
> > >
> > >
> > >
> > >
> > >
> > > On 1/29/17, 11:54 PM, "Murali Reddy" <
> murali.reddy@shapeblue.com> wrote:
> > >
> > >
> > > I agree with some good views Will has shared and I also
> agree to
> > > the concerns raised by Wido and Eric. IMO we need balance of
> staying
> > > relevant/add new features vs stability of CloudStack and take
> > > corrective action if needed. We have two good examples for
> both.
> > > When SDN was hot technology CloudStack ended up with bunch of
> SDN
> > > controller
> > integrations.
> > > Few years later, now CloudStack is carrying baggage with no
> > > maintainers for those plugins, with probably not many of
> CloudStack
> > users needing overlays.
> > > On the other hand, other than attempt to liaison with ETSI for
> NFV
> > > no effort was done. OpenStack has become de-facto for NFV. Now
> for
> > > OpenStack, arguably NFV has become larger use case than cloud
> it self.
> > > I concur with Will’s point that with minimal viable solution
> that
> > > does not change the core of CloudStack, and can keep CloudStack
> > > relevant is worth to be taken in.
> > >
> > > Will,
> > >
> > > To your question of how different is from ShapeBlue’s
> container
> > > service, its design, implementation and API semantics etc
> remain same.
> > > ShapeBlue’s container service was true drop in plug-in to
> > > CloudStack, with this proposal I am trying to re-work to make
> it a
> > > core CloudStack pluggable service which is part of CloudStack.
> > >
> > > Key concepts that this proposal is trying to add
> > >
> > > - add notion of ‘container cluster’ as a first class
> entity
> > > in CloudStack. Which is bacially collection of other CloudStack
> > > resources (like VM’s, networks, public ip, network rules etc)
> > > - life cycle operation to manage ‘container cluster’
> like
> > > create, delete, start, stop, scale-up, scale-down, heal etc
> > > - orchestrate container orchestrator control plane on
> top of
> > > provisioned resources
> > >
> > > At-least for k8s (which is what this proposal is
> targeting),
> > > integration with k8s is bare minimum. There are cloud-config
> scripts
> > > that automatically setup k8s cluster master and node VM’s. All
> > > CloudStack is doing in passing the cloud-config to the core OS
> VM’s
> > > as
> > user data.
> > >
> > > Regards
> > > Murali Reddy
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 29/01/17, 8:54 AM, "Will Stevens" <
> williamstevens@gmail.com
> > > on behalf of wstevens@cloudops.com> wrote:
> > >
> > > >I agree that we need to be careful what we take on and
> own inside
> > > >CloudStack. I feel like some of the plugins or
> integrations
> > > which we have
> > > >been "maintaining" may serve us better to abandon, but I
> feel
> > > like that is
> > > >a whole discussion on its own.
> > > >
> > > >In this case, I feel like there is a minimum viable
> solution
> > > which puts
> > > >CloudStack in a pretty good place to enable container
> orchestration.
> > > For
> > > >example, one of the biggest challenges with K8S is the
> fact
> > > that it
> > is
> > > >single tenant. CloudStack has good multi tenancy support
> and
> > > has
> > the
> > > >ability to orchestrate the underlying infra quite well.
> We
> > > will have to be
> > > >very careful not to try to own too deep into the K8S world
> > > though, in my
> > > >opinion. We only want to be responsible for providing the
> > > infra (and a way
> > > >to bootstrap K8S ideally) and be able to scale the infra,
> > > everything else
> > > >should be owned by the K8S on top. That is the way I see
> it
> > > anyway, but
> > > >please add your input.
> > > >
> > > >I think it is a liability to try to go too deep, for the
> same
> > > reasons Wido
> > > >and Erik have mentioned. But I also think we need to
> take it
> > > seriously
> > > >because that train is moving and this may be a good
> opportunity
> > > to stay
> > > >relevant in a rapidly changing market.
> > > >
> > > >*Will STEVENS*
> > > >Lead Developer
> > > >
> > > ><https://goo.gl/NYZ8KK>
> > > >
> > > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > > <wi...@widodh.nl>
> > > wrote:
> > > >
> > > >>
> > > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > > wstevens@cloudops.com
> > > >> >:
> > > >> >
> > > >> >
> > > >> > Hey Murali,
> > > >> > How different is this proposal than what ShapeBlue
> already
> > > built. It
> > > >> looks
> > > >> > pretty consistent with the functionality that you
> guys open
> > > sourced in
> > > >> > Seville.
> > > >> >
> > > >> > I have not yet used this functionality, but I have
> reports
> > > that it works
> > > >> > quite well.
> > > >> >
> > > >> > I believe the premise here is to only orchestrate the
> VM
> > > layer
> > and
> > > >> > basically expose a "group" of running VMs to the
> user. The
> > > user is
> > > >> > responsible for configuring K8S or whatever other
> container
> > > orchestrator
> > > >> on
> > > >> > top. I saw mention of the "cloud-config" scripts in
> the
> > > FS, how are
> > > >> those
> > > >> > exposed to the cluster? Maybe the FS can expand on
> that a bit?
> > > >> >
> > > >> > I believe the core feature that is being requested to
> be
> > > added is the
> > > >> > ability to create a group of VMs which will be kept
> active
> > > as a group if
> > > >> at
> > > >> > all possible. ACS would be responsible for making
> sure
> > > that the number
> > > >> of
> > > >> > VMs specified for the group are in running state and
> it
> > > would spin up new
> > > >> > VMs as needed in order to satisfy the group
> settings. In
> > > general, it is
> > > >> > understood that any application running on this group
> would
> > > have to be
> > > >> > fault tolerant enough to be able to rediscover a new
> VM if
> > > one fails and
> > > >> is
> > > >> > replaced by a fresh copy. Is that fair to say? How
> is it
> > > expected that
> > > >> > this service discovery is done, just by VMs being
> present
> > > on the network?
> > > >> >
> > > >> > As for some of the other people's concerns in this
> thread.
> > > >> >
> > > >> > - Regarding Wido's remarks. I understand that there
> is
> > > some
> > added
> > > >> > complexity, but I don't feel like the scope of the
> addition is
> > > >> > unrealistic. I think the LXC integration was a lot
> farther
> > > out of the
> > > >> > scope of what ACS does then this is. This does not
> change
> > > the "things"
> > > >> > which ACS orchestrates, it just adds the concept of a
> > > grouping of things
> > > >> > which ACS already manages. I think this is the right
> > > approach since it
> > > >> is
> > > >> > not trying to be a container orchestrator. We will
> never
> > > compete with
> > > >> K8S,
> > > >> > for example, and we should not try, but K8S is here
> and the
> > > market wants
> > > >> > it. I do think we should be keeping our head up
> about that
> > > fact because
> > > >> > being able to provide a the underlay for K8S is very
> > > valuable in the
> > > >> > current marketplace. I see this functionality as a
> way to
> > > enable K8S
> > > >> > adoption on top of ACS without changing our core
> values.
> > > >> >
> > > >> > - Regarding Erik's remarks. The container space is
> moving
> > > fast, but so
> > > >> is
> > > >> > the industry. If we want to remain relevant, we need
> to be
> > > able to
> > > >> adapt a
> > > >> > bit. I don't think this is a big shift in what we
> do, but
> > > it is one that
> > > >> > enables people to be able to start running with
> something
> > > like K8S on top
> > > >> > of their existing ACS. This is something we are
> interested
> > > in doing and
> > > >> so
> > > >> > are our customers. If we can have a thin layer in ACS
> > > which helps enable
> > > >> > the use of K8S (or other container orchestrators) by
> > orchestrating
> > > >> > infrastructure, as we already do, and making it
> easier to
> > > adopt
> > a
> > > >> container
> > > >> > orchestrator running on top of ACS, I think that
> gives us a
> > > nice foothold
> > > >> > in the market. I don't really feel it is fair to
> compare
> > > containers to
> > > >> > IPv6. IPv6 has been out forever and it has taken
> almost a
> > > decade to get
> > > >> > anyone to adopt it. Containers have really only been
> here
> > > for like 2
> > > >> years
> > > >> > and they are changing the market landscape in a very
> real way.
> > > >> >
> > > >> > Kind of on topic and kind of off topic. I think
> > > understanding
> > our
> > > >> approach
> > > >> > to containers is going to be important for the ACS
> > > community as a whole.
> > > >> > If we don't offer that market anything, then we will
> not be
> > > considered
> > > >> and
> > > >> > we will lose market share we can't afford to lose.
> If we
> > > try to hitch
> > > >> our
> > > >> > horse to that cart too much, we will not be able to be
> > > agile enough and
> > > >> > will fail. I feel like the right approach is for us
> to
> > > know that it is a
> > > >> > thriving market and continue to do what we do, but to
> > > extend an olive
> > > >> > branch to that market. I think this sort of
> implementation
> > > is the right
> > > >> > approach because we are not trying to do too much.
> We are
> > > simply giving
> > > >> a
> > > >> > foundation on which the next big thing in the
> container
> > > orchestration
> > > >> world
> > > >> > can adopt without us having to compete directly in
> that
> > > space. I think
> > > >> we
> > > >> > have to focus on what we do best, but at the same
> time,
> > > think about what
> > > >> we
> > > >> > can do to enable that huge market of users to adopt
> ACS as their
> > > >> > foundation. The ability to offer VMs and containers
> in the
> > > same data
> > > >> plane
> > > >> > is something we have the ability to do, especially
> with
> > > this approach,
> > > >> and
> > > >> > is something that most other softwares can not do.
> The
> > > adoption of
> > > >> > containers by bigger organizations will be only part
> of
> > > their workload,
> > > >> > they will still be running VMs for the foreseeable
> future.
> > > Being able to
> > > >> > appeal to that market is going to be important for us.
> > > >> >
> > > >> > Hopefully I don't have too many strong opinions here,
> but I
> > > do think we
> > > >> > need to be thinking about how we move forward in a
> world
> > > which
> > is
> > > >> adopting
> > > >> > containers in a very real way.
> > > >> >
> > > >>
> > > >> Understood. I just want to prevent that we add more
> features
> > > to CloudStack
> > > >> which are poorly maintained. Not judging Murali here at
> all,
> > > but I'd rather
> > > >> see CloudStack loose code then have it added.
> > > >>
> > > >> Thinking about LXC, would like to see that removed
> together
> > > with various
> > > >> other network plugins which I think are rarely used.
> > > >>
> > > >> Now, the idea of Murali was misunderstood by me. I
> think it
> > > would be worth
> > > >> more if we would improve our cloud-init support and
> > > integration in various
> > > >> tools which makes it much easier to deploy VMs running
> > > containers
> > ON
> > > >> CloudStack.
> > > >>
> > > >> Not so much changing CloudStack code, but rather tooling
> > > around
> > it.
> > > >>
> > > >> If we have CloudStack talking to Kubernetes we suddenly
> have
> > > to maintain
> > > >> that API integration. Who's going to do that.
> Realistically.
> > > >>
> > > >> That's my main worry in this regard.
> > > >>
> > > >> We have so much more work to do in ACS in total that I
> don't
> > > know if this
> > > >> is the realistic route. I talk to many people who are
> not
> > > looking
> > at
> > > >> containers and are still working with VMs.
> > > >>
> > > >> There is not a single truth which is true, it really
> depends
> > > on who you
> > > >> ask.
> > > >>
> > > >> Wido
> > > >>
> > > >> > Cheers,
> > > >> >
> > > >> > Will
> > > >> >
> > > >> > *Will STEVENS*
> > > >> > Lead Developer
> > > >> >
> > > >> > <https://goo.gl/NYZ8KK>
> > > >> >
> > > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > > <te...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > > muralimmreddy@gmail.com
> > > >> >
> > > >> > > wrote:
> > > >> > > > All,
> > > >> > > >
> > > >> > > > I would like propose native functionality into
> > > CloudStack to provide
> > > >> a
> > > >> > > container service through which users out-of-the
> box can
> > > use to launch
> > > >> > > container based application. Idea is to support
> ability
> > > to orchestrate
> > > >> the
> > > >> > > resources and automate aspects of setting up
> container
> > > orchestrator
> > > >> through
> > > >> > > CloudStack. Public IAAS service providers AWS with
> its
> > > ECS [1] and
> > > >> google
> > > >> > > with GKE [2] already provides ability container
> applications.
> > > >> Competitive
> > > >> > > cloud orchestration platforms already have native
> support
> > > for container
> > > >> > > service. Users of CloudStack both as public cloud
> > > providers and users
> > > >> with
> > > >> > > private clouds will benefit with such functionality.
> > > >> > > >
> > > >> > > > While container orchestrator of user choice can be
> > > provisioned on
> > > >> top of
> > > >> > > CloudStack (with out CloudStack being involved) with
> > > tools
> > like
> > > >> > > TerraForm[3], Ansible[4] etc, advantage of having
> native
> > > orchestration
> > > >> is
> > > >> > > giving user a nice cohesive integration. This
> proposal
> > > would like add a
> > > >> > > notion of first class CloudStack entity called
> container
> > > cluster which
> > > >> can
> > > >> > > be used to provision resources, scale up, scale
> down,
> > > start and stop
> > > >> the
> > > >> > > cluster of VM’s on which containerised applications
> can
> > > be
> > run.
> > > For
> > > >> actual
> > > >> > > container orchestration we will still need container
> > > orchestrator like
> > > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > > container service
> > > >> can
> > > >> > > automate setting up of control place automatically.
> > > >> > > >
> > > >> > >
> > > >> > > To be honest I'm torn on this one.
> > > >> > >
> > > >> > > Containers are a rapid changing thing, and while
> docker swam,
> > > >> > > kubernetes, rancher or whatnot is popular today,
> they
> > > might not be
> > > >> > > tomorrow.
> > > >> > > They might use CoreOS today, but might not tomorrow.
> > > >> > >
> > > >> > > We have a rather poor track record of staying up to
> date
> > > with new
> > > >> > > features/versions, and adding a feature that is so
> > > rapidly changing
> > > >> > > is, I fear, going to be hard to maintain.
> > > >> > > Want an example, look at xenserver. It is one of
> the most used
> > > >> > > hypervisors we support, yet it took 6 months or so
> for us
> > > to support
> > > >> > > the latest release.
> > > >> > > Or IPv6...
> > > >> > >
> > > >> > > I don't mean to bash at maintainers/implementers of
> those
> > > features, I
> > > >> > > appreciate all the work being done in every aspect,
> but I
> > > believe we
> > > >> > > should be realistic and realize that we have issues
> with
> > > keeping stuff
> > > >> > > up to date.
> > > >> > >
> > > >> > > I'd say focus on making sure other tools can do
> their job
> > > well against
> > > >> > > CloudStack (kops, rancher, ++), but that does not
> mean I
> > > will
> > > -1 the
> > > >> > > idea if anyone really wants to go through with it.
> > > >> > >
> > > >> > > --
> > > >> > > Erik
> > > >> > >
> > > >>
> > >
> > > murali.reddy@shapeblue.com
> > > www.shapeblue.com<http://www.shapeblue.com>
> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information
> > > which is the property of Accelerite, a Persistent Systems
> business.
> > > It is intended only for the use of the individual or entity to
> which
> > > it is addressed. If you are not the intended recipient, you
> are not
> > > authorized to read, retain, copy, print, distribute or use this
> > > message. If you have received this communication in error,
> please
> > > notify the sender and delete all copies of this message.
> Accelerite,
> > > a Persistent Systems business does not accept any liability for
> > > virus
> > infected mails.
> > >
> > >
> > >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> which
> > is the property of Accelerite, a Persistent Systems business. It
> is
> > intended only for the use of the individual or entity to which
> it is
> > addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message.
> Accelerite, a
> > Persistent Systems business does not accept any liability for
> virus infected mails.
> >
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information
> which is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Accelerite, a Persistent Systems business does
> not accept any liability for virus infected mails.
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands
> @shapeblue
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Accelerite, a Persistent Systems business does
> not accept any liability for virus infected mails.
>
>
>
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
Re: [PROPOSAL] add native vm-cluster orchestration service
(was: [PROPOSAL] add native container orchestration service)
Posted by Daan Hoogland <da...@shapeblue.com>.
Kishan, I see some sensible additions but also some unnecessary omissions. Most of it seems to be Murali’s text so I’ll c&p your improvements back and rename the page to the more sensible title of “MachineCluster service” and delete the other.
About naming, I was thinking MachineCluster instead of ServiceCluster, makes sense? Or even GuestMachineCluster. ServiceCluster could mean a supporting cluster that delivers e.g. backup as a service to guests, or maybe some build, artifact or networking service. For this ambiguity im am :-1: on the name ServiceCluster.
On 28/02/17 11:16, "Kishan Kavala" <ki...@accelerite.com> wrote:
Daan,
I've updated the earlier spec to support any Vm cluster. Please let me know your thoughts on this.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Service+Cluster+Functional+Specification
regards,
Kishan
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
Sent: 27 February 2017 04:02 PM
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native vm-cluster orchestration service (was: [PROPOSAL] add native container orchestration service)
Any follow up Koushik? I want to refactor our proof of concept and integrate it in master.
On 21/02/17 10:42, "Kishan Kavala" <ki...@accelerite.com> wrote:
Sure Daan. I'll publish the design on cwiki and share the link.
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
Sent: Monday, February 20, 2017 7:27 PM
To: dev@cloudstack.apache.org
Subject: [PROPOSAL] add native vm-cluster orchestration service (was: [PROPOSAL] add native container orchestration service)
So, being very late in the discussion but havingread the whole thread before editting the title of this thread,
Can we agree that we want a generic vm-cluster service and leave the container bits to containers? Kishan can you share your design? Shapeblue wants to rebase their k8 service on top of this and I would like yours and Murali's work to not conflict.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands @shapeblue
-----Original Message-----
From: Paul Angus [mailto:paul.angus@shapeblue.com]
Sent: dinsdag 7 februari 2017 08:14
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
Will is 100% correct. As I mentioned the Title is misleading. However, Murali did clarify in his explanation; this is really about vm cluster orchestration.
________________________________
From: Will Stevens <ws...@cloudops.com>
Sent: 6 Feb 2017 22:54
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
My understanding is that what Paul is talking about is what is already built and IS what the thread is talking about.
*Will STEVENS*
Lead Developer
<https://goo.gl/NYZ8KK>
On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani < rajesh.ramchandani@accelerite.com> wrote:
> Hi Paul - I think this is different from what the thread was about.
> The conversation was specifically about adding support for container
> orchestrators. You are talking about provisioning a group of VMs.
> Although, this is something I think several Cloudstack users have
> requested before and we should propose a solution to this.
>
> Raj
>
> ________________________________
> From: Paul Angus <pa...@shapeblue.com>
> Sent: Monday, February 6, 2017 11:16:41 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] add native container orchestration service
>
> #WhatHeSaid
>
> The title is misleading. The proposal is purely to add the notion of
> Clusters of VMs to CloudStack. These may be for databases, containers
> or anything else that needs 'clusters' of compute. Self-healing and
> autoscaling are logical next steps to be added.
>
> Those guys at ShapeBlue have open-sourced their whole k8s container
> service piece. If/when the 'cluster' part of that work is added into
> CloudStack, the k8s specific pieces can be used by anyone who wants
> to, alternatively they could be used for reference in order to create
> another types of cluster. (or ignored completely).
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
> -----Original Message-----
> From: Will Stevens [mailto:williamstevens@gmail.com]
> Sent: 31 January 2017 13:26
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> s/cloud-init/cloud-config/
>
> On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:
>
> > I think that is covered in this proposal. There is nothing k8s
> > specific in this integration (from what I understand), all the k8s
> > details are passed in via the cloud-init configuration after the
> > cluster
> has been provisioned.
> >
> > On Jan 31, 2017 3:06 AM, "Lianghwa Jou"
> > <li...@accelerite.com>
> > wrote:
> >
> >
> > There are many container orchestrators. Those container
> > orchestrators are happy to run on any VMs or bare metal machines.
> > K8s is just one of them and there will be more in the future. It may
> > not be a good idea to make CloudStack to be k8s aware. IMO, the
> > relationship between k8s and cloudstack should be similar to
> > application and os. It probably not a good idea to make your OS to
> > be aware of any specific applications so IMHO I don’t think k8s should be native to CloudStack.
> > It makes more sense to provide some generic services that many
> > applications can take advantages of. Some generic resource grouping
> > service makes sense so a group of VMs, baremetal machines or network
> > can be treated as a single entity. Some life cycle management will
> > be necessary for these entities too. We can deploy k8s, swarm, dcos
> > or even non-container specific services on top of CloudStack. The
> > k8s is changing fast. One single tenant installation may need more
> > than one VM group and may actually requires more (k8s federation).
> > It will be a struggle to be in sync if we try to bring k8s specific
> > knowledge into
> cloudstack.
> >
> > Regards,
> >
> >
> > --
> > Lianghwa Jou
> > VP Engineering, Accelerite
> > email: lianghwa.jou@accelerite.com
> >
> >
> >
> >
> >
> > On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
> >
> >
> > I agree with some good views Will has shared and I also agree to
> > the concerns raised by Wido and Eric. IMO we need balance of staying
> > relevant/add new features vs stability of CloudStack and take
> > corrective action if needed. We have two good examples for both.
> > When SDN was hot technology CloudStack ended up with bunch of SDN
> > controller
> integrations.
> > Few years later, now CloudStack is carrying baggage with no
> > maintainers for those plugins, with probably not many of CloudStack
> users needing overlays.
> > On the other hand, other than attempt to liaison with ETSI for NFV
> > no effort was done. OpenStack has become de-facto for NFV. Now for
> > OpenStack, arguably NFV has become larger use case than cloud it self.
> > I concur with Will’s point that with minimal viable solution that
> > does not change the core of CloudStack, and can keep CloudStack
> > relevant is worth to be taken in.
> >
> > Will,
> >
> > To your question of how different is from ShapeBlue’s container
> > service, its design, implementation and API semantics etc remain same.
> > ShapeBlue’s container service was true drop in plug-in to
> > CloudStack, with this proposal I am trying to re-work to make it a
> > core CloudStack pluggable service which is part of CloudStack.
> >
> > Key concepts that this proposal is trying to add
> >
> > - add notion of ‘container cluster’ as a first class entity
> > in CloudStack. Which is bacially collection of other CloudStack
> > resources (like VM’s, networks, public ip, network rules etc)
> > - life cycle operation to manage ‘container cluster’ like
> > create, delete, start, stop, scale-up, scale-down, heal etc
> > - orchestrate container orchestrator control plane on top of
> > provisioned resources
> >
> > At-least for k8s (which is what this proposal is targeting),
> > integration with k8s is bare minimum. There are cloud-config scripts
> > that automatically setup k8s cluster master and node VM’s. All
> > CloudStack is doing in passing the cloud-config to the core OS VM’s
> > as
> user data.
> >
> > Regards
> > Murali Reddy
> >
> >
> >
> >
> >
> >
> >
> > On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com
> > on behalf of wstevens@cloudops.com> wrote:
> >
> > >I agree that we need to be careful what we take on and own inside
> > >CloudStack. I feel like some of the plugins or integrations
> > which we have
> > >been "maintaining" may serve us better to abandon, but I feel
> > like that is
> > >a whole discussion on its own.
> > >
> > >In this case, I feel like there is a minimum viable solution
> > which puts
> > >CloudStack in a pretty good place to enable container orchestration.
> > For
> > >example, one of the biggest challenges with K8S is the fact
> > that it
> is
> > >single tenant. CloudStack has good multi tenancy support and
> > has
> the
> > >ability to orchestrate the underlying infra quite well. We
> > will have to be
> > >very careful not to try to own too deep into the K8S world
> > though, in my
> > >opinion. We only want to be responsible for providing the
> > infra (and a way
> > >to bootstrap K8S ideally) and be able to scale the infra,
> > everything else
> > >should be owned by the K8S on top. That is the way I see it
> > anyway, but
> > >please add your input.
> > >
> > >I think it is a liability to try to go too deep, for the same
> > reasons Wido
> > >and Erik have mentioned. But I also think we need to take it
> > seriously
> > >because that train is moving and this may be a good opportunity
> > to stay
> > >relevant in a rapidly changing market.
> > >
> > >*Will STEVENS*
> > >Lead Developer
> > >
> > ><https://goo.gl/NYZ8KK>
> > >
> > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > <wi...@widodh.nl>
> > wrote:
> > >
> > >>
> > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > wstevens@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > Hey Murali,
> > >> > How different is this proposal than what ShapeBlue already
> > built. It
> > >> looks
> > >> > pretty consistent with the functionality that you guys open
> > sourced in
> > >> > Seville.
> > >> >
> > >> > I have not yet used this functionality, but I have reports
> > that it works
> > >> > quite well.
> > >> >
> > >> > I believe the premise here is to only orchestrate the VM
> > layer
> and
> > >> > basically expose a "group" of running VMs to the user. The
> > user is
> > >> > responsible for configuring K8S or whatever other container
> > orchestrator
> > >> on
> > >> > top. I saw mention of the "cloud-config" scripts in the
> > FS, how are
> > >> those
> > >> > exposed to the cluster? Maybe the FS can expand on that a bit?
> > >> >
> > >> > I believe the core feature that is being requested to be
> > added is the
> > >> > ability to create a group of VMs which will be kept active
> > as a group if
> > >> at
> > >> > all possible. ACS would be responsible for making sure
> > that the number
> > >> of
> > >> > VMs specified for the group are in running state and it
> > would spin up new
> > >> > VMs as needed in order to satisfy the group settings. In
> > general, it is
> > >> > understood that any application running on this group would
> > have to be
> > >> > fault tolerant enough to be able to rediscover a new VM if
> > one fails and
> > >> is
> > >> > replaced by a fresh copy. Is that fair to say? How is it
> > expected that
> > >> > this service discovery is done, just by VMs being present
> > on the network?
> > >> >
> > >> > As for some of the other people's concerns in this thread.
> > >> >
> > >> > - Regarding Wido's remarks. I understand that there is
> > some
> added
> > >> > complexity, but I don't feel like the scope of the addition is
> > >> > unrealistic. I think the LXC integration was a lot farther
> > out of the
> > >> > scope of what ACS does then this is. This does not change
> > the "things"
> > >> > which ACS orchestrates, it just adds the concept of a
> > grouping of things
> > >> > which ACS already manages. I think this is the right
> > approach since it
> > >> is
> > >> > not trying to be a container orchestrator. We will never
> > compete with
> > >> K8S,
> > >> > for example, and we should not try, but K8S is here and the
> > market wants
> > >> > it. I do think we should be keeping our head up about that
> > fact because
> > >> > being able to provide a the underlay for K8S is very
> > valuable in the
> > >> > current marketplace. I see this functionality as a way to
> > enable K8S
> > >> > adoption on top of ACS without changing our core values.
> > >> >
> > >> > - Regarding Erik's remarks. The container space is moving
> > fast, but so
> > >> is
> > >> > the industry. If we want to remain relevant, we need to be
> > able to
> > >> adapt a
> > >> > bit. I don't think this is a big shift in what we do, but
> > it is one that
> > >> > enables people to be able to start running with something
> > like K8S on top
> > >> > of their existing ACS. This is something we are interested
> > in doing and
> > >> so
> > >> > are our customers. If we can have a thin layer in ACS
> > which helps enable
> > >> > the use of K8S (or other container orchestrators) by
> orchestrating
> > >> > infrastructure, as we already do, and making it easier to
> > adopt
> a
> > >> container
> > >> > orchestrator running on top of ACS, I think that gives us a
> > nice foothold
> > >> > in the market. I don't really feel it is fair to compare
> > containers to
> > >> > IPv6. IPv6 has been out forever and it has taken almost a
> > decade to get
> > >> > anyone to adopt it. Containers have really only been here
> > for like 2
> > >> years
> > >> > and they are changing the market landscape in a very real way.
> > >> >
> > >> > Kind of on topic and kind of off topic. I think
> > understanding
> our
> > >> approach
> > >> > to containers is going to be important for the ACS
> > community as a whole.
> > >> > If we don't offer that market anything, then we will not be
> > considered
> > >> and
> > >> > we will lose market share we can't afford to lose. If we
> > try to hitch
> > >> our
> > >> > horse to that cart too much, we will not be able to be
> > agile enough and
> > >> > will fail. I feel like the right approach is for us to
> > know that it is a
> > >> > thriving market and continue to do what we do, but to
> > extend an olive
> > >> > branch to that market. I think this sort of implementation
> > is the right
> > >> > approach because we are not trying to do too much. We are
> > simply giving
> > >> a
> > >> > foundation on which the next big thing in the container
> > orchestration
> > >> world
> > >> > can adopt without us having to compete directly in that
> > space. I think
> > >> we
> > >> > have to focus on what we do best, but at the same time,
> > think about what
> > >> we
> > >> > can do to enable that huge market of users to adopt ACS as their
> > >> > foundation. The ability to offer VMs and containers in the
> > same data
> > >> plane
> > >> > is something we have the ability to do, especially with
> > this approach,
> > >> and
> > >> > is something that most other softwares can not do. The
> > adoption of
> > >> > containers by bigger organizations will be only part of
> > their workload,
> > >> > they will still be running VMs for the foreseeable future.
> > Being able to
> > >> > appeal to that market is going to be important for us.
> > >> >
> > >> > Hopefully I don't have too many strong opinions here, but I
> > do think we
> > >> > need to be thinking about how we move forward in a world
> > which
> is
> > >> adopting
> > >> > containers in a very real way.
> > >> >
> > >>
> > >> Understood. I just want to prevent that we add more features
> > to CloudStack
> > >> which are poorly maintained. Not judging Murali here at all,
> > but I'd rather
> > >> see CloudStack loose code then have it added.
> > >>
> > >> Thinking about LXC, would like to see that removed together
> > with various
> > >> other network plugins which I think are rarely used.
> > >>
> > >> Now, the idea of Murali was misunderstood by me. I think it
> > would be worth
> > >> more if we would improve our cloud-init support and
> > integration in various
> > >> tools which makes it much easier to deploy VMs running
> > containers
> ON
> > >> CloudStack.
> > >>
> > >> Not so much changing CloudStack code, but rather tooling
> > around
> it.
> > >>
> > >> If we have CloudStack talking to Kubernetes we suddenly have
> > to maintain
> > >> that API integration. Who's going to do that. Realistically.
> > >>
> > >> That's my main worry in this regard.
> > >>
> > >> We have so much more work to do in ACS in total that I don't
> > know if this
> > >> is the realistic route. I talk to many people who are not
> > looking
> at
> > >> containers and are still working with VMs.
> > >>
> > >> There is not a single truth which is true, it really depends
> > on who you
> > >> ask.
> > >>
> > >> Wido
> > >>
> > >> > Cheers,
> > >> >
> > >> > Will
> > >> >
> > >> > *Will STEVENS*
> > >> > Lead Developer
> > >> >
> > >> > <https://goo.gl/NYZ8KK>
> > >> >
> > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > <te...@gmail.com>
> > wrote:
> > >> >
> > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > muralimmreddy@gmail.com
> > >> >
> > >> > > wrote:
> > >> > > > All,
> > >> > > >
> > >> > > > I would like propose native functionality into
> > CloudStack to provide
> > >> a
> > >> > > container service through which users out-of-the box can
> > use to launch
> > >> > > container based application. Idea is to support ability
> > to orchestrate
> > >> the
> > >> > > resources and automate aspects of setting up container
> > orchestrator
> > >> through
> > >> > > CloudStack. Public IAAS service providers AWS with its
> > ECS [1] and
> > >> google
> > >> > > with GKE [2] already provides ability container applications.
> > >> Competitive
> > >> > > cloud orchestration platforms already have native support
> > for container
> > >> > > service. Users of CloudStack both as public cloud
> > providers and users
> > >> with
> > >> > > private clouds will benefit with such functionality.
> > >> > > >
> > >> > > > While container orchestrator of user choice can be
> > provisioned on
> > >> top of
> > >> > > CloudStack (with out CloudStack being involved) with
> > tools
> like
> > >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> > orchestration
> > >> is
> > >> > > giving user a nice cohesive integration. This proposal
> > would like add a
> > >> > > notion of first class CloudStack entity called container
> > cluster which
> > >> can
> > >> > > be used to provision resources, scale up, scale down,
> > start and stop
> > >> the
> > >> > > cluster of VM’s on which containerised applications can
> > be
> run.
> > For
> > >> actual
> > >> > > container orchestration we will still need container
> > orchestrator like
> > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > container service
> > >> can
> > >> > > automate setting up of control place automatically.
> > >> > > >
> > >> > >
> > >> > > To be honest I'm torn on this one.
> > >> > >
> > >> > > Containers are a rapid changing thing, and while docker swam,
> > >> > > kubernetes, rancher or whatnot is popular today, they
> > might not be
> > >> > > tomorrow.
> > >> > > They might use CoreOS today, but might not tomorrow.
> > >> > >
> > >> > > We have a rather poor track record of staying up to date
> > with new
> > >> > > features/versions, and adding a feature that is so
> > rapidly changing
> > >> > > is, I fear, going to be hard to maintain.
> > >> > > Want an example, look at xenserver. It is one of the most used
> > >> > > hypervisors we support, yet it took 6 months or so for us
> > to support
> > >> > > the latest release.
> > >> > > Or IPv6...
> > >> > >
> > >> > > I don't mean to bash at maintainers/implementers of those
> > features, I
> > >> > > appreciate all the work being done in every aspect, but I
> > believe we
> > >> > > should be realistic and realize that we have issues with
> > keeping stuff
> > >> > > up to date.
> > >> > >
> > >> > > I'd say focus on making sure other tools can do their job
> > well against
> > >> > > CloudStack (kops, rancher, ++), but that does not mean I
> > will
> > -1 the
> > >> > > idea if anyone really wants to go through with it.
> > >> > >
> > >> > > --
> > >> > > Erik
> > >> > >
> > >>
> >
> > murali.reddy@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> > which is the property of Accelerite, a Persistent Systems business.
> > It is intended only for the use of the individual or entity to which
> > it is addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Accelerite,
> > a Persistent Systems business does not accept any liability for
> > virus
> infected mails.
> >
> >
> >
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not
> authorized to read, retain, copy, print, distribute or use this
> message. If you have received this communication in error, please
> notify the sender and delete all copies of this message. Accelerite, a
> Persistent Systems business does not accept any liability for virus infected mails.
>
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands @shapeblue
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
RE: [PROPOSAL] add native vm-cluster orchestration service (was:
[PROPOSAL] add native container orchestration service)
Posted by Kishan Kavala <ki...@accelerite.com>.
Daan,
I've updated the earlier spec to support any Vm cluster. Please let me know your thoughts on this.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Service+Cluster+Functional+Specification
regards,
Kishan
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
Sent: 27 February 2017 04:02 PM
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native vm-cluster orchestration service (was: [PROPOSAL] add native container orchestration service)
Any follow up Koushik? I want to refactor our proof of concept and integrate it in master.
On 21/02/17 10:42, "Kishan Kavala" <ki...@accelerite.com> wrote:
Sure Daan. I'll publish the design on cwiki and share the link.
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
Sent: Monday, February 20, 2017 7:27 PM
To: dev@cloudstack.apache.org
Subject: [PROPOSAL] add native vm-cluster orchestration service (was: [PROPOSAL] add native container orchestration service)
So, being very late in the discussion but havingread the whole thread before editting the title of this thread,
Can we agree that we want a generic vm-cluster service and leave the container bits to containers? Kishan can you share your design? Shapeblue wants to rebase their k8 service on top of this and I would like yours and Murali's work to not conflict.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands @shapeblue
-----Original Message-----
From: Paul Angus [mailto:paul.angus@shapeblue.com]
Sent: dinsdag 7 februari 2017 08:14
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
Will is 100% correct. As I mentioned the Title is misleading. However, Murali did clarify in his explanation; this is really about vm cluster orchestration.
________________________________
From: Will Stevens <ws...@cloudops.com>
Sent: 6 Feb 2017 22:54
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
My understanding is that what Paul is talking about is what is already built and IS what the thread is talking about.
*Will STEVENS*
Lead Developer
<https://goo.gl/NYZ8KK>
On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani < rajesh.ramchandani@accelerite.com> wrote:
> Hi Paul - I think this is different from what the thread was about.
> The conversation was specifically about adding support for container
> orchestrators. You are talking about provisioning a group of VMs.
> Although, this is something I think several Cloudstack users have
> requested before and we should propose a solution to this.
>
> Raj
>
> ________________________________
> From: Paul Angus <pa...@shapeblue.com>
> Sent: Monday, February 6, 2017 11:16:41 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] add native container orchestration service
>
> #WhatHeSaid
>
> The title is misleading. The proposal is purely to add the notion of
> Clusters of VMs to CloudStack. These may be for databases, containers
> or anything else that needs 'clusters' of compute. Self-healing and
> autoscaling are logical next steps to be added.
>
> Those guys at ShapeBlue have open-sourced their whole k8s container
> service piece. If/when the 'cluster' part of that work is added into
> CloudStack, the k8s specific pieces can be used by anyone who wants
> to, alternatively they could be used for reference in order to create
> another types of cluster. (or ignored completely).
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
> -----Original Message-----
> From: Will Stevens [mailto:williamstevens@gmail.com]
> Sent: 31 January 2017 13:26
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> s/cloud-init/cloud-config/
>
> On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:
>
> > I think that is covered in this proposal. There is nothing k8s
> > specific in this integration (from what I understand), all the k8s
> > details are passed in via the cloud-init configuration after the
> > cluster
> has been provisioned.
> >
> > On Jan 31, 2017 3:06 AM, "Lianghwa Jou"
> > <li...@accelerite.com>
> > wrote:
> >
> >
> > There are many container orchestrators. Those container
> > orchestrators are happy to run on any VMs or bare metal machines.
> > K8s is just one of them and there will be more in the future. It may
> > not be a good idea to make CloudStack to be k8s aware. IMO, the
> > relationship between k8s and cloudstack should be similar to
> > application and os. It probably not a good idea to make your OS to
> > be aware of any specific applications so IMHO I don’t think k8s should be native to CloudStack.
> > It makes more sense to provide some generic services that many
> > applications can take advantages of. Some generic resource grouping
> > service makes sense so a group of VMs, baremetal machines or network
> > can be treated as a single entity. Some life cycle management will
> > be necessary for these entities too. We can deploy k8s, swarm, dcos
> > or even non-container specific services on top of CloudStack. The
> > k8s is changing fast. One single tenant installation may need more
> > than one VM group and may actually requires more (k8s federation).
> > It will be a struggle to be in sync if we try to bring k8s specific
> > knowledge into
> cloudstack.
> >
> > Regards,
> >
> >
> > --
> > Lianghwa Jou
> > VP Engineering, Accelerite
> > email: lianghwa.jou@accelerite.com
> >
> >
> >
> >
> >
> > On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
> >
> >
> > I agree with some good views Will has shared and I also agree to
> > the concerns raised by Wido and Eric. IMO we need balance of staying
> > relevant/add new features vs stability of CloudStack and take
> > corrective action if needed. We have two good examples for both.
> > When SDN was hot technology CloudStack ended up with bunch of SDN
> > controller
> integrations.
> > Few years later, now CloudStack is carrying baggage with no
> > maintainers for those plugins, with probably not many of CloudStack
> users needing overlays.
> > On the other hand, other than attempt to liaison with ETSI for NFV
> > no effort was done. OpenStack has become de-facto for NFV. Now for
> > OpenStack, arguably NFV has become larger use case than cloud it self.
> > I concur with Will’s point that with minimal viable solution that
> > does not change the core of CloudStack, and can keep CloudStack
> > relevant is worth to be taken in.
> >
> > Will,
> >
> > To your question of how different is from ShapeBlue’s container
> > service, its design, implementation and API semantics etc remain same.
> > ShapeBlue’s container service was true drop in plug-in to
> > CloudStack, with this proposal I am trying to re-work to make it a
> > core CloudStack pluggable service which is part of CloudStack.
> >
> > Key concepts that this proposal is trying to add
> >
> > - add notion of ‘container cluster’ as a first class entity
> > in CloudStack. Which is bacially collection of other CloudStack
> > resources (like VM’s, networks, public ip, network rules etc)
> > - life cycle operation to manage ‘container cluster’ like
> > create, delete, start, stop, scale-up, scale-down, heal etc
> > - orchestrate container orchestrator control plane on top of
> > provisioned resources
> >
> > At-least for k8s (which is what this proposal is targeting),
> > integration with k8s is bare minimum. There are cloud-config scripts
> > that automatically setup k8s cluster master and node VM’s. All
> > CloudStack is doing in passing the cloud-config to the core OS VM’s
> > as
> user data.
> >
> > Regards
> > Murali Reddy
> >
> >
> >
> >
> >
> >
> >
> > On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com
> > on behalf of wstevens@cloudops.com> wrote:
> >
> > >I agree that we need to be careful what we take on and own inside
> > >CloudStack. I feel like some of the plugins or integrations
> > which we have
> > >been "maintaining" may serve us better to abandon, but I feel
> > like that is
> > >a whole discussion on its own.
> > >
> > >In this case, I feel like there is a minimum viable solution
> > which puts
> > >CloudStack in a pretty good place to enable container orchestration.
> > For
> > >example, one of the biggest challenges with K8S is the fact
> > that it
> is
> > >single tenant. CloudStack has good multi tenancy support and
> > has
> the
> > >ability to orchestrate the underlying infra quite well. We
> > will have to be
> > >very careful not to try to own too deep into the K8S world
> > though, in my
> > >opinion. We only want to be responsible for providing the
> > infra (and a way
> > >to bootstrap K8S ideally) and be able to scale the infra,
> > everything else
> > >should be owned by the K8S on top. That is the way I see it
> > anyway, but
> > >please add your input.
> > >
> > >I think it is a liability to try to go too deep, for the same
> > reasons Wido
> > >and Erik have mentioned. But I also think we need to take it
> > seriously
> > >because that train is moving and this may be a good opportunity
> > to stay
> > >relevant in a rapidly changing market.
> > >
> > >*Will STEVENS*
> > >Lead Developer
> > >
> > ><https://goo.gl/NYZ8KK>
> > >
> > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > <wi...@widodh.nl>
> > wrote:
> > >
> > >>
> > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > wstevens@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > Hey Murali,
> > >> > How different is this proposal than what ShapeBlue already
> > built. It
> > >> looks
> > >> > pretty consistent with the functionality that you guys open
> > sourced in
> > >> > Seville.
> > >> >
> > >> > I have not yet used this functionality, but I have reports
> > that it works
> > >> > quite well.
> > >> >
> > >> > I believe the premise here is to only orchestrate the VM
> > layer
> and
> > >> > basically expose a "group" of running VMs to the user. The
> > user is
> > >> > responsible for configuring K8S or whatever other container
> > orchestrator
> > >> on
> > >> > top. I saw mention of the "cloud-config" scripts in the
> > FS, how are
> > >> those
> > >> > exposed to the cluster? Maybe the FS can expand on that a bit?
> > >> >
> > >> > I believe the core feature that is being requested to be
> > added is the
> > >> > ability to create a group of VMs which will be kept active
> > as a group if
> > >> at
> > >> > all possible. ACS would be responsible for making sure
> > that the number
> > >> of
> > >> > VMs specified for the group are in running state and it
> > would spin up new
> > >> > VMs as needed in order to satisfy the group settings. In
> > general, it is
> > >> > understood that any application running on this group would
> > have to be
> > >> > fault tolerant enough to be able to rediscover a new VM if
> > one fails and
> > >> is
> > >> > replaced by a fresh copy. Is that fair to say? How is it
> > expected that
> > >> > this service discovery is done, just by VMs being present
> > on the network?
> > >> >
> > >> > As for some of the other people's concerns in this thread.
> > >> >
> > >> > - Regarding Wido's remarks. I understand that there is
> > some
> added
> > >> > complexity, but I don't feel like the scope of the addition is
> > >> > unrealistic. I think the LXC integration was a lot farther
> > out of the
> > >> > scope of what ACS does then this is. This does not change
> > the "things"
> > >> > which ACS orchestrates, it just adds the concept of a
> > grouping of things
> > >> > which ACS already manages. I think this is the right
> > approach since it
> > >> is
> > >> > not trying to be a container orchestrator. We will never
> > compete with
> > >> K8S,
> > >> > for example, and we should not try, but K8S is here and the
> > market wants
> > >> > it. I do think we should be keeping our head up about that
> > fact because
> > >> > being able to provide a the underlay for K8S is very
> > valuable in the
> > >> > current marketplace. I see this functionality as a way to
> > enable K8S
> > >> > adoption on top of ACS without changing our core values.
> > >> >
> > >> > - Regarding Erik's remarks. The container space is moving
> > fast, but so
> > >> is
> > >> > the industry. If we want to remain relevant, we need to be
> > able to
> > >> adapt a
> > >> > bit. I don't think this is a big shift in what we do, but
> > it is one that
> > >> > enables people to be able to start running with something
> > like K8S on top
> > >> > of their existing ACS. This is something we are interested
> > in doing and
> > >> so
> > >> > are our customers. If we can have a thin layer in ACS
> > which helps enable
> > >> > the use of K8S (or other container orchestrators) by
> orchestrating
> > >> > infrastructure, as we already do, and making it easier to
> > adopt
> a
> > >> container
> > >> > orchestrator running on top of ACS, I think that gives us a
> > nice foothold
> > >> > in the market. I don't really feel it is fair to compare
> > containers to
> > >> > IPv6. IPv6 has been out forever and it has taken almost a
> > decade to get
> > >> > anyone to adopt it. Containers have really only been here
> > for like 2
> > >> years
> > >> > and they are changing the market landscape in a very real way.
> > >> >
> > >> > Kind of on topic and kind of off topic. I think
> > understanding
> our
> > >> approach
> > >> > to containers is going to be important for the ACS
> > community as a whole.
> > >> > If we don't offer that market anything, then we will not be
> > considered
> > >> and
> > >> > we will lose market share we can't afford to lose. If we
> > try to hitch
> > >> our
> > >> > horse to that cart too much, we will not be able to be
> > agile enough and
> > >> > will fail. I feel like the right approach is for us to
> > know that it is a
> > >> > thriving market and continue to do what we do, but to
> > extend an olive
> > >> > branch to that market. I think this sort of implementation
> > is the right
> > >> > approach because we are not trying to do too much. We are
> > simply giving
> > >> a
> > >> > foundation on which the next big thing in the container
> > orchestration
> > >> world
> > >> > can adopt without us having to compete directly in that
> > space. I think
> > >> we
> > >> > have to focus on what we do best, but at the same time,
> > think about what
> > >> we
> > >> > can do to enable that huge market of users to adopt ACS as their
> > >> > foundation. The ability to offer VMs and containers in the
> > same data
> > >> plane
> > >> > is something we have the ability to do, especially with
> > this approach,
> > >> and
> > >> > is something that most other softwares can not do. The
> > adoption of
> > >> > containers by bigger organizations will be only part of
> > their workload,
> > >> > they will still be running VMs for the foreseeable future.
> > Being able to
> > >> > appeal to that market is going to be important for us.
> > >> >
> > >> > Hopefully I don't have too many strong opinions here, but I
> > do think we
> > >> > need to be thinking about how we move forward in a world
> > which
> is
> > >> adopting
> > >> > containers in a very real way.
> > >> >
> > >>
> > >> Understood. I just want to prevent that we add more features
> > to CloudStack
> > >> which are poorly maintained. Not judging Murali here at all,
> > but I'd rather
> > >> see CloudStack loose code then have it added.
> > >>
> > >> Thinking about LXC, would like to see that removed together
> > with various
> > >> other network plugins which I think are rarely used.
> > >>
> > >> Now, the idea of Murali was misunderstood by me. I think it
> > would be worth
> > >> more if we would improve our cloud-init support and
> > integration in various
> > >> tools which makes it much easier to deploy VMs running
> > containers
> ON
> > >> CloudStack.
> > >>
> > >> Not so much changing CloudStack code, but rather tooling
> > around
> it.
> > >>
> > >> If we have CloudStack talking to Kubernetes we suddenly have
> > to maintain
> > >> that API integration. Who's going to do that. Realistically.
> > >>
> > >> That's my main worry in this regard.
> > >>
> > >> We have so much more work to do in ACS in total that I don't
> > know if this
> > >> is the realistic route. I talk to many people who are not
> > looking
> at
> > >> containers and are still working with VMs.
> > >>
> > >> There is not a single truth which is true, it really depends
> > on who you
> > >> ask.
> > >>
> > >> Wido
> > >>
> > >> > Cheers,
> > >> >
> > >> > Will
> > >> >
> > >> > *Will STEVENS*
> > >> > Lead Developer
> > >> >
> > >> > <https://goo.gl/NYZ8KK>
> > >> >
> > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > <te...@gmail.com>
> > wrote:
> > >> >
> > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > muralimmreddy@gmail.com
> > >> >
> > >> > > wrote:
> > >> > > > All,
> > >> > > >
> > >> > > > I would like propose native functionality into
> > CloudStack to provide
> > >> a
> > >> > > container service through which users out-of-the box can
> > use to launch
> > >> > > container based application. Idea is to support ability
> > to orchestrate
> > >> the
> > >> > > resources and automate aspects of setting up container
> > orchestrator
> > >> through
> > >> > > CloudStack. Public IAAS service providers AWS with its
> > ECS [1] and
> > >> google
> > >> > > with GKE [2] already provides ability container applications.
> > >> Competitive
> > >> > > cloud orchestration platforms already have native support
> > for container
> > >> > > service. Users of CloudStack both as public cloud
> > providers and users
> > >> with
> > >> > > private clouds will benefit with such functionality.
> > >> > > >
> > >> > > > While container orchestrator of user choice can be
> > provisioned on
> > >> top of
> > >> > > CloudStack (with out CloudStack being involved) with
> > tools
> like
> > >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> > orchestration
> > >> is
> > >> > > giving user a nice cohesive integration. This proposal
> > would like add a
> > >> > > notion of first class CloudStack entity called container
> > cluster which
> > >> can
> > >> > > be used to provision resources, scale up, scale down,
> > start and stop
> > >> the
> > >> > > cluster of VM’s on which containerised applications can
> > be
> run.
> > For
> > >> actual
> > >> > > container orchestration we will still need container
> > orchestrator like
> > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > container service
> > >> can
> > >> > > automate setting up of control place automatically.
> > >> > > >
> > >> > >
> > >> > > To be honest I'm torn on this one.
> > >> > >
> > >> > > Containers are a rapid changing thing, and while docker swam,
> > >> > > kubernetes, rancher or whatnot is popular today, they
> > might not be
> > >> > > tomorrow.
> > >> > > They might use CoreOS today, but might not tomorrow.
> > >> > >
> > >> > > We have a rather poor track record of staying up to date
> > with new
> > >> > > features/versions, and adding a feature that is so
> > rapidly changing
> > >> > > is, I fear, going to be hard to maintain.
> > >> > > Want an example, look at xenserver. It is one of the most used
> > >> > > hypervisors we support, yet it took 6 months or so for us
> > to support
> > >> > > the latest release.
> > >> > > Or IPv6...
> > >> > >
> > >> > > I don't mean to bash at maintainers/implementers of those
> > features, I
> > >> > > appreciate all the work being done in every aspect, but I
> > believe we
> > >> > > should be realistic and realize that we have issues with
> > keeping stuff
> > >> > > up to date.
> > >> > >
> > >> > > I'd say focus on making sure other tools can do their job
> > well against
> > >> > > CloudStack (kops, rancher, ++), but that does not mean I
> > will
> > -1 the
> > >> > > idea if anyone really wants to go through with it.
> > >> > >
> > >> > > --
> > >> > > Erik
> > >> > >
> > >>
> >
> > murali.reddy@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> > which is the property of Accelerite, a Persistent Systems business.
> > It is intended only for the use of the individual or entity to which
> > it is addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Accelerite,
> > a Persistent Systems business does not accept any liability for
> > virus
> infected mails.
> >
> >
> >
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not
> authorized to read, retain, copy, print, distribute or use this
> message. If you have received this communication in error, please
> notify the sender and delete all copies of this message. Accelerite, a
> Persistent Systems business does not accept any liability for virus infected mails.
>
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands @shapeblue
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: [PROPOSAL] add native vm-cluster orchestration service
(was: [PROPOSAL] add native container orchestration service)
Posted by Daan Hoogland <da...@shapeblue.com>.
Any follow up Koushik? I want to refactor our proof of concept and integrate it in master.
On 21/02/17 10:42, "Kishan Kavala" <ki...@accelerite.com> wrote:
Sure Daan. I'll publish the design on cwiki and share the link.
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
Sent: Monday, February 20, 2017 7:27 PM
To: dev@cloudstack.apache.org
Subject: [PROPOSAL] add native vm-cluster orchestration service (was: [PROPOSAL] add native container orchestration service)
So, being very late in the discussion but havingread the whole thread before editting the title of this thread,
Can we agree that we want a generic vm-cluster service and leave the container bits to containers? Kishan can you share your design? Shapeblue wants to rebase their k8 service on top of this and I would like yours and Murali's work to not conflict.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands @shapeblue
-----Original Message-----
From: Paul Angus [mailto:paul.angus@shapeblue.com]
Sent: dinsdag 7 februari 2017 08:14
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
Will is 100% correct. As I mentioned the Title is misleading. However, Murali did clarify in his explanation; this is really about vm cluster orchestration.
________________________________
From: Will Stevens <ws...@cloudops.com>
Sent: 6 Feb 2017 22:54
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
My understanding is that what Paul is talking about is what is already built and IS what the thread is talking about.
*Will STEVENS*
Lead Developer
<https://goo.gl/NYZ8KK>
On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani < rajesh.ramchandani@accelerite.com> wrote:
> Hi Paul - I think this is different from what the thread was about.
> The conversation was specifically about adding support for container
> orchestrators. You are talking about provisioning a group of VMs.
> Although, this is something I think several Cloudstack users have
> requested before and we should propose a solution to this.
>
> Raj
>
> ________________________________
> From: Paul Angus <pa...@shapeblue.com>
> Sent: Monday, February 6, 2017 11:16:41 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] add native container orchestration service
>
> #WhatHeSaid
>
> The title is misleading. The proposal is purely to add the notion of
> Clusters of VMs to CloudStack. These may be for databases, containers
> or anything else that needs 'clusters' of compute. Self-healing and
> autoscaling are logical next steps to be added.
>
> Those guys at ShapeBlue have open-sourced their whole k8s container
> service piece. If/when the 'cluster' part of that work is added into
> CloudStack, the k8s specific pieces can be used by anyone who wants
> to, alternatively they could be used for reference in order to create
> another types of cluster. (or ignored completely).
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
> -----Original Message-----
> From: Will Stevens [mailto:williamstevens@gmail.com]
> Sent: 31 January 2017 13:26
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> s/cloud-init/cloud-config/
>
> On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:
>
> > I think that is covered in this proposal. There is nothing k8s
> > specific in this integration (from what I understand), all the k8s
> > details are passed in via the cloud-init configuration after the
> > cluster
> has been provisioned.
> >
> > On Jan 31, 2017 3:06 AM, "Lianghwa Jou"
> > <li...@accelerite.com>
> > wrote:
> >
> >
> > There are many container orchestrators. Those container
> > orchestrators are happy to run on any VMs or bare metal machines.
> > K8s is just one of them and there will be more in the future. It may
> > not be a good idea to make CloudStack to be k8s aware. IMO, the
> > relationship between k8s and cloudstack should be similar to
> > application and os. It probably not a good idea to make your OS to
> > be aware of any specific applications so IMHO I don’t think k8s should be native to CloudStack.
> > It makes more sense to provide some generic services that many
> > applications can take advantages of. Some generic resource grouping
> > service makes sense so a group of VMs, baremetal machines or network
> > can be treated as a single entity. Some life cycle management will
> > be necessary for these entities too. We can deploy k8s, swarm, dcos
> > or even non-container specific services on top of CloudStack. The
> > k8s is changing fast. One single tenant installation may need more
> > than one VM group and may actually requires more (k8s federation).
> > It will be a struggle to be in sync if we try to bring k8s specific
> > knowledge into
> cloudstack.
> >
> > Regards,
> >
> >
> > --
> > Lianghwa Jou
> > VP Engineering, Accelerite
> > email: lianghwa.jou@accelerite.com
> >
> >
> >
> >
> >
> > On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
> >
> >
> > I agree with some good views Will has shared and I also agree to
> > the concerns raised by Wido and Eric. IMO we need balance of staying
> > relevant/add new features vs stability of CloudStack and take
> > corrective action if needed. We have two good examples for both.
> > When SDN was hot technology CloudStack ended up with bunch of SDN
> > controller
> integrations.
> > Few years later, now CloudStack is carrying baggage with no
> > maintainers for those plugins, with probably not many of CloudStack
> users needing overlays.
> > On the other hand, other than attempt to liaison with ETSI for NFV
> > no effort was done. OpenStack has become de-facto for NFV. Now for
> > OpenStack, arguably NFV has become larger use case than cloud it self.
> > I concur with Will’s point that with minimal viable solution that
> > does not change the core of CloudStack, and can keep CloudStack
> > relevant is worth to be taken in.
> >
> > Will,
> >
> > To your question of how different is from ShapeBlue’s container
> > service, its design, implementation and API semantics etc remain same.
> > ShapeBlue’s container service was true drop in plug-in to
> > CloudStack, with this proposal I am trying to re-work to make it a
> > core CloudStack pluggable service which is part of CloudStack.
> >
> > Key concepts that this proposal is trying to add
> >
> > - add notion of ‘container cluster’ as a first class entity
> > in CloudStack. Which is bacially collection of other CloudStack
> > resources (like VM’s, networks, public ip, network rules etc)
> > - life cycle operation to manage ‘container cluster’ like
> > create, delete, start, stop, scale-up, scale-down, heal etc
> > - orchestrate container orchestrator control plane on top of
> > provisioned resources
> >
> > At-least for k8s (which is what this proposal is targeting),
> > integration with k8s is bare minimum. There are cloud-config scripts
> > that automatically setup k8s cluster master and node VM’s. All
> > CloudStack is doing in passing the cloud-config to the core OS VM’s
> > as
> user data.
> >
> > Regards
> > Murali Reddy
> >
> >
> >
> >
> >
> >
> >
> > On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com
> > on behalf of wstevens@cloudops.com> wrote:
> >
> > >I agree that we need to be careful what we take on and own inside
> > >CloudStack. I feel like some of the plugins or integrations
> > which we have
> > >been "maintaining" may serve us better to abandon, but I feel
> > like that is
> > >a whole discussion on its own.
> > >
> > >In this case, I feel like there is a minimum viable solution
> > which puts
> > >CloudStack in a pretty good place to enable container orchestration.
> > For
> > >example, one of the biggest challenges with K8S is the fact
> > that it
> is
> > >single tenant. CloudStack has good multi tenancy support and
> > has
> the
> > >ability to orchestrate the underlying infra quite well. We
> > will have to be
> > >very careful not to try to own too deep into the K8S world
> > though, in my
> > >opinion. We only want to be responsible for providing the
> > infra (and a way
> > >to bootstrap K8S ideally) and be able to scale the infra,
> > everything else
> > >should be owned by the K8S on top. That is the way I see it
> > anyway, but
> > >please add your input.
> > >
> > >I think it is a liability to try to go too deep, for the same
> > reasons Wido
> > >and Erik have mentioned. But I also think we need to take it
> > seriously
> > >because that train is moving and this may be a good opportunity
> > to stay
> > >relevant in a rapidly changing market.
> > >
> > >*Will STEVENS*
> > >Lead Developer
> > >
> > ><https://goo.gl/NYZ8KK>
> > >
> > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > <wi...@widodh.nl>
> > wrote:
> > >
> > >>
> > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > wstevens@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > Hey Murali,
> > >> > How different is this proposal than what ShapeBlue already
> > built. It
> > >> looks
> > >> > pretty consistent with the functionality that you guys open
> > sourced in
> > >> > Seville.
> > >> >
> > >> > I have not yet used this functionality, but I have reports
> > that it works
> > >> > quite well.
> > >> >
> > >> > I believe the premise here is to only orchestrate the VM
> > layer
> and
> > >> > basically expose a "group" of running VMs to the user. The
> > user is
> > >> > responsible for configuring K8S or whatever other container
> > orchestrator
> > >> on
> > >> > top. I saw mention of the "cloud-config" scripts in the
> > FS, how are
> > >> those
> > >> > exposed to the cluster? Maybe the FS can expand on that a bit?
> > >> >
> > >> > I believe the core feature that is being requested to be
> > added is the
> > >> > ability to create a group of VMs which will be kept active
> > as a group if
> > >> at
> > >> > all possible. ACS would be responsible for making sure
> > that the number
> > >> of
> > >> > VMs specified for the group are in running state and it
> > would spin up new
> > >> > VMs as needed in order to satisfy the group settings. In
> > general, it is
> > >> > understood that any application running on this group would
> > have to be
> > >> > fault tolerant enough to be able to rediscover a new VM if
> > one fails and
> > >> is
> > >> > replaced by a fresh copy. Is that fair to say? How is it
> > expected that
> > >> > this service discovery is done, just by VMs being present
> > on the network?
> > >> >
> > >> > As for some of the other people's concerns in this thread.
> > >> >
> > >> > - Regarding Wido's remarks. I understand that there is
> > some
> added
> > >> > complexity, but I don't feel like the scope of the addition is
> > >> > unrealistic. I think the LXC integration was a lot farther
> > out of the
> > >> > scope of what ACS does then this is. This does not change
> > the "things"
> > >> > which ACS orchestrates, it just adds the concept of a
> > grouping of things
> > >> > which ACS already manages. I think this is the right
> > approach since it
> > >> is
> > >> > not trying to be a container orchestrator. We will never
> > compete with
> > >> K8S,
> > >> > for example, and we should not try, but K8S is here and the
> > market wants
> > >> > it. I do think we should be keeping our head up about that
> > fact because
> > >> > being able to provide a the underlay for K8S is very
> > valuable in the
> > >> > current marketplace. I see this functionality as a way to
> > enable K8S
> > >> > adoption on top of ACS without changing our core values.
> > >> >
> > >> > - Regarding Erik's remarks. The container space is moving
> > fast, but so
> > >> is
> > >> > the industry. If we want to remain relevant, we need to be
> > able to
> > >> adapt a
> > >> > bit. I don't think this is a big shift in what we do, but
> > it is one that
> > >> > enables people to be able to start running with something
> > like K8S on top
> > >> > of their existing ACS. This is something we are interested
> > in doing and
> > >> so
> > >> > are our customers. If we can have a thin layer in ACS
> > which helps enable
> > >> > the use of K8S (or other container orchestrators) by
> orchestrating
> > >> > infrastructure, as we already do, and making it easier to
> > adopt
> a
> > >> container
> > >> > orchestrator running on top of ACS, I think that gives us a
> > nice foothold
> > >> > in the market. I don't really feel it is fair to compare
> > containers to
> > >> > IPv6. IPv6 has been out forever and it has taken almost a
> > decade to get
> > >> > anyone to adopt it. Containers have really only been here
> > for like 2
> > >> years
> > >> > and they are changing the market landscape in a very real way.
> > >> >
> > >> > Kind of on topic and kind of off topic. I think
> > understanding
> our
> > >> approach
> > >> > to containers is going to be important for the ACS
> > community as a whole.
> > >> > If we don't offer that market anything, then we will not be
> > considered
> > >> and
> > >> > we will lose market share we can't afford to lose. If we
> > try to hitch
> > >> our
> > >> > horse to that cart too much, we will not be able to be
> > agile enough and
> > >> > will fail. I feel like the right approach is for us to
> > know that it is a
> > >> > thriving market and continue to do what we do, but to
> > extend an olive
> > >> > branch to that market. I think this sort of implementation
> > is the right
> > >> > approach because we are not trying to do too much. We are
> > simply giving
> > >> a
> > >> > foundation on which the next big thing in the container
> > orchestration
> > >> world
> > >> > can adopt without us having to compete directly in that
> > space. I think
> > >> we
> > >> > have to focus on what we do best, but at the same time,
> > think about what
> > >> we
> > >> > can do to enable that huge market of users to adopt ACS as their
> > >> > foundation. The ability to offer VMs and containers in the
> > same data
> > >> plane
> > >> > is something we have the ability to do, especially with
> > this approach,
> > >> and
> > >> > is something that most other softwares can not do. The
> > adoption of
> > >> > containers by bigger organizations will be only part of
> > their workload,
> > >> > they will still be running VMs for the foreseeable future.
> > Being able to
> > >> > appeal to that market is going to be important for us.
> > >> >
> > >> > Hopefully I don't have too many strong opinions here, but I
> > do think we
> > >> > need to be thinking about how we move forward in a world
> > which
> is
> > >> adopting
> > >> > containers in a very real way.
> > >> >
> > >>
> > >> Understood. I just want to prevent that we add more features
> > to CloudStack
> > >> which are poorly maintained. Not judging Murali here at all,
> > but I'd rather
> > >> see CloudStack loose code then have it added.
> > >>
> > >> Thinking about LXC, would like to see that removed together
> > with various
> > >> other network plugins which I think are rarely used.
> > >>
> > >> Now, the idea of Murali was misunderstood by me. I think it
> > would be worth
> > >> more if we would improve our cloud-init support and
> > integration in various
> > >> tools which makes it much easier to deploy VMs running
> > containers
> ON
> > >> CloudStack.
> > >>
> > >> Not so much changing CloudStack code, but rather tooling
> > around
> it.
> > >>
> > >> If we have CloudStack talking to Kubernetes we suddenly have
> > to maintain
> > >> that API integration. Who's going to do that. Realistically.
> > >>
> > >> That's my main worry in this regard.
> > >>
> > >> We have so much more work to do in ACS in total that I don't
> > know if this
> > >> is the realistic route. I talk to many people who are not
> > looking
> at
> > >> containers and are still working with VMs.
> > >>
> > >> There is not a single truth which is true, it really depends
> > on who you
> > >> ask.
> > >>
> > >> Wido
> > >>
> > >> > Cheers,
> > >> >
> > >> > Will
> > >> >
> > >> > *Will STEVENS*
> > >> > Lead Developer
> > >> >
> > >> > <https://goo.gl/NYZ8KK>
> > >> >
> > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > <te...@gmail.com>
> > wrote:
> > >> >
> > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > muralimmreddy@gmail.com
> > >> >
> > >> > > wrote:
> > >> > > > All,
> > >> > > >
> > >> > > > I would like propose native functionality into
> > CloudStack to provide
> > >> a
> > >> > > container service through which users out-of-the box can
> > use to launch
> > >> > > container based application. Idea is to support ability
> > to orchestrate
> > >> the
> > >> > > resources and automate aspects of setting up container
> > orchestrator
> > >> through
> > >> > > CloudStack. Public IAAS service providers AWS with its
> > ECS [1] and
> > >> google
> > >> > > with GKE [2] already provides ability container applications.
> > >> Competitive
> > >> > > cloud orchestration platforms already have native support
> > for container
> > >> > > service. Users of CloudStack both as public cloud
> > providers and users
> > >> with
> > >> > > private clouds will benefit with such functionality.
> > >> > > >
> > >> > > > While container orchestrator of user choice can be
> > provisioned on
> > >> top of
> > >> > > CloudStack (with out CloudStack being involved) with
> > tools
> like
> > >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> > orchestration
> > >> is
> > >> > > giving user a nice cohesive integration. This proposal
> > would like add a
> > >> > > notion of first class CloudStack entity called container
> > cluster which
> > >> can
> > >> > > be used to provision resources, scale up, scale down,
> > start and stop
> > >> the
> > >> > > cluster of VM’s on which containerised applications can
> > be
> run.
> > For
> > >> actual
> > >> > > container orchestration we will still need container
> > orchestrator like
> > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > container service
> > >> can
> > >> > > automate setting up of control place automatically.
> > >> > > >
> > >> > >
> > >> > > To be honest I'm torn on this one.
> > >> > >
> > >> > > Containers are a rapid changing thing, and while docker swam,
> > >> > > kubernetes, rancher or whatnot is popular today, they
> > might not be
> > >> > > tomorrow.
> > >> > > They might use CoreOS today, but might not tomorrow.
> > >> > >
> > >> > > We have a rather poor track record of staying up to date
> > with new
> > >> > > features/versions, and adding a feature that is so
> > rapidly changing
> > >> > > is, I fear, going to be hard to maintain.
> > >> > > Want an example, look at xenserver. It is one of the most used
> > >> > > hypervisors we support, yet it took 6 months or so for us
> > to support
> > >> > > the latest release.
> > >> > > Or IPv6...
> > >> > >
> > >> > > I don't mean to bash at maintainers/implementers of those
> > features, I
> > >> > > appreciate all the work being done in every aspect, but I
> > believe we
> > >> > > should be realistic and realize that we have issues with
> > keeping stuff
> > >> > > up to date.
> > >> > >
> > >> > > I'd say focus on making sure other tools can do their job
> > well against
> > >> > > CloudStack (kops, rancher, ++), but that does not mean I
> > will
> > -1 the
> > >> > > idea if anyone really wants to go through with it.
> > >> > >
> > >> > > --
> > >> > > Erik
> > >> > >
> > >>
> >
> > murali.reddy@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> > which is the property of Accelerite, a Persistent Systems business.
> > It is intended only for the use of the individual or entity to which
> > it is addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Accelerite,
> > a Persistent Systems business does not accept any liability for
> > virus
> infected mails.
> >
> >
> >
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not
> authorized to read, retain, copy, print, distribute or use this
> message. If you have received this communication in error, please
> notify the sender and delete all copies of this message. Accelerite, a
> Persistent Systems business does not accept any liability for virus infected mails.
>
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands
@shapeblue
RE: [PROPOSAL] add native vm-cluster orchestration service (was:
[PROPOSAL] add native container orchestration service)
Posted by Kishan Kavala <ki...@accelerite.com>.
Sure Daan. I'll publish the design on cwiki and share the link.
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@shapeblue.com]
Sent: Monday, February 20, 2017 7:27 PM
To: dev@cloudstack.apache.org
Subject: [PROPOSAL] add native vm-cluster orchestration service (was: [PROPOSAL] add native container orchestration service)
So, being very late in the discussion but havingread the whole thread before editting the title of this thread,
Can we agree that we want a generic vm-cluster service and leave the container bits to containers? Kishan can you share your design? Shapeblue wants to rebase their k8 service on top of this and I would like yours and Murali's work to not conflict.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands @shapeblue
-----Original Message-----
From: Paul Angus [mailto:paul.angus@shapeblue.com]
Sent: dinsdag 7 februari 2017 08:14
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
Will is 100% correct. As I mentioned the Title is misleading. However, Murali did clarify in his explanation; this is really about vm cluster orchestration.
________________________________
From: Will Stevens <ws...@cloudops.com>
Sent: 6 Feb 2017 22:54
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] add native container orchestration service
My understanding is that what Paul is talking about is what is already built and IS what the thread is talking about.
*Will STEVENS*
Lead Developer
<https://goo.gl/NYZ8KK>
On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani < rajesh.ramchandani@accelerite.com> wrote:
> Hi Paul - I think this is different from what the thread was about.
> The conversation was specifically about adding support for container
> orchestrators. You are talking about provisioning a group of VMs.
> Although, this is something I think several Cloudstack users have
> requested before and we should propose a solution to this.
>
> Raj
>
> ________________________________
> From: Paul Angus <pa...@shapeblue.com>
> Sent: Monday, February 6, 2017 11:16:41 AM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] add native container orchestration service
>
> #WhatHeSaid
>
> The title is misleading. The proposal is purely to add the notion of
> Clusters of VMs to CloudStack. These may be for databases, containers
> or anything else that needs 'clusters' of compute. Self-healing and
> autoscaling are logical next steps to be added.
>
> Those guys at ShapeBlue have open-sourced their whole k8s container
> service piece. If/when the 'cluster' part of that work is added into
> CloudStack, the k8s specific pieces can be used by anyone who wants
> to, alternatively they could be used for reference in order to create
> another types of cluster. (or ignored completely).
>
>
>
>
> paul.angus@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
> -----Original Message-----
> From: Will Stevens [mailto:williamstevens@gmail.com]
> Sent: 31 January 2017 13:26
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> s/cloud-init/cloud-config/
>
> On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:
>
> > I think that is covered in this proposal. There is nothing k8s
> > specific in this integration (from what I understand), all the k8s
> > details are passed in via the cloud-init configuration after the
> > cluster
> has been provisioned.
> >
> > On Jan 31, 2017 3:06 AM, "Lianghwa Jou"
> > <li...@accelerite.com>
> > wrote:
> >
> >
> > There are many container orchestrators. Those container
> > orchestrators are happy to run on any VMs or bare metal machines.
> > K8s is just one of them and there will be more in the future. It may
> > not be a good idea to make CloudStack to be k8s aware. IMO, the
> > relationship between k8s and cloudstack should be similar to
> > application and os. It probably not a good idea to make your OS to
> > be aware of any specific applications so IMHO I don’t think k8s should be native to CloudStack.
> > It makes more sense to provide some generic services that many
> > applications can take advantages of. Some generic resource grouping
> > service makes sense so a group of VMs, baremetal machines or network
> > can be treated as a single entity. Some life cycle management will
> > be necessary for these entities too. We can deploy k8s, swarm, dcos
> > or even non-container specific services on top of CloudStack. The
> > k8s is changing fast. One single tenant installation may need more
> > than one VM group and may actually requires more (k8s federation).
> > It will be a struggle to be in sync if we try to bring k8s specific
> > knowledge into
> cloudstack.
> >
> > Regards,
> >
> >
> > --
> > Lianghwa Jou
> > VP Engineering, Accelerite
> > email: lianghwa.jou@accelerite.com
> >
> >
> >
> >
> >
> > On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
> >
> >
> > I agree with some good views Will has shared and I also agree to
> > the concerns raised by Wido and Eric. IMO we need balance of staying
> > relevant/add new features vs stability of CloudStack and take
> > corrective action if needed. We have two good examples for both.
> > When SDN was hot technology CloudStack ended up with bunch of SDN
> > controller
> integrations.
> > Few years later, now CloudStack is carrying baggage with no
> > maintainers for those plugins, with probably not many of CloudStack
> users needing overlays.
> > On the other hand, other than attempt to liaison with ETSI for NFV
> > no effort was done. OpenStack has become de-facto for NFV. Now for
> > OpenStack, arguably NFV has become larger use case than cloud it self.
> > I concur with Will’s point that with minimal viable solution that
> > does not change the core of CloudStack, and can keep CloudStack
> > relevant is worth to be taken in.
> >
> > Will,
> >
> > To your question of how different is from ShapeBlue’s container
> > service, its design, implementation and API semantics etc remain same.
> > ShapeBlue’s container service was true drop in plug-in to
> > CloudStack, with this proposal I am trying to re-work to make it a
> > core CloudStack pluggable service which is part of CloudStack.
> >
> > Key concepts that this proposal is trying to add
> >
> > - add notion of ‘container cluster’ as a first class entity
> > in CloudStack. Which is bacially collection of other CloudStack
> > resources (like VM’s, networks, public ip, network rules etc)
> > - life cycle operation to manage ‘container cluster’ like
> > create, delete, start, stop, scale-up, scale-down, heal etc
> > - orchestrate container orchestrator control plane on top of
> > provisioned resources
> >
> > At-least for k8s (which is what this proposal is targeting),
> > integration with k8s is bare minimum. There are cloud-config scripts
> > that automatically setup k8s cluster master and node VM’s. All
> > CloudStack is doing in passing the cloud-config to the core OS VM’s
> > as
> user data.
> >
> > Regards
> > Murali Reddy
> >
> >
> >
> >
> >
> >
> >
> > On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com
> > on behalf of wstevens@cloudops.com> wrote:
> >
> > >I agree that we need to be careful what we take on and own inside
> > >CloudStack. I feel like some of the plugins or integrations
> > which we have
> > >been "maintaining" may serve us better to abandon, but I feel
> > like that is
> > >a whole discussion on its own.
> > >
> > >In this case, I feel like there is a minimum viable solution
> > which puts
> > >CloudStack in a pretty good place to enable container orchestration.
> > For
> > >example, one of the biggest challenges with K8S is the fact
> > that it
> is
> > >single tenant. CloudStack has good multi tenancy support and
> > has
> the
> > >ability to orchestrate the underlying infra quite well. We
> > will have to be
> > >very careful not to try to own too deep into the K8S world
> > though, in my
> > >opinion. We only want to be responsible for providing the
> > infra (and a way
> > >to bootstrap K8S ideally) and be able to scale the infra,
> > everything else
> > >should be owned by the K8S on top. That is the way I see it
> > anyway, but
> > >please add your input.
> > >
> > >I think it is a liability to try to go too deep, for the same
> > reasons Wido
> > >and Erik have mentioned. But I also think we need to take it
> > seriously
> > >because that train is moving and this may be a good opportunity
> > to stay
> > >relevant in a rapidly changing market.
> > >
> > >*Will STEVENS*
> > >Lead Developer
> > >
> > ><https://goo.gl/NYZ8KK>
> > >
> > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > <wi...@widodh.nl>
> > wrote:
> > >
> > >>
> > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > wstevens@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > Hey Murali,
> > >> > How different is this proposal than what ShapeBlue already
> > built. It
> > >> looks
> > >> > pretty consistent with the functionality that you guys open
> > sourced in
> > >> > Seville.
> > >> >
> > >> > I have not yet used this functionality, but I have reports
> > that it works
> > >> > quite well.
> > >> >
> > >> > I believe the premise here is to only orchestrate the VM
> > layer
> and
> > >> > basically expose a "group" of running VMs to the user. The
> > user is
> > >> > responsible for configuring K8S or whatever other container
> > orchestrator
> > >> on
> > >> > top. I saw mention of the "cloud-config" scripts in the
> > FS, how are
> > >> those
> > >> > exposed to the cluster? Maybe the FS can expand on that a bit?
> > >> >
> > >> > I believe the core feature that is being requested to be
> > added is the
> > >> > ability to create a group of VMs which will be kept active
> > as a group if
> > >> at
> > >> > all possible. ACS would be responsible for making sure
> > that the number
> > >> of
> > >> > VMs specified for the group are in running state and it
> > would spin up new
> > >> > VMs as needed in order to satisfy the group settings. In
> > general, it is
> > >> > understood that any application running on this group would
> > have to be
> > >> > fault tolerant enough to be able to rediscover a new VM if
> > one fails and
> > >> is
> > >> > replaced by a fresh copy. Is that fair to say? How is it
> > expected that
> > >> > this service discovery is done, just by VMs being present
> > on the network?
> > >> >
> > >> > As for some of the other people's concerns in this thread.
> > >> >
> > >> > - Regarding Wido's remarks. I understand that there is
> > some
> added
> > >> > complexity, but I don't feel like the scope of the addition is
> > >> > unrealistic. I think the LXC integration was a lot farther
> > out of the
> > >> > scope of what ACS does then this is. This does not change
> > the "things"
> > >> > which ACS orchestrates, it just adds the concept of a
> > grouping of things
> > >> > which ACS already manages. I think this is the right
> > approach since it
> > >> is
> > >> > not trying to be a container orchestrator. We will never
> > compete with
> > >> K8S,
> > >> > for example, and we should not try, but K8S is here and the
> > market wants
> > >> > it. I do think we should be keeping our head up about that
> > fact because
> > >> > being able to provide a the underlay for K8S is very
> > valuable in the
> > >> > current marketplace. I see this functionality as a way to
> > enable K8S
> > >> > adoption on top of ACS without changing our core values.
> > >> >
> > >> > - Regarding Erik's remarks. The container space is moving
> > fast, but so
> > >> is
> > >> > the industry. If we want to remain relevant, we need to be
> > able to
> > >> adapt a
> > >> > bit. I don't think this is a big shift in what we do, but
> > it is one that
> > >> > enables people to be able to start running with something
> > like K8S on top
> > >> > of their existing ACS. This is something we are interested
> > in doing and
> > >> so
> > >> > are our customers. If we can have a thin layer in ACS
> > which helps enable
> > >> > the use of K8S (or other container orchestrators) by
> orchestrating
> > >> > infrastructure, as we already do, and making it easier to
> > adopt
> a
> > >> container
> > >> > orchestrator running on top of ACS, I think that gives us a
> > nice foothold
> > >> > in the market. I don't really feel it is fair to compare
> > containers to
> > >> > IPv6. IPv6 has been out forever and it has taken almost a
> > decade to get
> > >> > anyone to adopt it. Containers have really only been here
> > for like 2
> > >> years
> > >> > and they are changing the market landscape in a very real way.
> > >> >
> > >> > Kind of on topic and kind of off topic. I think
> > understanding
> our
> > >> approach
> > >> > to containers is going to be important for the ACS
> > community as a whole.
> > >> > If we don't offer that market anything, then we will not be
> > considered
> > >> and
> > >> > we will lose market share we can't afford to lose. If we
> > try to hitch
> > >> our
> > >> > horse to that cart too much, we will not be able to be
> > agile enough and
> > >> > will fail. I feel like the right approach is for us to
> > know that it is a
> > >> > thriving market and continue to do what we do, but to
> > extend an olive
> > >> > branch to that market. I think this sort of implementation
> > is the right
> > >> > approach because we are not trying to do too much. We are
> > simply giving
> > >> a
> > >> > foundation on which the next big thing in the container
> > orchestration
> > >> world
> > >> > can adopt without us having to compete directly in that
> > space. I think
> > >> we
> > >> > have to focus on what we do best, but at the same time,
> > think about what
> > >> we
> > >> > can do to enable that huge market of users to adopt ACS as their
> > >> > foundation. The ability to offer VMs and containers in the
> > same data
> > >> plane
> > >> > is something we have the ability to do, especially with
> > this approach,
> > >> and
> > >> > is something that most other softwares can not do. The
> > adoption of
> > >> > containers by bigger organizations will be only part of
> > their workload,
> > >> > they will still be running VMs for the foreseeable future.
> > Being able to
> > >> > appeal to that market is going to be important for us.
> > >> >
> > >> > Hopefully I don't have too many strong opinions here, but I
> > do think we
> > >> > need to be thinking about how we move forward in a world
> > which
> is
> > >> adopting
> > >> > containers in a very real way.
> > >> >
> > >>
> > >> Understood. I just want to prevent that we add more features
> > to CloudStack
> > >> which are poorly maintained. Not judging Murali here at all,
> > but I'd rather
> > >> see CloudStack loose code then have it added.
> > >>
> > >> Thinking about LXC, would like to see that removed together
> > with various
> > >> other network plugins which I think are rarely used.
> > >>
> > >> Now, the idea of Murali was misunderstood by me. I think it
> > would be worth
> > >> more if we would improve our cloud-init support and
> > integration in various
> > >> tools which makes it much easier to deploy VMs running
> > containers
> ON
> > >> CloudStack.
> > >>
> > >> Not so much changing CloudStack code, but rather tooling
> > around
> it.
> > >>
> > >> If we have CloudStack talking to Kubernetes we suddenly have
> > to maintain
> > >> that API integration. Who's going to do that. Realistically.
> > >>
> > >> That's my main worry in this regard.
> > >>
> > >> We have so much more work to do in ACS in total that I don't
> > know if this
> > >> is the realistic route. I talk to many people who are not
> > looking
> at
> > >> containers and are still working with VMs.
> > >>
> > >> There is not a single truth which is true, it really depends
> > on who you
> > >> ask.
> > >>
> > >> Wido
> > >>
> > >> > Cheers,
> > >> >
> > >> > Will
> > >> >
> > >> > *Will STEVENS*
> > >> > Lead Developer
> > >> >
> > >> > <https://goo.gl/NYZ8KK>
> > >> >
> > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > <te...@gmail.com>
> > wrote:
> > >> >
> > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > muralimmreddy@gmail.com
> > >> >
> > >> > > wrote:
> > >> > > > All,
> > >> > > >
> > >> > > > I would like propose native functionality into
> > CloudStack to provide
> > >> a
> > >> > > container service through which users out-of-the box can
> > use to launch
> > >> > > container based application. Idea is to support ability
> > to orchestrate
> > >> the
> > >> > > resources and automate aspects of setting up container
> > orchestrator
> > >> through
> > >> > > CloudStack. Public IAAS service providers AWS with its
> > ECS [1] and
> > >> google
> > >> > > with GKE [2] already provides ability container applications.
> > >> Competitive
> > >> > > cloud orchestration platforms already have native support
> > for container
> > >> > > service. Users of CloudStack both as public cloud
> > providers and users
> > >> with
> > >> > > private clouds will benefit with such functionality.
> > >> > > >
> > >> > > > While container orchestrator of user choice can be
> > provisioned on
> > >> top of
> > >> > > CloudStack (with out CloudStack being involved) with
> > tools
> like
> > >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> > orchestration
> > >> is
> > >> > > giving user a nice cohesive integration. This proposal
> > would like add a
> > >> > > notion of first class CloudStack entity called container
> > cluster which
> > >> can
> > >> > > be used to provision resources, scale up, scale down,
> > start and stop
> > >> the
> > >> > > cluster of VM’s on which containerised applications can
> > be
> run.
> > For
> > >> actual
> > >> > > container orchestration we will still need container
> > orchestrator like
> > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > container service
> > >> can
> > >> > > automate setting up of control place automatically.
> > >> > > >
> > >> > >
> > >> > > To be honest I'm torn on this one.
> > >> > >
> > >> > > Containers are a rapid changing thing, and while docker swam,
> > >> > > kubernetes, rancher or whatnot is popular today, they
> > might not be
> > >> > > tomorrow.
> > >> > > They might use CoreOS today, but might not tomorrow.
> > >> > >
> > >> > > We have a rather poor track record of staying up to date
> > with new
> > >> > > features/versions, and adding a feature that is so
> > rapidly changing
> > >> > > is, I fear, going to be hard to maintain.
> > >> > > Want an example, look at xenserver. It is one of the most used
> > >> > > hypervisors we support, yet it took 6 months or so for us
> > to support
> > >> > > the latest release.
> > >> > > Or IPv6...
> > >> > >
> > >> > > I don't mean to bash at maintainers/implementers of those
> > features, I
> > >> > > appreciate all the work being done in every aspect, but I
> > believe we
> > >> > > should be realistic and realize that we have issues with
> > keeping stuff
> > >> > > up to date.
> > >> > >
> > >> > > I'd say focus on making sure other tools can do their job
> > well against
> > >> > > CloudStack (kops, rancher, ++), but that does not mean I
> > will
> > -1 the
> > >> > > idea if anyone really wants to go through with it.
> > >> > >
> > >> > > --
> > >> > > Erik
> > >> > >
> > >>
> >
> > murali.reddy@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> > which is the property of Accelerite, a Persistent Systems business.
> > It is intended only for the use of the individual or entity to which
> > it is addressed. If you are not the intended recipient, you are not
> > authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Accelerite,
> > a Persistent Systems business does not accept any liability for
> > virus
> infected mails.
> >
> >
> >
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not
> authorized to read, retain, copy, print, distribute or use this
> message. If you have received this communication in error, please
> notify the sender and delete all copies of this message. Accelerite, a
> Persistent Systems business does not accept any liability for virus infected mails.
>
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: [PROPOSAL] add native vm-cluster orchestration service (was:
[PROPOSAL] add native container orchestration service)
Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 20 februari 2017 om 14:56 schreef Daan Hoogland <da...@shapeblue.com>:
>
>
> So, being very late in the discussion but havingread the whole thread before editting the title of this thread,
>
> Can we agree that we want a generic vm-cluster service and leave the container bits to containers? Kishan can you share your design? Shapeblue wants to rebase their k8 service on top of this and I would like yours and Murali's work to not conflict.
>
I like that a lot more. This way it wouldn't only be containers, but think of spawning a cluster of VMs together which form a MariaDB Galera cluster for example.
Such features are very welcome!
Wido
> daan.hoogland@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, Utrecht Utrecht 3531 VENetherlands
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Paul Angus [mailto:paul.angus@shapeblue.com]
> Sent: dinsdag 7 februari 2017 08:14
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> Will is 100% correct. As I mentioned the Title is misleading. However, Murali did clarify in his explanation; this is really about vm cluster orchestration.
>
>
>
> ________________________________
> From: Will Stevens <ws...@cloudops.com>
> Sent: 6 Feb 2017 22:54
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] add native container orchestration service
>
> My understanding is that what Paul is talking about is what is already built and IS what the thread is talking about.
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Mon, Feb 6, 2017 at 2:29 PM, Rajesh Ramchandani < rajesh.ramchandani@accelerite.com> wrote:
>
> > Hi Paul - I think this is different from what the thread was about.
> > The conversation was specifically about adding support for container
> > orchestrators. You are talking about provisioning a group of VMs.
> > Although, this is something I think several Cloudstack users have
> > requested before and we should propose a solution to this.
> >
> > Raj
> >
> > ________________________________
> > From: Paul Angus <pa...@shapeblue.com>
> > Sent: Monday, February 6, 2017 11:16:41 AM
> > To: dev@cloudstack.apache.org
> > Subject: RE: [PROPOSAL] add native container orchestration service
> >
> > #WhatHeSaid
> >
> > The title is misleading. The proposal is purely to add the notion of
> > Clusters of VMs to CloudStack. These may be for databases, containers
> > or anything else that needs 'clusters' of compute. Self-healing and
> > autoscaling are logical next steps to be added.
> >
> > Those guys at ShapeBlue have open-sourced their whole k8s container
> > service piece. If/when the 'cluster' part of that work is added into
> > CloudStack, the k8s specific pieces can be used by anyone who wants
> > to, alternatively they could be used for reference in order to create
> > another types of cluster. (or ignored completely).
> >
> >
> >
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Will Stevens [mailto:williamstevens@gmail.com]
> > Sent: 31 January 2017 13:26
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] add native container orchestration service
> >
> > s/cloud-init/cloud-config/
> >
> > On Jan 31, 2017 7:24 AM, "Will Stevens" <wi...@gmail.com> wrote:
> >
> > > I think that is covered in this proposal. There is nothing k8s
> > > specific in this integration (from what I understand), all the k8s
> > > details are passed in via the cloud-init configuration after the cluster
> > has been provisioned.
> > >
> > > On Jan 31, 2017 3:06 AM, "Lianghwa Jou" <li...@accelerite.com>
> > > wrote:
> > >
> > >
> > > There are many container orchestrators. Those container orchestrators
> > > are happy to run on any VMs or bare metal machines. K8s is just one of
> > > them and there will be more in the future. It may not be a good idea
> > > to make CloudStack to be k8s aware. IMO, the relationship between k8s
> > > and cloudstack should be similar to application and os. It probably
> > > not a good idea to make your OS to be aware of any specific
> > > applications so IMHO I don’t think k8s should be native to CloudStack.
> > > It makes more sense to provide some generic services that many
> > > applications can take advantages of. Some generic resource grouping
> > > service makes sense so a group of VMs, baremetal machines or network
> > > can be treated as a single entity. Some life cycle management will be
> > > necessary for these entities too. We can deploy k8s, swarm, dcos or
> > > even non-container specific services on top of CloudStack. The k8s is
> > > changing fast. One single tenant installation may need more than one
> > > VM group and may actually requires more (k8s federation). It will be a
> > > struggle to be in sync if we try to bring k8s specific knowledge into
> > cloudstack.
> > >
> > > Regards,
> > >
> > >
> > > --
> > > Lianghwa Jou
> > > VP Engineering, Accelerite
> > > email: lianghwa.jou@accelerite.com
> > >
> > >
> > >
> > >
> > >
> > > On 1/29/17, 11:54 PM, "Murali Reddy" <mu...@shapeblue.com> wrote:
> > >
> > >
> > > I agree with some good views Will has shared and I also agree to
> > > the concerns raised by Wido and Eric. IMO we need balance of staying
> > > relevant/add new features vs stability of CloudStack and take
> > > corrective action if needed. We have two good examples for both. When
> > > SDN was hot technology CloudStack ended up with bunch of SDN controller
> > integrations.
> > > Few years later, now CloudStack is carrying baggage with no
> > > maintainers for those plugins, with probably not many of CloudStack
> > users needing overlays.
> > > On the other hand, other than attempt to liaison with ETSI for NFV no
> > > effort was done. OpenStack has become de-facto for NFV. Now for
> > > OpenStack, arguably NFV has become larger use case than cloud it self.
> > > I concur with Will’s point that with minimal viable solution that does
> > > not change the core of CloudStack, and can keep CloudStack relevant is
> > > worth to be taken in.
> > >
> > > Will,
> > >
> > > To your question of how different is from ShapeBlue’s container
> > > service, its design, implementation and API semantics etc remain same.
> > > ShapeBlue’s container service was true drop in plug-in to CloudStack,
> > > with this proposal I am trying to re-work to make it a core CloudStack
> > > pluggable service which is part of CloudStack.
> > >
> > > Key concepts that this proposal is trying to add
> > >
> > > - add notion of ‘container cluster’ as a first class entity in
> > > CloudStack. Which is bacially collection of other CloudStack resources
> > > (like VM’s, networks, public ip, network rules etc)
> > > - life cycle operation to manage ‘container cluster’ like
> > > create, delete, start, stop, scale-up, scale-down, heal etc
> > > - orchestrate container orchestrator control plane on top of
> > > provisioned resources
> > >
> > > At-least for k8s (which is what this proposal is targeting),
> > > integration with k8s is bare minimum. There are cloud-config scripts
> > > that automatically setup k8s cluster master and node VM’s. All
> > > CloudStack is doing in passing the cloud-config to the core OS VM’s as
> > user data.
> > >
> > > Regards
> > > Murali Reddy
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 29/01/17, 8:54 AM, "Will Stevens" <williamstevens@gmail.com on
> > > behalf of wstevens@cloudops.com> wrote:
> > >
> > > >I agree that we need to be careful what we take on and own inside
> > > >CloudStack. I feel like some of the plugins or integrations
> > > which we have
> > > >been "maintaining" may serve us better to abandon, but I feel
> > > like that is
> > > >a whole discussion on its own.
> > > >
> > > >In this case, I feel like there is a minimum viable solution
> > > which puts
> > > >CloudStack in a pretty good place to enable container orchestration.
> > > For
> > > >example, one of the biggest challenges with K8S is the fact that it
> > is
> > > >single tenant. CloudStack has good multi tenancy support and has
> > the
> > > >ability to orchestrate the underlying infra quite well. We will
> > > have to be
> > > >very careful not to try to own too deep into the K8S world
> > > though, in my
> > > >opinion. We only want to be responsible for providing the infra
> > > (and a way
> > > >to bootstrap K8S ideally) and be able to scale the infra,
> > > everything else
> > > >should be owned by the K8S on top. That is the way I see it
> > > anyway, but
> > > >please add your input.
> > > >
> > > >I think it is a liability to try to go too deep, for the same
> > > reasons Wido
> > > >and Erik have mentioned. But I also think we need to take it
> > > seriously
> > > >because that train is moving and this may be a good opportunity
> > > to stay
> > > >relevant in a rapidly changing market.
> > > >
> > > >*Will STEVENS*
> > > >Lead Developer
> > > >
> > > ><https://goo.gl/NYZ8KK>
> > > >
> > > >On Sat, Jan 28, 2017 at 1:13 PM, Wido den Hollander
> > > <wi...@widodh.nl>
> > > wrote:
> > > >
> > > >>
> > > >> > Op 27 januari 2017 om 16:08 schreef Will Stevens <
> > > wstevens@cloudops.com
> > > >> >:
> > > >> >
> > > >> >
> > > >> > Hey Murali,
> > > >> > How different is this proposal than what ShapeBlue already
> > > built. It
> > > >> looks
> > > >> > pretty consistent with the functionality that you guys open
> > > sourced in
> > > >> > Seville.
> > > >> >
> > > >> > I have not yet used this functionality, but I have reports
> > > that it works
> > > >> > quite well.
> > > >> >
> > > >> > I believe the premise here is to only orchestrate the VM layer
> > and
> > > >> > basically expose a "group" of running VMs to the user. The
> > > user is
> > > >> > responsible for configuring K8S or whatever other container
> > > orchestrator
> > > >> on
> > > >> > top. I saw mention of the "cloud-config" scripts in the FS,
> > > how are
> > > >> those
> > > >> > exposed to the cluster? Maybe the FS can expand on that a bit?
> > > >> >
> > > >> > I believe the core feature that is being requested to be
> > > added is the
> > > >> > ability to create a group of VMs which will be kept active as
> > > a group if
> > > >> at
> > > >> > all possible. ACS would be responsible for making sure that
> > > the number
> > > >> of
> > > >> > VMs specified for the group are in running state and it would
> > > spin up new
> > > >> > VMs as needed in order to satisfy the group settings. In
> > > general, it is
> > > >> > understood that any application running on this group would
> > > have to be
> > > >> > fault tolerant enough to be able to rediscover a new VM if
> > > one fails and
> > > >> is
> > > >> > replaced by a fresh copy. Is that fair to say? How is it
> > > expected that
> > > >> > this service discovery is done, just by VMs being present on
> > > the network?
> > > >> >
> > > >> > As for some of the other people's concerns in this thread.
> > > >> >
> > > >> > - Regarding Wido's remarks. I understand that there is some
> > added
> > > >> > complexity, but I don't feel like the scope of the addition is
> > > >> > unrealistic. I think the LXC integration was a lot farther
> > > out of the
> > > >> > scope of what ACS does then this is. This does not change
> > > the "things"
> > > >> > which ACS orchestrates, it just adds the concept of a
> > > grouping of things
> > > >> > which ACS already manages. I think this is the right
> > > approach since it
> > > >> is
> > > >> > not trying to be a container orchestrator. We will never
> > > compete with
> > > >> K8S,
> > > >> > for example, and we should not try, but K8S is here and the
> > > market wants
> > > >> > it. I do think we should be keeping our head up about that
> > > fact because
> > > >> > being able to provide a the underlay for K8S is very valuable
> > > in the
> > > >> > current marketplace. I see this functionality as a way to
> > > enable K8S
> > > >> > adoption on top of ACS without changing our core values.
> > > >> >
> > > >> > - Regarding Erik's remarks. The container space is moving
> > > fast, but so
> > > >> is
> > > >> > the industry. If we want to remain relevant, we need to be
> > > able to
> > > >> adapt a
> > > >> > bit. I don't think this is a big shift in what we do, but it
> > > is one that
> > > >> > enables people to be able to start running with something
> > > like K8S on top
> > > >> > of their existing ACS. This is something we are interested
> > > in doing and
> > > >> so
> > > >> > are our customers. If we can have a thin layer in ACS which
> > > helps enable
> > > >> > the use of K8S (or other container orchestrators) by
> > orchestrating
> > > >> > infrastructure, as we already do, and making it easier to adopt
> > a
> > > >> container
> > > >> > orchestrator running on top of ACS, I think that gives us a
> > > nice foothold
> > > >> > in the market. I don't really feel it is fair to compare
> > > containers to
> > > >> > IPv6. IPv6 has been out forever and it has taken almost a
> > > decade to get
> > > >> > anyone to adopt it. Containers have really only been here
> > > for like 2
> > > >> years
> > > >> > and they are changing the market landscape in a very real way.
> > > >> >
> > > >> > Kind of on topic and kind of off topic. I think understanding
> > our
> > > >> approach
> > > >> > to containers is going to be important for the ACS community
> > > as a whole.
> > > >> > If we don't offer that market anything, then we will not be
> > > considered
> > > >> and
> > > >> > we will lose market share we can't afford to lose. If we try
> > > to hitch
> > > >> our
> > > >> > horse to that cart too much, we will not be able to be agile
> > > enough and
> > > >> > will fail. I feel like the right approach is for us to know
> > > that it is a
> > > >> > thriving market and continue to do what we do, but to extend
> > > an olive
> > > >> > branch to that market. I think this sort of implementation
> > > is the right
> > > >> > approach because we are not trying to do too much. We are
> > > simply giving
> > > >> a
> > > >> > foundation on which the next big thing in the container
> > > orchestration
> > > >> world
> > > >> > can adopt without us having to compete directly in that
> > > space. I think
> > > >> we
> > > >> > have to focus on what we do best, but at the same time, think
> > > about what
> > > >> we
> > > >> > can do to enable that huge market of users to adopt ACS as their
> > > >> > foundation. The ability to offer VMs and containers in the
> > > same data
> > > >> plane
> > > >> > is something we have the ability to do, especially with this
> > > approach,
> > > >> and
> > > >> > is something that most other softwares can not do. The
> > > adoption of
> > > >> > containers by bigger organizations will be only part of their
> > > workload,
> > > >> > they will still be running VMs for the foreseeable future.
> > > Being able to
> > > >> > appeal to that market is going to be important for us.
> > > >> >
> > > >> > Hopefully I don't have too many strong opinions here, but I
> > > do think we
> > > >> > need to be thinking about how we move forward in a world which
> > is
> > > >> adopting
> > > >> > containers in a very real way.
> > > >> >
> > > >>
> > > >> Understood. I just want to prevent that we add more features to
> > > CloudStack
> > > >> which are poorly maintained. Not judging Murali here at all,
> > > but I'd rather
> > > >> see CloudStack loose code then have it added.
> > > >>
> > > >> Thinking about LXC, would like to see that removed together
> > > with various
> > > >> other network plugins which I think are rarely used.
> > > >>
> > > >> Now, the idea of Murali was misunderstood by me. I think it
> > > would be worth
> > > >> more if we would improve our cloud-init support and integration
> > > in various
> > > >> tools which makes it much easier to deploy VMs running containers
> > ON
> > > >> CloudStack.
> > > >>
> > > >> Not so much changing CloudStack code, but rather tooling around
> > it.
> > > >>
> > > >> If we have CloudStack talking to Kubernetes we suddenly have to
> > > maintain
> > > >> that API integration. Who's going to do that. Realistically.
> > > >>
> > > >> That's my main worry in this regard.
> > > >>
> > > >> We have so much more work to do in ACS in total that I don't
> > > know if this
> > > >> is the realistic route. I talk to many people who are not looking
> > at
> > > >> containers and are still working with VMs.
> > > >>
> > > >> There is not a single truth which is true, it really depends on
> > > who you
> > > >> ask.
> > > >>
> > > >> Wido
> > > >>
> > > >> > Cheers,
> > > >> >
> > > >> > Will
> > > >> >
> > > >> > *Will STEVENS*
> > > >> > Lead Developer
> > > >> >
> > > >> > <https://goo.gl/NYZ8KK>
> > > >> >
> > > >> > On Fri, Jan 27, 2017 at 5:38 AM, Erik Weber
> > > <te...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > On Fri, Jan 27, 2017 at 7:20 AM, Murali Reddy <
> > > muralimmreddy@gmail.com
> > > >> >
> > > >> > > wrote:
> > > >> > > > All,
> > > >> > > >
> > > >> > > > I would like propose native functionality into CloudStack
> > > to provide
> > > >> a
> > > >> > > container service through which users out-of-the box can
> > > use to launch
> > > >> > > container based application. Idea is to support ability to
> > > orchestrate
> > > >> the
> > > >> > > resources and automate aspects of setting up container
> > > orchestrator
> > > >> through
> > > >> > > CloudStack. Public IAAS service providers AWS with its ECS
> > > [1] and
> > > >> google
> > > >> > > with GKE [2] already provides ability container applications.
> > > >> Competitive
> > > >> > > cloud orchestration platforms already have native support
> > > for container
> > > >> > > service. Users of CloudStack both as public cloud providers
> > > and users
> > > >> with
> > > >> > > private clouds will benefit with such functionality.
> > > >> > > >
> > > >> > > > While container orchestrator of user choice can be
> > > provisioned on
> > > >> top of
> > > >> > > CloudStack (with out CloudStack being involved) with tools
> > like
> > > >> > > TerraForm[3], Ansible[4] etc, advantage of having native
> > > orchestration
> > > >> is
> > > >> > > giving user a nice cohesive integration. This proposal
> > > would like add a
> > > >> > > notion of first class CloudStack entity called container
> > > cluster which
> > > >> can
> > > >> > > be used to provision resources, scale up, scale down, start
> > > and stop
> > > >> the
> > > >> > > cluster of VM’s on which containerised applications can be
> > run.
> > > For
> > > >> actual
> > > >> > > container orchestration we will still need container
> > > orchestrator like
> > > >> > > docker swarm, marathon, kubernetes, but CloudStack
> > > container service
> > > >> can
> > > >> > > automate setting up of control place automatically.
> > > >> > > >
> > > >> > >
> > > >> > > To be honest I'm torn on this one.
> > > >> > >
> > > >> > > Containers are a rapid changing thing, and while docker swam,
> > > >> > > kubernetes, rancher or whatnot is popular today, they might
> > > not be
> > > >> > > tomorrow.
> > > >> > > They might use CoreOS today, but might not tomorrow.
> > > >> > >
> > > >> > > We have a rather poor track record of staying up to date
> > > with new
> > > >> > > features/versions, and adding a feature that is so rapidly
> > > changing
> > > >> > > is, I fear, going to be hard to maintain.
> > > >> > > Want an example, look at xenserver. It is one of the most used
> > > >> > > hypervisors we support, yet it took 6 months or so for us
> > > to support
> > > >> > > the latest release.
> > > >> > > Or IPv6...
> > > >> > >
> > > >> > > I don't mean to bash at maintainers/implementers of those
> > > features, I
> > > >> > > appreciate all the work being done in every aspect, but I
> > > believe we
> > > >> > > should be realistic and realize that we have issues with
> > > keeping stuff
> > > >> > > up to date.
> > > >> > >
> > > >> > > I'd say focus on making sure other tools can do their job
> > > well against
> > > >> > > CloudStack (kops, rancher, ++), but that does not mean I
> > > will
> > > -1 the
> > > >> > > idea if anyone really wants to go through with it.
> > > >> > >
> > > >> > > --
> > > >> > > Erik
> > > >> > >
> > > >>
> > >
> > > murali.reddy@shapeblue.com
> > > www.shapeblue.com<http://www.shapeblue.com>
> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information which
> > > is the property of Accelerite, a Persistent Systems business. It is
> > > intended only for the use of the individual or entity to which it is
> > > addressed. If you are not the intended recipient, you are not
> > > authorized to read, retain, copy, print, distribute or use this
> > > message. If you have received this communication in error, please
> > > notify the sender and delete all copies of this message. Accelerite, a
> > > Persistent Systems business does not accept any liability for virus
> > infected mails.
> > >
> > >
> > >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which is
> > the property of Accelerite, a Persistent Systems business. It is intended
> > only for the use of the individual or entity to which it is addressed. If
> > you are not the intended recipient, you are not authorized to read, retain,
> > copy, print, distribute or use this message. If you have received this
> > communication in error, please notify the sender and delete all copies of
> > this message. Accelerite, a Persistent Systems business does not accept any
> > liability for virus infected mails.
> >
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>