Mesos integration

[Tyson Norris <tn...@adobe.com.INVALID>]

Hi All - 
I’ve been considering how we can leverage mesos resource scheduling within OpenWhisk, and wanted to share some ideas and solicit some feedback. 

In general, I’ve been experimenting with an approach that replaces the DockerContainer implementation with a MesosTask. The MesosTask will have effectively the same lifecycle (ignoring pause/resume functions) and the same interface (HTTP api to the container), the main difference being that the container will be deployed to some arbitrary host in the cluster. 

There are a few broad topics around this, including:
- docker client usage needs to be better isolated (e.g. cleaning up existing containers, reconciling containers already running at time of invoker startup, etc). I think this will be mostly straightforward.
- log collection - I’m not sure the best approach here, but one is to completely decouple log collection from activation execution, and then provide a mesos-specific impl
- container state tracking + load balancing - obviously there is potential for conflict if 2 invokers schedule activations on the same container in the cluster (since the state tracking of the container would still be in the invoker). This would imply some extension to the ContainerPool as well.
- mesos framework - we’ve discussed internally a bit, and some preferences I have are: leverage the mesos http api (avoid the older jni libs if possible) and provide an independent framework application which provides a (simpler)  http api that is consumed by the invoker (if mesos integration is enabled in the deployment). This way the framework deployment can be isolated from controller/invoker deployment, and interaction with docker containers is mostly the same (except for logging).  
- There are a few options of implementing mesos http api clients like: rxjava client, and nodejs client, but I've not seen any good Scala client to date, so we may either provide a scala app using the rxjava client, or create.a new client in Scala.  

Let me know if you have any thoughts around any of these, and I will share more details as they come.

Thanks
Tyson

Re: Mesos integration

[Tyson Norris <tn...@adobe.com.INVALID>]

Thanks.

One thing I think will help is isolating the http contract with containers from the management (start/stop etc) of the container. (Currently the http api client is built into the DockerContainer class)

After that, providing an interface for a configured container management would be able to dictate the approach to use in a particular env. Note that this would not preclude using the “docker” container management within a mesos or kubernetes env, if people wanted to still do that (there will be tradeoffs, like start/stop time vs cluster-level pool visibility).  

RE: mesos details:I have some concerns with using existing java libs from Scala - the old libs all use libmesos.so, which can be a nuisance to deal with, and the newer libs rely on some java method overloading which brings up some compiler issues in Scala. I haven’t exhaustively looked at it, but currently I’m testing out the mesos-rxjava lib. If anyone has opinions or experience with this lib, let me know. (I work with Dragos BTW, so will be soliciting feedback from him regularly)



Thanks
Tyson  


> On May 31, 2017, at 3:19 PM, Carlos Santana <cs...@gmail.com> wrote:
> 
> Yeah I agree between Mesos and Kubernetes work there should be some common
> integration extension points that would serve both use cases to some extent
> 
> 
> 
> On Wed, May 31, 2017 at 5:42 PM Ben Browning <bb...@redhat.com> wrote:
> 
>> I believe a lot of Tyson's observations around Mesos integration also
>> apply to Kubernetes integration. For example, I've been experimenting
>> with an approach that replaces the DockerContainer with a
>> KubernetesContainer. The KubernetesContainer also ignores the pause
>> and resume of functions and also deploys containers to arbitrary nodes
>> in the Kubernetes cluster.
>> 
>> Here's a rough but working (except for log shipping) prototype of that
>> -
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprojectodd%2Fincubator-openwhisk%2Fblob%2Fkube-container%2Fcore%2Finvoker%2Fsrc%2Fmain%2Fscala%2Fwhisk%2Fcore%2Fcontainerpool%2Fkubernetes%2FKubernetesContainer.scala&data=02%7C01%7C%7C869495e5dcfb4facf10d08d4a8732801%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636318659922108109&sdata=K%2BAKqn87gawixF2GdofV5ik3ETosndqPSbhVO5mPfBY%3D&reserved=0
>> 
>> 
>> So, I'd like to add a +1 Tyson's notes around isolating docker client
>> usage and log collection. I believe they both equally apply to a
>> Kubernetes environment as well. And, more broadly, we'll need to
>> figure out how to make the various Container implementations pluggable
>> (or at least configurable) for Docker, Kubernetes, & Mesos.
>> 
>> Ben
>> 
>> 
>> On Wed, May 31, 2017 at 5:24 PM, Carlos Santana <cs...@gmail.com>
>> wrote:
>>> Hi Tyson
>>> 
>>> Thanks for reaching out, having Mesos integration in OpenWhisk is
>> something
>>> I have heard some users asking about and any help to figure out how to
>> best
>>> integrate are very welcome.
>>> 
>>> As you already aware OpenWhisk Invoker is a critical path component
>>> optimized for performance, to get those container pause and unpause in
>>> single digit milliseconds that's the reason for the recent change from
>>> using docker engine down to runc
>>> 
>>> Here is my initial notes, take into account I don't have first hand
>>> experience with Mesos, which is why is good to have folks like you join
>> :-)
>>> 
>>> I think extending in a way that a different mode can be use would be
>> useful
>>> to plug a different orchestration methodology.
>>> 
>>> For log collection we are looking into sending logs into ELK (Elastic
>>> search) there is some early  investigation going on, so I see after this
>>> integration, mesos can be told to which ELK to send the logs, and the
>>> Invoker to get the results.
>>> 
>>> In terms of scala vs. nodejs, I would recommend to use scala with the
>> java
>>> client, since there could be some components being integrated into
>> Invoker
>>> and code sharing.
>>> 
>>> How you want to follow up, maybe you have something to show us or maybe
>>> setup a hangout call to meet
>>> 
>>> I think Dragos from Adobe, might have a lot more feedback than me since I
>>> think he has first hand experience
>>> 
>>> --Carlos
>>> 
>>> 
>>> 
>>> On Sat, May 27, 2017 at 2:44 PM Tyson Norris <tn...@adobe.com.invalid>
>>> wrote:
>>> 
>>>> Hi All -
>>>> I’ve been considering how we can leverage mesos resource scheduling
>> within
>>>> OpenWhisk, and wanted to share some ideas and solicit some feedback.
>>>> 
>>>> In general, I’ve been experimenting with an approach that replaces the
>>>> DockerContainer implementation with a MesosTask. The MesosTask will have
>>>> effectively the same lifecycle (ignoring pause/resume functions) and the
>>>> same interface (HTTP api to the container), the main difference being
>> that
>>>> the container will be deployed to some arbitrary host in the cluster.
>>>> 
>>>> There are a few broad topics around this, including:
>>>> - docker client usage needs to be better isolated (e.g. cleaning up
>>>> existing containers, reconciling containers already running at time of
>>>> invoker startup, etc). I think this will be mostly straightforward.
>>>> - log collection - I’m not sure the best approach here, but one is to
>>>> completely decouple log collection from activation execution, and then
>>>> provide a mesos-specific impl
>>>> - container state tracking + load balancing - obviously there is
>> potential
>>>> for conflict if 2 invokers schedule activations on the same container in
>>>> the cluster (since the state tracking of the container would still be in
>>>> the invoker). This would imply some extension to the ContainerPool as
>> well.
>>>> - mesos framework - we’ve discussed internally a bit, and some
>> preferences
>>>> I have are: leverage the mesos http api (avoid the older jni libs if
>>>> possible) and provide an independent framework application which
>> provides a
>>>> (simpler)  http api that is consumed by the invoker (if mesos
>> integration
>>>> is enabled in the deployment). This way the framework deployment can be
>>>> isolated from controller/invoker deployment, and interaction with docker
>>>> containers is mostly the same (except for logging).
>>>> - There are a few options of implementing mesos http api clients like:
>>>> rxjava client, and nodejs client, but I've not seen any good Scala
>> client
>>>> to date, so we may either provide a scala app using the rxjava client,
>> or
>>>> create.a new client in Scala.
>>>> 
>>>> Let me know if you have any thoughts around any of these, and I will
>> share
>>>> more details as they come.
>>>> 
>>>> Thanks
>>>> Tyson
>>>> 
>>>> 
>>>> 
>> 

Re: Mesos integration

[Carlos Santana <cs...@gmail.com>]

Yeah I agree between Mesos and Kubernetes work there should be some common
integration extension points that would serve both use cases to some extent



On Wed, May 31, 2017 at 5:42 PM Ben Browning <bb...@redhat.com> wrote:

> I believe a lot of Tyson's observations around Mesos integration also
> apply to Kubernetes integration. For example, I've been experimenting
> with an approach that replaces the DockerContainer with a
> KubernetesContainer. The KubernetesContainer also ignores the pause
> and resume of functions and also deploys containers to arbitrary nodes
> in the Kubernetes cluster.
>
> Here's a rough but working (except for log shipping) prototype of that
> -
> https://github.com/projectodd/incubator-openwhisk/blob/kube-container/core/invoker/src/main/scala/whisk/core/containerpool/kubernetes/KubernetesContainer.scala
>
>
> So, I'd like to add a +1 Tyson's notes around isolating docker client
> usage and log collection. I believe they both equally apply to a
> Kubernetes environment as well. And, more broadly, we'll need to
> figure out how to make the various Container implementations pluggable
> (or at least configurable) for Docker, Kubernetes, & Mesos.
>
> Ben
>
>
> On Wed, May 31, 2017 at 5:24 PM, Carlos Santana <cs...@gmail.com>
> wrote:
> > Hi Tyson
> >
> > Thanks for reaching out, having Mesos integration in OpenWhisk is
> something
> > I have heard some users asking about and any help to figure out how to
> best
> > integrate are very welcome.
> >
> > As you already aware OpenWhisk Invoker is a critical path component
> > optimized for performance, to get those container pause and unpause in
> > single digit milliseconds that's the reason for the recent change from
> > using docker engine down to runc
> >
> > Here is my initial notes, take into account I don't have first hand
> > experience with Mesos, which is why is good to have folks like you join
> :-)
> >
> > I think extending in a way that a different mode can be use would be
> useful
> > to plug a different orchestration methodology.
> >
> > For log collection we are looking into sending logs into ELK (Elastic
> > search) there is some early  investigation going on, so I see after this
> > integration, mesos can be told to which ELK to send the logs, and the
> > Invoker to get the results.
> >
> > In terms of scala vs. nodejs, I would recommend to use scala with the
> java
> > client, since there could be some components being integrated into
> Invoker
> > and code sharing.
> >
> > How you want to follow up, maybe you have something to show us or maybe
> > setup a hangout call to meet
> >
> > I think Dragos from Adobe, might have a lot more feedback than me since I
> > think he has first hand experience
> >
> > --Carlos
> >
> >
> >
> > On Sat, May 27, 2017 at 2:44 PM Tyson Norris <tn...@adobe.com.invalid>
> > wrote:
> >
> >> Hi All -
> >> I’ve been considering how we can leverage mesos resource scheduling
> within
> >> OpenWhisk, and wanted to share some ideas and solicit some feedback.
> >>
> >> In general, I’ve been experimenting with an approach that replaces the
> >> DockerContainer implementation with a MesosTask. The MesosTask will have
> >> effectively the same lifecycle (ignoring pause/resume functions) and the
> >> same interface (HTTP api to the container), the main difference being
> that
> >> the container will be deployed to some arbitrary host in the cluster.
> >>
> >> There are a few broad topics around this, including:
> >> - docker client usage needs to be better isolated (e.g. cleaning up
> >> existing containers, reconciling containers already running at time of
> >> invoker startup, etc). I think this will be mostly straightforward.
> >> - log collection - I’m not sure the best approach here, but one is to
> >> completely decouple log collection from activation execution, and then
> >> provide a mesos-specific impl
> >> - container state tracking + load balancing - obviously there is
> potential
> >> for conflict if 2 invokers schedule activations on the same container in
> >> the cluster (since the state tracking of the container would still be in
> >> the invoker). This would imply some extension to the ContainerPool as
> well.
> >> - mesos framework - we’ve discussed internally a bit, and some
> preferences
> >> I have are: leverage the mesos http api (avoid the older jni libs if
> >> possible) and provide an independent framework application which
> provides a
> >> (simpler)  http api that is consumed by the invoker (if mesos
> integration
> >> is enabled in the deployment). This way the framework deployment can be
> >> isolated from controller/invoker deployment, and interaction with docker
> >> containers is mostly the same (except for logging).
> >> - There are a few options of implementing mesos http api clients like:
> >> rxjava client, and nodejs client, but I've not seen any good Scala
> client
> >> to date, so we may either provide a scala app using the rxjava client,
> or
> >> create.a new client in Scala.
> >>
> >> Let me know if you have any thoughts around any of these, and I will
> share
> >> more details as they come.
> >>
> >> Thanks
> >> Tyson
> >>
> >>
> >>
>

Re: Mesos integration

[Ben Browning <bb...@redhat.com>]

I believe a lot of Tyson's observations around Mesos integration also
apply to Kubernetes integration. For example, I've been experimenting
with an approach that replaces the DockerContainer with a
KubernetesContainer. The KubernetesContainer also ignores the pause
and resume of functions and also deploys containers to arbitrary nodes
in the Kubernetes cluster.

Here's a rough but working (except for log shipping) prototype of that
- https://github.com/projectodd/incubator-openwhisk/blob/kube-container/core/invoker/src/main/scala/whisk/core/containerpool/kubernetes/KubernetesContainer.scala


So, I'd like to add a +1 Tyson's notes around isolating docker client
usage and log collection. I believe they both equally apply to a
Kubernetes environment as well. And, more broadly, we'll need to
figure out how to make the various Container implementations pluggable
(or at least configurable) for Docker, Kubernetes, & Mesos.

Ben


On Wed, May 31, 2017 at 5:24 PM, Carlos Santana <cs...@gmail.com> wrote:
> Hi Tyson
>
> Thanks for reaching out, having Mesos integration in OpenWhisk is something
> I have heard some users asking about and any help to figure out how to best
> integrate are very welcome.
>
> As you already aware OpenWhisk Invoker is a critical path component
> optimized for performance, to get those container pause and unpause in
> single digit milliseconds that's the reason for the recent change from
> using docker engine down to runc
>
> Here is my initial notes, take into account I don't have first hand
> experience with Mesos, which is why is good to have folks like you join :-)
>
> I think extending in a way that a different mode can be use would be useful
> to plug a different orchestration methodology.
>
> For log collection we are looking into sending logs into ELK (Elastic
> search) there is some early  investigation going on, so I see after this
> integration, mesos can be told to which ELK to send the logs, and the
> Invoker to get the results.
>
> In terms of scala vs. nodejs, I would recommend to use scala with the java
> client, since there could be some components being integrated into Invoker
> and code sharing.
>
> How you want to follow up, maybe you have something to show us or maybe
> setup a hangout call to meet
>
> I think Dragos from Adobe, might have a lot more feedback than me since I
> think he has first hand experience
>
> --Carlos
>
>
>
> On Sat, May 27, 2017 at 2:44 PM Tyson Norris <tn...@adobe.com.invalid>
> wrote:
>
>> Hi All -
>> I’ve been considering how we can leverage mesos resource scheduling within
>> OpenWhisk, and wanted to share some ideas and solicit some feedback.
>>
>> In general, I’ve been experimenting with an approach that replaces the
>> DockerContainer implementation with a MesosTask. The MesosTask will have
>> effectively the same lifecycle (ignoring pause/resume functions) and the
>> same interface (HTTP api to the container), the main difference being that
>> the container will be deployed to some arbitrary host in the cluster.
>>
>> There are a few broad topics around this, including:
>> - docker client usage needs to be better isolated (e.g. cleaning up
>> existing containers, reconciling containers already running at time of
>> invoker startup, etc). I think this will be mostly straightforward.
>> - log collection - I’m not sure the best approach here, but one is to
>> completely decouple log collection from activation execution, and then
>> provide a mesos-specific impl
>> - container state tracking + load balancing - obviously there is potential
>> for conflict if 2 invokers schedule activations on the same container in
>> the cluster (since the state tracking of the container would still be in
>> the invoker). This would imply some extension to the ContainerPool as well.
>> - mesos framework - we’ve discussed internally a bit, and some preferences
>> I have are: leverage the mesos http api (avoid the older jni libs if
>> possible) and provide an independent framework application which provides a
>> (simpler)  http api that is consumed by the invoker (if mesos integration
>> is enabled in the deployment). This way the framework deployment can be
>> isolated from controller/invoker deployment, and interaction with docker
>> containers is mostly the same (except for logging).
>> - There are a few options of implementing mesos http api clients like:
>> rxjava client, and nodejs client, but I've not seen any good Scala client
>> to date, so we may either provide a scala app using the rxjava client, or
>> create.a new client in Scala.
>>
>> Let me know if you have any thoughts around any of these, and I will share
>> more details as they come.
>>
>> Thanks
>> Tyson
>>
>>
>>

Re: Mesos integration

[Carlos Santana <cs...@gmail.com>]

Hi Tyson

Thanks for reaching out, having Mesos integration in OpenWhisk is something
I have heard some users asking about and any help to figure out how to best
integrate are very welcome.

As you already aware OpenWhisk Invoker is a critical path component
optimized for performance, to get those container pause and unpause in
single digit milliseconds that's the reason for the recent change from
using docker engine down to runc

Here is my initial notes, take into account I don't have first hand
experience with Mesos, which is why is good to have folks like you join :-)

I think extending in a way that a different mode can be use would be useful
to plug a different orchestration methodology.

For log collection we are looking into sending logs into ELK (Elastic
search) there is some early  investigation going on, so I see after this
integration, mesos can be told to which ELK to send the logs, and the
Invoker to get the results.

In terms of scala vs. nodejs, I would recommend to use scala with the java
client, since there could be some components being integrated into Invoker
and code sharing.

How you want to follow up, maybe you have something to show us or maybe
setup a hangout call to meet

I think Dragos from Adobe, might have a lot more feedback than me since I
think he has first hand experience

--Carlos



On Sat, May 27, 2017 at 2:44 PM Tyson Norris <tn...@adobe.com.invalid>
wrote:

> Hi All -
> I’ve been considering how we can leverage mesos resource scheduling within
> OpenWhisk, and wanted to share some ideas and solicit some feedback.
>
> In general, I’ve been experimenting with an approach that replaces the
> DockerContainer implementation with a MesosTask. The MesosTask will have
> effectively the same lifecycle (ignoring pause/resume functions) and the
> same interface (HTTP api to the container), the main difference being that
> the container will be deployed to some arbitrary host in the cluster.
>
> There are a few broad topics around this, including:
> - docker client usage needs to be better isolated (e.g. cleaning up
> existing containers, reconciling containers already running at time of
> invoker startup, etc). I think this will be mostly straightforward.
> - log collection - I’m not sure the best approach here, but one is to
> completely decouple log collection from activation execution, and then
> provide a mesos-specific impl
> - container state tracking + load balancing - obviously there is potential
> for conflict if 2 invokers schedule activations on the same container in
> the cluster (since the state tracking of the container would still be in
> the invoker). This would imply some extension to the ContainerPool as well.
> - mesos framework - we’ve discussed internally a bit, and some preferences
> I have are: leverage the mesos http api (avoid the older jni libs if
> possible) and provide an independent framework application which provides a
> (simpler)  http api that is consumed by the invoker (if mesos integration
> is enabled in the deployment). This way the framework deployment can be
> isolated from controller/invoker deployment, and interaction with docker
> containers is mostly the same (except for logging).
> - There are a few options of implementing mesos http api clients like:
> rxjava client, and nodejs client, but I've not seen any good Scala client
> to date, so we may either provide a scala app using the rxjava client, or
> create.a new client in Scala.
>
> Let me know if you have any thoughts around any of these, and I will share
> more details as they come.
>
> Thanks
> Tyson
>
>
>