You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Gurvinder Singh <gu...@uninett.no> on 2015/03/09 22:52:49 UTC
mesos on coreos
Hi,
I am wondering if anybody in the community has looked into or are
running mesos on top of coreos. I would be interested to hear out your
experiences around following areas
- Users management on coreos cluster and containers running with Mesos
- Are you using fleet to run mesos or run it as service in cloud-config
and don't use fleet at all
- Networking among hosts flannel or ?
- Any other interesting insights you found considering such setup
Thanks,
Gurvinder
RE: mesos on coreos
Posted by Aaron Carey <ac...@ilm.com>.
FWIW we'd also love to play with Mesos as a CoreOS service
________________________________
From: Anton Kirillov [antonv.kirillov@gmail.com]
Sent: 10 March 2015 14:57
To: Gurvinder Singh
Cc: user@mesos.apache.org
Subject: Re: mesos on coreos
Gurvinder,
your points are really interesting to consider, but as for me it is still looks pretty like a bit narrow solution, because not all widespread systems are having frameworks to run on Mesos. But it really depends on your goals.
We have pretty specific use cases, one of them is using Spark on Mesos to achieve HA alongside with Cassandra as datastore. So we install Mesos Slave with Cassandra node on the same machine to achieve greater data locality. And Docker is pretty poor choice for running Cassandra in it but there is no other way to run Cassandra on CoreOS (afaik)
Another idea is that when you go to “big iron” OS footprint doesn’t matter a lot while you have multi-core and huge RAM hardware. It looks like premature optimization.
My points just come from recent experience and more problem-oriented. But it would be really nice to see Mesos as native CoreOS service to experiment with.
--
Anton Kirillov
Sent with Sparrow<http://www.sparrowmailapp.com/?sig>
On Tuesday, March 10, 2015 at 3:12 PM, Gurvinder Singh wrote:
Thanks Anton for sharing your experience. Response in line.
On 03/10/2015 01:01 PM, Anton Kirillov wrote:
Hi Gurvinder,
our team have an experience with Mesos on CoreOS with fleet, and we
decided to switch to bare metal deployments and here are our main reasons.
First of all, it doesn’t look like a great idea to package resource
manager into Docker putting one more abstraction layer between a
resource itself and resource manager.
I agree. That was the main reason I asked about closer integeration of
mesos with coreos. If you look here kubernetes
(https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos/cloud-configs/master.yaml)
run natively in coreos, not in a container. As it is started by
cloud-init process. So something similar for mesos will resolve this
issue. Although kubernetes is possible due to simple go binary with no
dependency, I looked at the Mesos library dependency and compare with
library on coreos. Only 2 are missing (libmesos-<version>.so,
libsasl2.so). So I think it is possible for mesos to follow the same
model as kubernetes to run natively.
>From DevOps point of view it is hard to control such thing as ZooKeeper
restarts (and ensemble rolling restarts as well) which is the core
service discovery mechanism for Mesos. You have to put some sidekick
services to provide peer discovery mechanics and it doesn’t look very
robust.
That's why I mention about the JIRA where mesos can use etcd. The
sidekick could be solved by using the flannel
(https://github.com/coreos/flannel) to make each container addressable
inside your cluster.
Very common use case with Mesos is running Docker on top of it either
with Marathon or with Aurora. But Docker service needs to be installed
on worker nodes. So you’re coming to Docker-into-Docker situation which
cancels all advantages of both transparent resource management and
simple deployment configuration.
One more point on Mesos inside Docker here, is that you have to attach
Mesos data directories from container to a host. Given that you’re
already running Mesos container in privileged mode and sharing
directories with state with host there is no more reasons to run Mesos
inside a container. And consider container restart (not just failure)
with registry corruption and following synchronization issues.
Another our use case with multi-region cluster deployments showed some
issues with etcd heartbeat/leader election timeouts, which need to be
increased in order to handle bigger latencies between data centers. If
timeouts increase fleet starts to work in unpredictable way, loosing and
finding peer nodes again which is not appropriate in production environment.
I have not experience with multi region deployment. As such scenario for
zookeeper can also be hard, where as consul claims to address this issue.
The reason for asking is that with coreos we have a small footprint,
upto date OS which can boot the mesos to manage whole cluster. By using
docker, we can have multi tenancy support too. Just ideas :P
- Gurvinder
You can take a look at this configuration for Mesos-CoreOS-HA as
well https://github.com/akirillov/mesos-deploy/tree/master/mesos-coreos-ha
--
Anton Kirillov
Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
On Tuesday, March 10, 2015 at 11:08 AM, Gurvinder Singh wrote:
Hi Micheal,
Yes I tested the tutorial and it works fine for testing. Later on I used
fleet to run mesos workers on all coreos machines too. I was wondering
how the landscape is looking in the community regarding coreos. As is
there any interest from community or mesos team to support coreos in
general. If yes the then how you see where Mesos fits in with Fleet,
Kubernetes.
My current understanding is that Fleet is useful for lightweight
scheduling, where as Mesos and kubernetes are kind of serving the
similar purpose. Mesos has been here for a while and more feature
complete than kubernetes. But Kubernetes has more tight integeration
with coreos like use etcd for co-ordination, flannel for networking. I
wonder what's the plan are when it comes to Mesos for such. I have seen
the JIRA for etcd (https://issues.apache.org/jira/browse/MESOS-1806)
I understand that the landscape is changing fast but its good to know
about Mesos roadmap in this regard. Also would love to know if anybody
using Coreos with Mesos beyond testing.
Thanks,
Gurvinder
On 03/09/2015 11:35 PM, Michael Park wrote:
Hi Gurvinder,
We got started on this work at Mesosphere and there's a tutorial
<http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/>
on
how to do a single-node setup. We ran the mesos-master and slaves in
docker containers which led to this JIRA ticket
<https://issues.apache.org/jira/browse/MESOS-2115>. I haven't been able
to follow-up on this article recently, and I'd like to hear about others
who have made further progress as well.
At the time, we were thinking that using fleet shouldn't be too
difficult since it uses the systemd unit files but didn't quite get
around to it.
Perhaps you'll find the tutorial to be a decent starting point.
Thanks,
MPark.
On 9 March 2015 at 17:52, Gurvinder Singh <gu...@uninett.no>
<ma...@uninett.no>
<ma...@uninett.no>> wrote:
Hi,
I am wondering if anybody in the community has looked into or are
running mesos on top of coreos. I would be interested to hear out your
experiences around following areas
- Users management on coreos cluster and containers running with Mesos
- Are you using fleet to run mesos or run it as service in
cloud-config
and don't use fleet at all
- Networking among hosts flannel or ?
- Any other interesting insights you found considering such setup
Thanks,
Gurvinder
Re: mesos on coreos
Posted by Anton Kirillov <an...@gmail.com>.
I think that Cassandra in Docker is a bit out of scope of this discussion though, but here are my concerns.
I don’t see any profits of putting another layer between Cassandra and OS because:
- we have less control over system-level configuration (e.g. ulimits, propagation of /proc/sys properties to host, disk tuning)
- node configuration becomes very messy with bash init scripts or anything (yes we can provision Docker with Ansible, but it looks like configuration overkill)
- if node fails, you have to repair it, not just restart process
- seed nodes recovery becomes pretty challenging
- access to node via 'docker exec' over ssh to host machine is inconvenient
- Cassandra is tuned for hardware, you can’t just rollout container which works on your laptop to any infrastructure and get any predictable performance at any cloud/datacenter without appropriate tuning
- think about monitoring (DataStax OpsCenter, for example need to have ssh access to Cassandra node in order to install agents)
We now currently use CoreOS/Docker Cassandra installation at AWS. What I could say is that we spent much more time troubleshooting Docker instead of focusing on Cassandra performance.
It would be great to continue our talk outside this topic, so feel free to reach me out via email, we can discuss all the details.
--
Anton Kirillov
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday, March 10, 2015 at 5:49 PM, Gurvinder Singh wrote:
> On 03/10/2015 03:57 PM, Anton Kirillov wrote:
> > Gurvinder,
> >
> > your points are really interesting to consider, but as for me it
> > is still looks pretty like a bit narrow solution, because not all
> > widespread systems are having frameworks to run on Mesos. But it
> > really depends on your goals.
> >
> > We have pretty specific use cases, one of them is using Spark on
> > Mesos to achieve HA alongside with Cassandra as datastore. So we
> > install Mesos Slave with Cassandra node on the same machine to
> > achieve greater data locality. And Docker is pretty poor choice for
> > running Cassandra in it but there is no other way to run Cassandra
> > on CoreOS (afaik)
> >
>
> We also plan to run different frameworks on our cluster and Spark,
> Cassandra being one of them. I would like to know what are the issues
> you faced while running cassandra in docker. As docker with volume
> attached has almost bare metal performance.
> >
> > Another idea is that when you go to “big iron” OS footprint
> > doesn’t matter a lot while you have multi-core and huge RAM
> > hardware. It looks like premature optimization.
> >
>
> Optimization is one thing, but with CoreOS you get more determinstic
> updates of OS with rollback option which can be quite useful when
> running large infrastructure. The current discussion is to get to know
> about community feeling about these ideas, It is not to say that it's
> the best solution :)
>
> - Gurvinder
> > My points just come from recent experience and more
> > problem-oriented. But it would be really nice to see Mesos as
> > native CoreOS service to experiment with.
> >
> > -- Anton Kirillov Sent with Sparrow
> > <http://www.sparrowmailapp.com/?sig>
> >
> > On Tuesday, March 10, 2015 at 3:12 PM, Gurvinder Singh wrote:
> >
> > > Thanks Anton for sharing your experience. Response in line. On
> > > 03/10/2015 01:01 PM, Anton Kirillov wrote:
> > > > Hi Gurvinder,
> > > >
> > > > our team have an experience with Mesos on CoreOS with fleet,
> > > > and we decided to switch to bare metal deployments and here are
> > > > our main reasons.
> > > >
> > > > First of all, it doesn’t look like a great idea to package
> > > > resource manager into Docker putting one more abstraction layer
> > > > between a resource itself and resource manager.
> > > >
> > >
> > > I agree. That was the main reason I asked about closer
> > > integeration of mesos with coreos. If you look here kubernetes
> > > (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos/cloud-configs/master.yaml)
> > >
> >
> >
>
> run natively in coreos, not in a container. As it is started by
> > > cloud-init process. So something similar for mesos will resolve
> > > this issue. Although kubernetes is possible due to simple go
> > > binary with no dependency, I looked at the Mesos library
> > > dependency and compare with library on coreos. Only 2 are missing
> > > (libmesos-<version>.so, libsasl2.so). So I think it is possible
> > > for mesos to follow the same model as kubernetes to run
> > > natively.
> > >
> > > >
> > > > From DevOps point of view it is hard to control such thing as
> > > > ZooKeeper restarts (and ensemble rolling restarts as well)
> > > > which is the core service discovery mechanism for Mesos. You
> > > > have to put some sidekick services to provide peer discovery
> > > > mechanics and it doesn’t look very robust.
> > > >
> > >
> > > That's why I mention about the JIRA where mesos can use etcd.
> > > The sidekick could be solved by using the flannel
> > > (https://github.com/coreos/flannel) to make each container
> > > addressable inside your cluster.
> > > >
> > > > Very common use case with Mesos is running Docker on top of it
> > > > either with Marathon or with Aurora. But Docker service needs
> > > > to be installed on worker nodes. So you’re coming to
> > > > Docker-into-Docker situation which cancels all advantages of
> > > > both transparent resource management and simple deployment
> > > > configuration.
> > > >
> > > > One more point on Mesos inside Docker here, is that you have to
> > > > attach Mesos data directories from container to a host. Given
> > > > that you’re already running Mesos container in privileged mode
> > > > and sharing directories with state with host there is no more
> > > > reasons to run Mesos inside a container. And consider container
> > > > restart (not just failure) with registry corruption and
> > > > following synchronization issues.
> > > >
> > > > Another our use case with multi-region cluster deployments
> > > > showed some issues with etcd heartbeat/leader election
> > > > timeouts, which need to be increased in order to handle bigger
> > > > latencies between data centers. If timeouts increase fleet
> > > > starts to work in unpredictable way, loosing and finding peer
> > > > nodes again which is not appropriate in production
> > > > environment.
> > > >
> > >
> > > I have not experience with multi region deployment. As such
> > > scenario for zookeeper can also be hard, where as consul claims
> > > to address this issue.
> > >
> > > The reason for asking is that with coreos we have a small
> > > footprint, upto date OS which can boot the mesos to manage whole
> > > cluster. By using docker, we can have multi tenancy support too.
> > > Just ideas :P
> > >
> > > - Gurvinder
> > >
> > > >
> > > > You can take a look at this configuration for Mesos-CoreOS-HA
> > > > as well
> > > > https://github.com/akirillov/mesos-deploy/tree/master/mesos-coreos-ha
> > > >
> > >
> > >
> >
>
> --
> > > > Anton Kirillov Sent with Sparrow
> > > > <http://www.sparrowmailapp.com/?sig>
> > > >
> > > > On Tuesday, March 10, 2015 at 11:08 AM, Gurvinder Singh wrote:
> > > >
> > > > > Hi Micheal,
> > > > >
> > > > > Yes I tested the tutorial and it works fine for testing.
> > > > > Later on I used fleet to run mesos workers on all coreos
> > > > > machines too. I was wondering how the landscape is looking in
> > > > > the community regarding coreos. As is there any interest from
> > > > > community or mesos team to support coreos in general. If yes
> > > > > the then how you see where Mesos fits in with Fleet,
> > > > > Kubernetes.
> > > > >
> > > > > My current understanding is that Fleet is useful for
> > > > > lightweight scheduling, where as Mesos and kubernetes are
> > > > > kind of serving the similar purpose. Mesos has been here for
> > > > > a while and more feature complete than kubernetes. But
> > > > > Kubernetes has more tight integeration with coreos like use
> > > > > etcd for co-ordination, flannel for networking. I wonder
> > > > > what's the plan are when it comes to Mesos for such. I have
> > > > > seen the JIRA for etcd
> > > > > (https://issues.apache.org/jira/browse/MESOS-1806)
> > > > >
> > > > > I understand that the landscape is changing fast but its good
> > > > > to know about Mesos roadmap in this regard. Also would love
> > > > > to know if anybody using Coreos with Mesos beyond testing.
> > > > >
> > > > > Thanks, Gurvinder On 03/09/2015 11:35 PM, Michael Park
> > > > > wrote:
> > > > > > Hi Gurvinder,
> > > > > >
> > > > > > We got started on this work at Mesosphere and there's a
> > > > > > tutorial
> > > > > > <http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/>
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
> on
> > > > > > how to do a single-node setup. We ran the mesos-master and
> > > > > > slaves in docker containers which led to this JIRA ticket
> > > > > > <https://issues.apache.org/jira/browse/MESOS-2115>. I
> > > > > > haven't been able to follow-up on this article recently,
> > > > > > and I'd like to hear about others who have made further
> > > > > > progress as well.
> > > > > >
> > > > > > At the time, we were thinking that using fleet shouldn't be
> > > > > > too difficult since it uses the systemd unit files but
> > > > > > didn't quite get around to it.
> > > > > >
> > > > > > Perhaps you'll find the tutorial to be a decent starting
> > > > > > point.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > MPark.
> > > > > >
> > > > > > On 9 March 2015 at 17:52, Gurvinder Singh
> > > > > > <gurvinder.singh@uninett.no (mailto:gurvinder.singh@uninett.no)
> > > > > > <ma...@uninett.no>
> > > > > > <ma...@uninett.no>
> > > > > > <ma...@uninett.no>> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am wondering if anybody in the community has looked into
> > > > > > or are running mesos on top of coreos. I would be
> > > > > > interested to hear out your experiences around following
> > > > > > areas
> > > > > >
> > > > > > - Users management on coreos cluster and containers running
> > > > > > with Mesos - Are you using fleet to run mesos or run it as
> > > > > > service in cloud-config and don't use fleet at all -
> > > > > > Networking among hosts flannel or ? - Any other interesting
> > > > > > insights you found considering such setup
> > > > > >
> > > > > > Thanks, Gurvinder
Re: mesos on coreos
Posted by Gurvinder Singh <gu...@uninett.no>.
On 03/10/2015 03:57 PM, Anton Kirillov wrote:
> Gurvinder,
>
> your points are really interesting to consider, but as for me it
> is still looks pretty like a bit narrow solution, because not all
> widespread systems are having frameworks to run on Mesos. But it
> really depends on your goals.
>
> We have pretty specific use cases, one of them is using Spark on
> Mesos to achieve HA alongside with Cassandra as datastore. So we
> install Mesos Slave with Cassandra node on the same machine to
> achieve greater data locality. And Docker is pretty poor choice for
> running Cassandra in it but there is no other way to run Cassandra
> on CoreOS (afaik)
We also plan to run different frameworks on our cluster and Spark,
Cassandra being one of them. I would like to know what are the issues
you faced while running cassandra in docker. As docker with volume
attached has almost bare metal performance.
>
> Another idea is that when you go to “big iron” OS footprint
> doesn’t matter a lot while you have multi-core and huge RAM
> hardware. It looks like premature optimization.
>
Optimization is one thing, but with CoreOS you get more determinstic
updates of OS with rollback option which can be quite useful when
running large infrastructure. The current discussion is to get to know
about community feeling about these ideas, It is not to say that it's
the best solution :)
- Gurvinder
> My points just come from recent experience and more
> problem-oriented. But it would be really nice to see Mesos as
> native CoreOS service to experiment with.
>
> -- Anton Kirillov Sent with Sparrow
> <http://www.sparrowmailapp.com/?sig>
>
> On Tuesday, March 10, 2015 at 3:12 PM, Gurvinder Singh wrote:
>
>> Thanks Anton for sharing your experience. Response in line. On
>> 03/10/2015 01:01 PM, Anton Kirillov wrote:
>>> Hi Gurvinder,
>>>
>>> our team have an experience with Mesos on CoreOS with fleet,
>>> and we decided to switch to bare metal deployments and here are
>>> our main reasons.
>>>
>>> First of all, it doesn’t look like a great idea to package
>>> resource manager into Docker putting one more abstraction layer
>>> between a resource itself and resource manager.
>> I agree. That was the main reason I asked about closer
>> integeration of mesos with coreos. If you look here kubernetes
>> (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos/cloud-configs/master.yaml)
>>
>>
run natively in coreos, not in a container. As it is started by
>> cloud-init process. So something similar for mesos will resolve
>> this issue. Although kubernetes is possible due to simple go
>> binary with no dependency, I looked at the Mesos library
>> dependency and compare with library on coreos. Only 2 are missing
>> (libmesos-<version>.so, libsasl2.so). So I think it is possible
>> for mesos to follow the same model as kubernetes to run
>> natively.
>>
>>>
>>> From DevOps point of view it is hard to control such thing as
>>> ZooKeeper restarts (and ensemble rolling restarts as well)
>>> which is the core service discovery mechanism for Mesos. You
>>> have to put some sidekick services to provide peer discovery
>>> mechanics and it doesn’t look very robust.
>> That's why I mention about the JIRA where mesos can use etcd.
>> The sidekick could be solved by using the flannel
>> (https://github.com/coreos/flannel) to make each container
>> addressable inside your cluster.
>>>
>>> Very common use case with Mesos is running Docker on top of it
>>> either with Marathon or with Aurora. But Docker service needs
>>> to be installed on worker nodes. So you’re coming to
>>> Docker-into-Docker situation which cancels all advantages of
>>> both transparent resource management and simple deployment
>>> configuration.
>>>
>>> One more point on Mesos inside Docker here, is that you have to
>>> attach Mesos data directories from container to a host. Given
>>> that you’re already running Mesos container in privileged mode
>>> and sharing directories with state with host there is no more
>>> reasons to run Mesos inside a container. And consider container
>>> restart (not just failure) with registry corruption and
>>> following synchronization issues.
>>>
>>> Another our use case with multi-region cluster deployments
>>> showed some issues with etcd heartbeat/leader election
>>> timeouts, which need to be increased in order to handle bigger
>>> latencies between data centers. If timeouts increase fleet
>>> starts to work in unpredictable way, loosing and finding peer
>>> nodes again which is not appropriate in production
>>> environment.
>> I have not experience with multi region deployment. As such
>> scenario for zookeeper can also be hard, where as consul claims
>> to address this issue.
>>
>> The reason for asking is that with coreos we have a small
>> footprint, upto date OS which can boot the mesos to manage whole
>> cluster. By using docker, we can have multi tenancy support too.
>> Just ideas :P
>>
>> - Gurvinder
>>
>>>
>>> You can take a look at this configuration for Mesos-CoreOS-HA
>>> as well
>>> https://github.com/akirillov/mesos-deploy/tree/master/mesos-coreos-ha
>>>
>>>
>>>
--
>>> Anton Kirillov Sent with Sparrow
>>> <http://www.sparrowmailapp.com/?sig>
>>>
>>> On Tuesday, March 10, 2015 at 11:08 AM, Gurvinder Singh wrote:
>>>
>>>> Hi Micheal,
>>>>
>>>> Yes I tested the tutorial and it works fine for testing.
>>>> Later on I used fleet to run mesos workers on all coreos
>>>> machines too. I was wondering how the landscape is looking in
>>>> the community regarding coreos. As is there any interest from
>>>> community or mesos team to support coreos in general. If yes
>>>> the then how you see where Mesos fits in with Fleet,
>>>> Kubernetes.
>>>>
>>>> My current understanding is that Fleet is useful for
>>>> lightweight scheduling, where as Mesos and kubernetes are
>>>> kind of serving the similar purpose. Mesos has been here for
>>>> a while and more feature complete than kubernetes. But
>>>> Kubernetes has more tight integeration with coreos like use
>>>> etcd for co-ordination, flannel for networking. I wonder
>>>> what's the plan are when it comes to Mesos for such. I have
>>>> seen the JIRA for etcd
>>>> (https://issues.apache.org/jira/browse/MESOS-1806)
>>>>
>>>> I understand that the landscape is changing fast but its good
>>>> to know about Mesos roadmap in this regard. Also would love
>>>> to know if anybody using Coreos with Mesos beyond testing.
>>>>
>>>> Thanks, Gurvinder On 03/09/2015 11:35 PM, Michael Park
>>>> wrote:
>>>>> Hi Gurvinder,
>>>>>
>>>>> We got started on this work at Mesosphere and there's a
>>>>> tutorial
>>>>> <http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/>
>>>>>
>>>>>
on
>>>>> how to do a single-node setup. We ran the mesos-master and
>>>>> slaves in docker containers which led to this JIRA ticket
>>>>> <https://issues.apache.org/jira/browse/MESOS-2115>. I
>>>>> haven't been able to follow-up on this article recently,
>>>>> and I'd like to hear about others who have made further
>>>>> progress as well.
>>>>>
>>>>> At the time, we were thinking that using fleet shouldn't be
>>>>> too difficult since it uses the systemd unit files but
>>>>> didn't quite get around to it.
>>>>>
>>>>> Perhaps you'll find the tutorial to be a decent starting
>>>>> point.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> MPark.
>>>>>
>>>>> On 9 March 2015 at 17:52, Gurvinder Singh
>>>>> <gurvinder.singh@uninett.no
>>>>> <ma...@uninett.no>
>>>>> <ma...@uninett.no>
>>>>> <ma...@uninett.no>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am wondering if anybody in the community has looked into
>>>>> or are running mesos on top of coreos. I would be
>>>>> interested to hear out your experiences around following
>>>>> areas
>>>>>
>>>>> - Users management on coreos cluster and containers running
>>>>> with Mesos - Are you using fleet to run mesos or run it as
>>>>> service in cloud-config and don't use fleet at all -
>>>>> Networking among hosts flannel or ? - Any other interesting
>>>>> insights you found considering such setup
>>>>>
>>>>> Thanks, Gurvinder
>
Re: mesos on coreos
Posted by Anton Kirillov <an...@gmail.com>.
Gurvinder,
your points are really interesting to consider, but as for me it is still looks pretty like a bit narrow solution, because not all widespread systems are having frameworks to run on Mesos. But it really depends on your goals.
We have pretty specific use cases, one of them is using Spark on Mesos to achieve HA alongside with Cassandra as datastore. So we install Mesos Slave with Cassandra node on the same machine to achieve greater data locality. And Docker is pretty poor choice for running Cassandra in it but there is no other way to run Cassandra on CoreOS (afaik)
Another idea is that when you go to “big iron” OS footprint doesn’t matter a lot while you have multi-core and huge RAM hardware. It looks like premature optimization.
My points just come from recent experience and more problem-oriented. But it would be really nice to see Mesos as native CoreOS service to experiment with.
--
Anton Kirillov
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday, March 10, 2015 at 3:12 PM, Gurvinder Singh wrote:
> Thanks Anton for sharing your experience. Response in line.
> On 03/10/2015 01:01 PM, Anton Kirillov wrote:
> > Hi Gurvinder,
> >
> > our team have an experience with Mesos on CoreOS with fleet, and we
> > decided to switch to bare metal deployments and here are our main reasons.
> >
> > First of all, it doesn’t look like a great idea to package resource
> > manager into Docker putting one more abstraction layer between a
> > resource itself and resource manager.
> >
>
> I agree. That was the main reason I asked about closer integeration of
> mesos with coreos. If you look here kubernetes
> (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos/cloud-configs/master.yaml)
> run natively in coreos, not in a container. As it is started by
> cloud-init process. So something similar for mesos will resolve this
> issue. Although kubernetes is possible due to simple go binary with no
> dependency, I looked at the Mesos library dependency and compare with
> library on coreos. Only 2 are missing (libmesos-<version>.so,
> libsasl2.so). So I think it is possible for mesos to follow the same
> model as kubernetes to run natively.
>
> >
> > From DevOps point of view it is hard to control such thing as ZooKeeper
> > restarts (and ensemble rolling restarts as well) which is the core
> > service discovery mechanism for Mesos. You have to put some sidekick
> > services to provide peer discovery mechanics and it doesn’t look very
> > robust.
> >
>
> That's why I mention about the JIRA where mesos can use etcd. The
> sidekick could be solved by using the flannel
> (https://github.com/coreos/flannel) to make each container addressable
> inside your cluster.
> >
> > Very common use case with Mesos is running Docker on top of it either
> > with Marathon or with Aurora. But Docker service needs to be installed
> > on worker nodes. So you’re coming to Docker-into-Docker situation which
> > cancels all advantages of both transparent resource management and
> > simple deployment configuration.
> >
> > One more point on Mesos inside Docker here, is that you have to attach
> > Mesos data directories from container to a host. Given that you’re
> > already running Mesos container in privileged mode and sharing
> > directories with state with host there is no more reasons to run Mesos
> > inside a container. And consider container restart (not just failure)
> > with registry corruption and following synchronization issues.
> >
> > Another our use case with multi-region cluster deployments showed some
> > issues with etcd heartbeat/leader election timeouts, which need to be
> > increased in order to handle bigger latencies between data centers. If
> > timeouts increase fleet starts to work in unpredictable way, loosing and
> > finding peer nodes again which is not appropriate in production environment.
> >
>
> I have not experience with multi region deployment. As such scenario for
> zookeeper can also be hard, where as consul claims to address this issue.
>
> The reason for asking is that with coreos we have a small footprint,
> upto date OS which can boot the mesos to manage whole cluster. By using
> docker, we can have multi tenancy support too. Just ideas :P
>
> - Gurvinder
>
> >
> > You can take a look at this configuration for Mesos-CoreOS-HA as
> > well https://github.com/akirillov/mesos-deploy/tree/master/mesos-coreos-ha
> >
> > --
> > Anton Kirillov
> > Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
> >
> > On Tuesday, March 10, 2015 at 11:08 AM, Gurvinder Singh wrote:
> >
> > > Hi Micheal,
> > >
> > > Yes I tested the tutorial and it works fine for testing. Later on I used
> > > fleet to run mesos workers on all coreos machines too. I was wondering
> > > how the landscape is looking in the community regarding coreos. As is
> > > there any interest from community or mesos team to support coreos in
> > > general. If yes the then how you see where Mesos fits in with Fleet,
> > > Kubernetes.
> > >
> > > My current understanding is that Fleet is useful for lightweight
> > > scheduling, where as Mesos and kubernetes are kind of serving the
> > > similar purpose. Mesos has been here for a while and more feature
> > > complete than kubernetes. But Kubernetes has more tight integeration
> > > with coreos like use etcd for co-ordination, flannel for networking. I
> > > wonder what's the plan are when it comes to Mesos for such. I have seen
> > > the JIRA for etcd (https://issues.apache.org/jira/browse/MESOS-1806)
> > >
> > > I understand that the landscape is changing fast but its good to know
> > > about Mesos roadmap in this regard. Also would love to know if anybody
> > > using Coreos with Mesos beyond testing.
> > >
> > > Thanks,
> > > Gurvinder
> > > On 03/09/2015 11:35 PM, Michael Park wrote:
> > > > Hi Gurvinder,
> > > >
> > > > We got started on this work at Mesosphere and there's a tutorial
> > > > <http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/>
> > > > on
> > > > how to do a single-node setup. We ran the mesos-master and slaves in
> > > > docker containers which led to this JIRA ticket
> > > > <https://issues.apache.org/jira/browse/MESOS-2115>. I haven't been able
> > > > to follow-up on this article recently, and I'd like to hear about others
> > > > who have made further progress as well.
> > > >
> > > > At the time, we were thinking that using fleet shouldn't be too
> > > > difficult since it uses the systemd unit files but didn't quite get
> > > > around to it.
> > > >
> > > > Perhaps you'll find the tutorial to be a decent starting point.
> > > >
> > > > Thanks,
> > > >
> > > > MPark.
> > > >
> > > > On 9 March 2015 at 17:52, Gurvinder Singh <gurvinder.singh@uninett.no (mailto:gurvinder.singh@uninett.no)
> > > > <ma...@uninett.no>
> > > > <ma...@uninett.no>> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I am wondering if anybody in the community has looked into or are
> > > > running mesos on top of coreos. I would be interested to hear out your
> > > > experiences around following areas
> > > >
> > > > - Users management on coreos cluster and containers running with Mesos
> > > > - Are you using fleet to run mesos or run it as service in
> > > > cloud-config
> > > > and don't use fleet at all
> > > > - Networking among hosts flannel or ?
> > > > - Any other interesting insights you found considering such setup
> > > >
> > > > Thanks,
> > > > Gurvinder
> > > >
> > >
> > >
> >
> >
>
>
>
Re: mesos on coreos
Posted by Gurvinder Singh <gu...@uninett.no>.
Thanks Anton for sharing your experience. Response in line.
On 03/10/2015 01:01 PM, Anton Kirillov wrote:
> Hi Gurvinder,
>
> our team have an experience with Mesos on CoreOS with fleet, and we
> decided to switch to bare metal deployments and here are our main reasons.
>
> First of all, it doesn’t look like a great idea to package resource
> manager into Docker putting one more abstraction layer between a
> resource itself and resource manager.
I agree. That was the main reason I asked about closer integeration of
mesos with coreos. If you look here kubernetes
(https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos/cloud-configs/master.yaml)
run natively in coreos, not in a container. As it is started by
cloud-init process. So something similar for mesos will resolve this
issue. Although kubernetes is possible due to simple go binary with no
dependency, I looked at the Mesos library dependency and compare with
library on coreos. Only 2 are missing (libmesos-<version>.so,
libsasl2.so). So I think it is possible for mesos to follow the same
model as kubernetes to run natively.
>
> From DevOps point of view it is hard to control such thing as ZooKeeper
> restarts (and ensemble rolling restarts as well) which is the core
> service discovery mechanism for Mesos. You have to put some sidekick
> services to provide peer discovery mechanics and it doesn’t look very
> robust.
That's why I mention about the JIRA where mesos can use etcd. The
sidekick could be solved by using the flannel
(https://github.com/coreos/flannel) to make each container addressable
inside your cluster.
>
> Very common use case with Mesos is running Docker on top of it either
> with Marathon or with Aurora. But Docker service needs to be installed
> on worker nodes. So you’re coming to Docker-into-Docker situation which
> cancels all advantages of both transparent resource management and
> simple deployment configuration.
>
> One more point on Mesos inside Docker here, is that you have to attach
> Mesos data directories from container to a host. Given that you’re
> already running Mesos container in privileged mode and sharing
> directories with state with host there is no more reasons to run Mesos
> inside a container. And consider container restart (not just failure)
> with registry corruption and following synchronization issues.
>
> Another our use case with multi-region cluster deployments showed some
> issues with etcd heartbeat/leader election timeouts, which need to be
> increased in order to handle bigger latencies between data centers. If
> timeouts increase fleet starts to work in unpredictable way, loosing and
> finding peer nodes again which is not appropriate in production environment.
I have not experience with multi region deployment. As such scenario for
zookeeper can also be hard, where as consul claims to address this issue.
The reason for asking is that with coreos we have a small footprint,
upto date OS which can boot the mesos to manage whole cluster. By using
docker, we can have multi tenancy support too. Just ideas :P
- Gurvinder
>
> You can take a look at this configuration for Mesos-CoreOS-HA as
> well https://github.com/akirillov/mesos-deploy/tree/master/mesos-coreos-ha
>
> --
> Anton Kirillov
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Tuesday, March 10, 2015 at 11:08 AM, Gurvinder Singh wrote:
>
>> Hi Micheal,
>>
>> Yes I tested the tutorial and it works fine for testing. Later on I used
>> fleet to run mesos workers on all coreos machines too. I was wondering
>> how the landscape is looking in the community regarding coreos. As is
>> there any interest from community or mesos team to support coreos in
>> general. If yes the then how you see where Mesos fits in with Fleet,
>> Kubernetes.
>>
>> My current understanding is that Fleet is useful for lightweight
>> scheduling, where as Mesos and kubernetes are kind of serving the
>> similar purpose. Mesos has been here for a while and more feature
>> complete than kubernetes. But Kubernetes has more tight integeration
>> with coreos like use etcd for co-ordination, flannel for networking. I
>> wonder what's the plan are when it comes to Mesos for such. I have seen
>> the JIRA for etcd (https://issues.apache.org/jira/browse/MESOS-1806)
>>
>> I understand that the landscape is changing fast but its good to know
>> about Mesos roadmap in this regard. Also would love to know if anybody
>> using Coreos with Mesos beyond testing.
>>
>> Thanks,
>> Gurvinder
>> On 03/09/2015 11:35 PM, Michael Park wrote:
>>> Hi Gurvinder,
>>>
>>> We got started on this work at Mesosphere and there's a tutorial
>>> <http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/>
>>> on
>>> how to do a single-node setup. We ran the mesos-master and slaves in
>>> docker containers which led to this JIRA ticket
>>> <https://issues.apache.org/jira/browse/MESOS-2115>. I haven't been able
>>> to follow-up on this article recently, and I'd like to hear about others
>>> who have made further progress as well.
>>>
>>> At the time, we were thinking that using fleet shouldn't be too
>>> difficult since it uses the systemd unit files but didn't quite get
>>> around to it.
>>>
>>> Perhaps you'll find the tutorial to be a decent starting point.
>>>
>>> Thanks,
>>>
>>> MPark.
>>>
>>> On 9 March 2015 at 17:52, Gurvinder Singh <gurvinder.singh@uninett.no
>>> <ma...@uninett.no>
>>> <ma...@uninett.no>> wrote:
>>>
>>> Hi,
>>>
>>> I am wondering if anybody in the community has looked into or are
>>> running mesos on top of coreos. I would be interested to hear out your
>>> experiences around following areas
>>>
>>> - Users management on coreos cluster and containers running with Mesos
>>> - Are you using fleet to run mesos or run it as service in
>>> cloud-config
>>> and don't use fleet at all
>>> - Networking among hosts flannel or ?
>>> - Any other interesting insights you found considering such setup
>>>
>>> Thanks,
>>> Gurvinder
>
Re: mesos on coreos
Posted by Adam Bordelon <ad...@mesosphere.io>.
Some current issues are listed under
https://issues.apache.org/jira/browse/MESOS-2115
See also previous email discussions:
http://www.mail-archive.com/user%40mesos.apache.org/msg02123.html
http://www.mail-archive.com/user%40mesos.apache.org/msg01617.html
On Tue, Mar 10, 2015 at 4:58 PM, craig w <co...@gmail.com> wrote:
> Is there any documentation describing what's necessary to run mesos master
> and slaves in Docker containers? You already mentioned a few things
> (mounting work dir, /sys/fs, etc).
>
> Thanks
>
> On Tue, Mar 10, 2015 at 6:41 PM, Tim Chen <ti...@mesosphere.io> wrote:
>
>> Hi all,
>>
>> As Alex said you can run Mesos in CoreOS without Docker if you put in the
>> dependencies in.
>>
>> It is a common ask though to run Mesos-slave in a Docker container in
>> general, either on CoreOS or not. It's definitely a bit involved as you
>> need to mount in a directory for persisting work dir and also mounting in
>> /sys/fs for cgroups, also you should use the --pid=host flag since Docker
>> 1.5 so it shares the host pid namespace.
>>
>> Although you get a lot less isolation, there are still motivations to run
>> slave in Docker regardless.
>>
>> One thing that's missing from the mesos docker containerizer is that it
>> won't be able to recover tasks on restart, and I have a series of patches
>> pending review to fix that.
>>
>> Tim
>>
>> On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov <al...@mesosphere.io>
>> wrote:
>>
>>> My 2¢.
>>>
>>>
>>>> First of all, it doesn’t look like a great idea to package resource
>>>> manager into Docker putting one more abstraction layer between a resource
>>>> itself and resource manager.
>>>>
>>>
>>> You can run mesos-slave on CoreOS node without putting it into a Docker
>>> container.
>>>
>>> —Alex
>>>
>>
>>
>
>
> --
>
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links
>
>
Re: mesos on coreos
Posted by craig w <co...@gmail.com>.
Is there any documentation describing what's necessary to run mesos master
and slaves in Docker containers? You already mentioned a few things
(mounting work dir, /sys/fs, etc).
Thanks
On Tue, Mar 10, 2015 at 6:41 PM, Tim Chen <ti...@mesosphere.io> wrote:
> Hi all,
>
> As Alex said you can run Mesos in CoreOS without Docker if you put in the
> dependencies in.
>
> It is a common ask though to run Mesos-slave in a Docker container in
> general, either on CoreOS or not. It's definitely a bit involved as you
> need to mount in a directory for persisting work dir and also mounting in
> /sys/fs for cgroups, also you should use the --pid=host flag since Docker
> 1.5 so it shares the host pid namespace.
>
> Although you get a lot less isolation, there are still motivations to run
> slave in Docker regardless.
>
> One thing that's missing from the mesos docker containerizer is that it
> won't be able to recover tasks on restart, and I have a series of patches
> pending review to fix that.
>
> Tim
>
> On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov <al...@mesosphere.io>
> wrote:
>
>> My 2¢.
>>
>>
>>> First of all, it doesn’t look like a great idea to package resource
>>> manager into Docker putting one more abstraction layer between a resource
>>> itself and resource manager.
>>>
>>
>> You can run mesos-slave on CoreOS node without putting it into a Docker
>> container.
>>
>> —Alex
>>
>
>
--
https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links
Re: mesos on coreos
Posted by Gurvinder Singh <gu...@uninett.no>.
Thanks Alex for the information and others too for sharing their
experiences.
- Gurvinder
On 03/11/2015 07:50 PM, Alex Rukletsov wrote:
> Gurvinder,
>
> no, there are no publicly available binaries, neither is documentation
> at this point. We will publish either or both as soon as it is rock solid.
>
> On Wed, Mar 11, 2015 at 2:08 AM, Gurvinder Singh
> <gurvinder.singh@uninett.no <ma...@uninett.no>> wrote:
>
> On 03/10/2015 11:41 PM, Tim Chen wrote:
> > Hi all,
> >
> > As Alex said you can run Mesos in CoreOS without Docker if you put in
> > the dependencies in.
> >
> Tim, is there any documentation of using Mesos outside container in
> CoreOS available or binary available which we can wget in cloud-init
> file to fulfill dependencies. As we would like to test it out Mesos on
> CoreOS outside docker.
>
> - Gurvinder
> > It is a common ask though to run Mesos-slave in a Docker container in
> > general, either on CoreOS or not. It's definitely a bit involved as you
> > need to mount in a directory for persisting work dir and also mounting
> > in /sys/fs for cgroups, also you should use the --pid=host flag since
> > Docker 1.5 so it shares the host pid namespace.
> >
> > Although you get a lot less isolation, there are still motivations to
> > run slave in Docker regardless.
> >
> > One thing that's missing from the mesos docker containerizer is that it
> > won't be able to recover tasks on restart, and I have a series of
> > patches pending review to fix that.
> >
> > Tim
> >
> > On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov <alex@mesosphere.io <ma...@mesosphere.io>
> > <mailto:alex@mesosphere.io <ma...@mesosphere.io>>> wrote:
> >
> > My 2¢.
> >
> >
> > First of all, it doesn’t look like a great idea to package
> > resource manager into Docker putting one more abstraction
> layer
> > between a resource itself and resource manager.
> >
> >
> > You can run mesos-slave on CoreOS node without putting it into a
> > Docker container.
> >
> > —Alex
> >
> >
>
>
Re: mesos on coreos
Posted by Alex Rukletsov <al...@mesosphere.io>.
Gurvinder,
no, there are no publicly available binaries, neither is documentation at
this point. We will publish either or both as soon as it is rock solid.
On Wed, Mar 11, 2015 at 2:08 AM, Gurvinder Singh <gurvinder.singh@uninett.no
> wrote:
> On 03/10/2015 11:41 PM, Tim Chen wrote:
> > Hi all,
> >
> > As Alex said you can run Mesos in CoreOS without Docker if you put in
> > the dependencies in.
> >
> Tim, is there any documentation of using Mesos outside container in
> CoreOS available or binary available which we can wget in cloud-init
> file to fulfill dependencies. As we would like to test it out Mesos on
> CoreOS outside docker.
>
> - Gurvinder
> > It is a common ask though to run Mesos-slave in a Docker container in
> > general, either on CoreOS or not. It's definitely a bit involved as you
> > need to mount in a directory for persisting work dir and also mounting
> > in /sys/fs for cgroups, also you should use the --pid=host flag since
> > Docker 1.5 so it shares the host pid namespace.
> >
> > Although you get a lot less isolation, there are still motivations to
> > run slave in Docker regardless.
> >
> > One thing that's missing from the mesos docker containerizer is that it
> > won't be able to recover tasks on restart, and I have a series of
> > patches pending review to fix that.
> >
> > Tim
> >
> > On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov <alex@mesosphere.io
> > <ma...@mesosphere.io>> wrote:
> >
> > My 2¢.
> >
> >
> > First of all, it doesn’t look like a great idea to package
> > resource manager into Docker putting one more abstraction layer
> > between a resource itself and resource manager.
> >
> >
> > You can run mesos-slave on CoreOS node without putting it into a
> > Docker container.
> >
> > —Alex
> >
> >
>
>
Re: mesos on coreos
Posted by Gurvinder Singh <gu...@uninett.no>.
On 03/10/2015 11:41 PM, Tim Chen wrote:
> Hi all,
>
> As Alex said you can run Mesos in CoreOS without Docker if you put in
> the dependencies in.
>
Tim, is there any documentation of using Mesos outside container in
CoreOS available or binary available which we can wget in cloud-init
file to fulfill dependencies. As we would like to test it out Mesos on
CoreOS outside docker.
- Gurvinder
> It is a common ask though to run Mesos-slave in a Docker container in
> general, either on CoreOS or not. It's definitely a bit involved as you
> need to mount in a directory for persisting work dir and also mounting
> in /sys/fs for cgroups, also you should use the --pid=host flag since
> Docker 1.5 so it shares the host pid namespace.
>
> Although you get a lot less isolation, there are still motivations to
> run slave in Docker regardless.
>
> One thing that's missing from the mesos docker containerizer is that it
> won't be able to recover tasks on restart, and I have a series of
> patches pending review to fix that.
>
> Tim
>
> On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov <alex@mesosphere.io
> <ma...@mesosphere.io>> wrote:
>
> My 2¢.
>
>
> First of all, it doesn’t look like a great idea to package
> resource manager into Docker putting one more abstraction layer
> between a resource itself and resource manager.
>
>
> You can run mesos-slave on CoreOS node without putting it into a
> Docker container.
>
> —Alex
>
>
Re: mesos on coreos
Posted by Tim Chen <ti...@mesosphere.io>.
Hi all,
As Alex said you can run Mesos in CoreOS without Docker if you put in the
dependencies in.
It is a common ask though to run Mesos-slave in a Docker container in
general, either on CoreOS or not. It's definitely a bit involved as you
need to mount in a directory for persisting work dir and also mounting in
/sys/fs for cgroups, also you should use the --pid=host flag since Docker
1.5 so it shares the host pid namespace.
Although you get a lot less isolation, there are still motivations to run
slave in Docker regardless.
One thing that's missing from the mesos docker containerizer is that it
won't be able to recover tasks on restart, and I have a series of patches
pending review to fix that.
Tim
On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov <al...@mesosphere.io> wrote:
> My 2¢.
>
>
>> First of all, it doesn’t look like a great idea to package resource
>> manager into Docker putting one more abstraction layer between a resource
>> itself and resource manager.
>>
>
> You can run mesos-slave on CoreOS node without putting it into a Docker
> container.
>
> —Alex
>
Re: mesos on coreos
Posted by Alex Rukletsov <al...@mesosphere.io>.
My 2¢.
> First of all, it doesn’t look like a great idea to package resource
> manager into Docker putting one more abstraction layer between a resource
> itself and resource manager.
>
You can run mesos-slave on CoreOS node without putting it into a Docker
container.
—Alex
Re: mesos on coreos
Posted by Anton Kirillov <an...@gmail.com>.
Hi Gurvinder,
our team have an experience with Mesos on CoreOS with fleet, and we decided to switch to bare metal deployments and here are our main reasons.
First of all, it doesn’t look like a great idea to package resource manager into Docker putting one more abstraction layer between a resource itself and resource manager.
From DevOps point of view it is hard to control such thing as ZooKeeper restarts (and ensemble rolling restarts as well) which is the core service discovery mechanism for Mesos. You have to put some sidekick services to provide peer discovery mechanics and it doesn’t look very robust.
Very common use case with Mesos is running Docker on top of it either with Marathon or with Aurora. But Docker service needs to be installed on worker nodes. So you’re coming to Docker-into-Docker situation which cancels all advantages of both transparent resource management and simple deployment configuration.
One more point on Mesos inside Docker here, is that you have to attach Mesos data directories from container to a host. Given that you’re already running Mesos container in privileged mode and sharing directories with state with host there is no more reasons to run Mesos inside a container. And consider container restart (not just failure) with registry corruption and following synchronization issues.
Another our use case with multi-region cluster deployments showed some issues with etcd heartbeat/leader election timeouts, which need to be increased in order to handle bigger latencies between data centers. If timeouts increase fleet starts to work in unpredictable way, loosing and finding peer nodes again which is not appropriate in production environment.
You can take a look at this configuration for Mesos-CoreOS-HA as well https://github.com/akirillov/mesos-deploy/tree/master/mesos-coreos-ha
--
Anton Kirillov
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Tuesday, March 10, 2015 at 11:08 AM, Gurvinder Singh wrote:
> Hi Micheal,
>
> Yes I tested the tutorial and it works fine for testing. Later on I used
> fleet to run mesos workers on all coreos machines too. I was wondering
> how the landscape is looking in the community regarding coreos. As is
> there any interest from community or mesos team to support coreos in
> general. If yes the then how you see where Mesos fits in with Fleet,
> Kubernetes.
>
> My current understanding is that Fleet is useful for lightweight
> scheduling, where as Mesos and kubernetes are kind of serving the
> similar purpose. Mesos has been here for a while and more feature
> complete than kubernetes. But Kubernetes has more tight integeration
> with coreos like use etcd for co-ordination, flannel for networking. I
> wonder what's the plan are when it comes to Mesos for such. I have seen
> the JIRA for etcd (https://issues.apache.org/jira/browse/MESOS-1806)
>
> I understand that the landscape is changing fast but its good to know
> about Mesos roadmap in this regard. Also would love to know if anybody
> using Coreos with Mesos beyond testing.
>
> Thanks,
> Gurvinder
> On 03/09/2015 11:35 PM, Michael Park wrote:
> > Hi Gurvinder,
> >
> > We got started on this work at Mesosphere and there's a tutorial
> > <http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/>
> > on
> > how to do a single-node setup. We ran the mesos-master and slaves in
> > docker containers which led to this JIRA ticket
> > <https://issues.apache.org/jira/browse/MESOS-2115>. I haven't been able
> > to follow-up on this article recently, and I'd like to hear about others
> > who have made further progress as well.
> >
> > At the time, we were thinking that using fleet shouldn't be too
> > difficult since it uses the systemd unit files but didn't quite get
> > around to it.
> >
> > Perhaps you'll find the tutorial to be a decent starting point.
> >
> > Thanks,
> >
> > MPark.
> >
> > On 9 March 2015 at 17:52, Gurvinder Singh <gurvinder.singh@uninett.no (mailto:gurvinder.singh@uninett.no)
> > <ma...@uninett.no>> wrote:
> >
> > Hi,
> >
> > I am wondering if anybody in the community has looked into or are
> > running mesos on top of coreos. I would be interested to hear out your
> > experiences around following areas
> >
> > - Users management on coreos cluster and containers running with Mesos
> > - Are you using fleet to run mesos or run it as service in
> > cloud-config
> > and don't use fleet at all
> > - Networking among hosts flannel or ?
> > - Any other interesting insights you found considering such setup
> >
> > Thanks,
> > Gurvinder
> >
>
>
>
Re: mesos on coreos
Posted by Gurvinder Singh <gu...@uninett.no>.
Hi Micheal,
Yes I tested the tutorial and it works fine for testing. Later on I used
fleet to run mesos workers on all coreos machines too. I was wondering
how the landscape is looking in the community regarding coreos. As is
there any interest from community or mesos team to support coreos in
general. If yes the then how you see where Mesos fits in with Fleet,
Kubernetes.
My current understanding is that Fleet is useful for lightweight
scheduling, where as Mesos and kubernetes are kind of serving the
similar purpose. Mesos has been here for a while and more feature
complete than kubernetes. But Kubernetes has more tight integeration
with coreos like use etcd for co-ordination, flannel for networking. I
wonder what's the plan are when it comes to Mesos for such. I have seen
the JIRA for etcd (https://issues.apache.org/jira/browse/MESOS-1806)
I understand that the landscape is changing fast but its good to know
about Mesos roadmap in this regard. Also would love to know if anybody
using Coreos with Mesos beyond testing.
Thanks,
Gurvinder
On 03/09/2015 11:35 PM, Michael Park wrote:
> Hi Gurvinder,
>
> We got started on this work at Mesosphere and there's a tutorial
> <http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/>
> on
> how to do a single-node setup. We ran the mesos-master and slaves in
> docker containers which led to this JIRA ticket
> <https://issues.apache.org/jira/browse/MESOS-2115>. I haven't been able
> to follow-up on this article recently, and I'd like to hear about others
> who have made further progress as well.
>
> At the time, we were thinking that using fleet shouldn't be too
> difficult since it uses the systemd unit files but didn't quite get
> around to it.
>
> Perhaps you'll find the tutorial to be a decent starting point.
>
> Thanks,
>
> MPark.
>
> On 9 March 2015 at 17:52, Gurvinder Singh <gurvinder.singh@uninett.no
> <ma...@uninett.no>> wrote:
>
> Hi,
>
> I am wondering if anybody in the community has looked into or are
> running mesos on top of coreos. I would be interested to hear out your
> experiences around following areas
>
> - Users management on coreos cluster and containers running with Mesos
> - Are you using fleet to run mesos or run it as service in
> cloud-config
> and don't use fleet at all
> - Networking among hosts flannel or ?
> - Any other interesting insights you found considering such setup
>
> Thanks,
> Gurvinder
>
>
Re: mesos on coreos
Posted by Michael Park <mc...@gmail.com>.
Hi Gurvinder,
We got started on this work at Mesosphere and there's a tutorial
<http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/>
on
how to do a single-node setup. We ran the mesos-master and slaves in docker
containers which led to this JIRA ticket
<https://issues.apache.org/jira/browse/MESOS-2115>. I haven't been able to
follow-up on this article recently, and I'd like to hear about others who
have made further progress as well.
At the time, we were thinking that using fleet shouldn't be too difficult
since it uses the systemd unit files but didn't quite get around to it.
Perhaps you'll find the tutorial to be a decent starting point.
Thanks,
MPark.
On 9 March 2015 at 17:52, Gurvinder Singh <gu...@uninett.no>
wrote:
> Hi,
>
> I am wondering if anybody in the community has looked into or are
> running mesos on top of coreos. I would be interested to hear out your
> experiences around following areas
>
> - Users management on coreos cluster and containers running with Mesos
> - Are you using fleet to run mesos or run it as service in cloud-config
> and don't use fleet at all
> - Networking among hosts flannel or ?
> - Any other interesting insights you found considering such setup
>
> Thanks,
> Gurvinder
>