You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by "aurelien.dehay@gmail.com" <au...@gmail.com> on 2015/11/30 17:55:22 UTC
Team organization around Mesos cluster
Hello.
I'm in the process of demonstrate and talk about mesos all around my company. Everybody is quite interested, by anytime we talk, they always raise the "operation" problem.
We are a quite big company (100k in France), we're doing operation and system management the "old way", with big operation team taking care of a lot of projects, with dozen of operating procedure for each task (from Apache restart to database restoration). Project team think that Mesos fits quite badly in this way of doing things, and wonder how "real people" running a Mesos cluster are doing.
Therefore, If you don't mind how you are doing things for operations, without disclosing any sensible information of course, any info would be appreciated.
Thanks.
Re: Team organization around Mesos cluster
Posted by Shuai Lin <li...@gmail.com>.
I think that would depends on how would you use the mesos cluster.
We have a mesos cluster of ~20 nodes to run all the production web
services. From the POV of other teams, the mesos cluster is like an
internal PaaS, and they only need to know how to manage their own apps -
how to create app instances and upgrade them (we do that with a slack chat
bot), much like the way you use a public PaaS.
The operational team does all the heavy lifting - server provisioning and
monitoring, shared service management (e.g. mysql/cache/mq) for apps
running in mesos cluster.
If you would create your own mesos framework, it would require more close
interaction between the project team and the operational team.
Hope that helps.
Regards,
Shuai
On Tue, Dec 1, 2015 at 12:55 AM, aurelien.dehay@gmail.com <
aurelien.dehay@gmail.com> wrote:
> Hello.
>
>
> I'm in the process of demonstrate and talk about mesos all around my
> company. Everybody is quite interested, by anytime we talk, they always
> raise the "operation" problem.
>
>
> We are a quite big company (100k in France), we're doing operation and
> system management the "old way", with big operation team taking care of a
> lot of projects, with dozen of operating procedure for each task (from
> Apache restart to database restoration). Project team think that Mesos fits
> quite badly in this way of doing things, and wonder how "real people"
> running a Mesos cluster are doing.
>
>
> Therefore, If you don't mind how you are doing things for operations,
> without disclosing any sensible information of course, any info would be
> appreciated.
>
>
> Thanks.
>
>
Re: Team organization around Mesos cluster
Posted by David Greenberg <ds...@gmail.com>.
Here's how we think of it:
Machines are the same. We just need lots of them. One team specializes in
keeping machines running (machines include Mesos).
Frameworks are typically powerful and multi-client (Aurora, Cook,
Marathon). One instance of a framework can serve anywhere from one team to
hundreds of users to tens of thousands of customers. So, most companies
need 1-10 frameworks. These teams could be tiny (0.5 person for something
simple, like Marathon) to moderate (2-4 developers adding features to a
framework).
Users can then engage directly with the framework team--an application
developer doesn't care about the details of how Marathon is interacting
with Mesos; they just want to set up their health checks and scale their
application.
On Tue, Dec 1, 2015 at 8:36 AM Aurélien DEHAY <au...@zorel.org>
wrote:
> Hello David.
>
>
> thanks for the feedback. I understand you have a quite restraint number of
> team to handle mesos+frameworks. Does the mesos team (and therefore the
> framework team) only take care of mesos (resp. frameworks)? Or do they have
> another tasks?
>
>
> thansk.
>
>
> ------------------------------
> *De :* David Greenberg <ds...@gmail.com>
> *Envoyé :* lundi 30 novembre 2015 18:01
> *À :* user@mesos.apache.org
> *Objet :* Re: Team organization around Mesos cluster
>
> We have a three-tier model:
>
> One team manages Mesos itself (from machine provioning to installing &
> configuring Mesos).
>
> A small group of teams each manage the "approved" production-quality
> frameworks; these teams are clients of the team above.
>
> Many teams use the frameworks, they are clients of the framework teams.
>
> Thus one team provides infrastructure, several teams provide different
> platforms (services, batch computing, etc), and many other teams consume
> the platforms for their particular applications.
>
> On Mon, Nov 30, 2015 at 11:55 AM aurelien.dehay@gmail.com <
> aurelien.dehay@gmail.com> wrote:
>
>> Hello.
>>
>>
>> I'm in the process of demonstrate and talk about mesos all around my
>> company. Everybody is quite interested, by anytime we talk, they always
>> raise the "operation" problem.
>>
>>
>> We are a quite big company (100k in France), we're doing operation and
>> system management the "old way", with big operation team taking care of a
>> lot of projects, with dozen of operating procedure for each task (from
>> Apache restart to database restoration). Project team think that Mesos fits
>> quite badly in this way of doing things, and wonder how "real people"
>> running a Mesos cluster are doing.
>>
>>
>> Therefore, If you don't mind how you are doing things for operations,
>> without disclosing any sensible information of course, any info would be
>> appreciated.
>>
>>
>> Thanks.
>>
>>
RE: Team organization around Mesos cluster
Posted by Aurélien DEHAY <au...@zorel.org>.
Hello David.
thanks for the feedback. I understand you have a quite restraint number of team to handle mesos+frameworks. Does the mesos team (and therefore the framework team) only take care of mesos (resp. frameworks)? Or do they have another tasks?
thansk.
________________________________
De : David Greenberg <ds...@gmail.com>
Envoyé : lundi 30 novembre 2015 18:01
À : user@mesos.apache.org
Objet : Re: Team organization around Mesos cluster
We have a three-tier model:
One team manages Mesos itself (from machine provioning to installing & configuring Mesos).
A small group of teams each manage the "approved" production-quality frameworks; these teams are clients of the team above.
Many teams use the frameworks, they are clients of the framework teams.
Thus one team provides infrastructure, several teams provide different platforms (services, batch computing, etc), and many other teams consume the platforms for their particular applications.
On Mon, Nov 30, 2015 at 11:55 AM aurelien.dehay@gmail.com<ma...@gmail.com> <au...@gmail.com>> wrote:
Hello.
I'm in the process of demonstrate and talk about mesos all around my company. Everybody is quite interested, by anytime we talk, they always raise the "operation" problem.
We are a quite big company (100k in France), we're doing operation and system management the "old way", with big operation team taking care of a lot of projects, with dozen of operating procedure for each task (from Apache restart to database restoration). Project team think that Mesos fits quite badly in this way of doing things, and wonder how "real people" running a Mesos cluster are doing.
Therefore, If you don't mind how you are doing things for operations, without disclosing any sensible information of course, any info would be appreciated.
Thanks.
Re: Team organization around Mesos cluster
Posted by Rodrick Brown <ro...@orchard-app.com>.
Right now we're using constraits on the slaves to isolate different workloads
to ensure one team can't DOS everyone else and this has been working very well
for us so far.
> On Nov 30 2015, at 12:41 pm, Harry Metske <harry.metske@gmail.com>
wrote:
>
> We are in a similar stage and also have similar questions, starting with
about 20 devops teams, but that could grow to over 50 teams.
>
> I am a bit worried having all these teams work in a shared mesos cluster.
>
> * they share the name-space for (marathon) applications
>
> * share all the mesos slave machines ==> share the CPU, share the same
diskspace, share the same memory, share the file hierarchy
>
> * I am not aware of any means to authorize the use of resources in the
cluster
>
> * within marathon you can run a task with any userid, even root (if you run
mesos slave with root)
>
> * hard to audit (who did what in the cluster)
>
>
>
> Having a couple of years experience with these same teams in a more classic
environment (JEE AppServers), I would opt for a Mesos cluster per devops and
take the "infrastructure waste" for granted.
>
>
>
> In short the question is how to properly operate the cluster(s), and how
"multi tenant proof" is it, right?
>
>
>
> kind regards,
>
> Harry
>
>
>
>
>
>
>
>
>
>
>
> On 30 November 2015 at 18:01, David Greenberg
<[dsg123456789@gmail.com](mailto:dsg123456789@gmail.com)> wrote:
>
>> We have a three-tier model:
>>
>>
>>
>> One team manages Mesos itself (from machine provioning to installing &
configuring Mesos).
>>
>>
>>
>> A small group of teams each manage the "approved" production-quality
frameworks; these teams are clients of the team above.
>>
>>
>>
>> Many teams use the frameworks, they are clients of the framework teams.
>>
>>
>>
>> Thus one team provides infrastructure, several teams provide different
platforms (services, batch computing, etc), and many other teams consume the
platforms for their particular applications.
>>
>>
>>
>> On Mon, Nov 30, 2015 at 11:55 AM
[aurelien.dehay@gmail.com](mailto:aurelien.dehay@gmail.com)
<[aurelien.dehay@gmail.com](mailto:aurelien.dehay@gmail.com)> wrote:
>>
>>> Hello.
>>>
>>>
>>>
>>> I'm in the process of demonstrate and talk about mesos all around my
company. Everybody is quite interested, by anytime we talk, they always raise
the "operation" problem.
>>>
>>>
>>>
>>> We are a quite big company (100k in France), we're doing operation and
system management the "old way", with big operation team taking care of a lot
of projects, with dozen of operating procedure for each task (from Apache
restart to database restoration). Project team think that Mesos fits quite
badly in this way of doing things, and wonder how "real people" running a
Mesos cluster are doing.
>>>
>>>
>>>
>>> Therefore, If you don't mind how you are doing things for operations,
without disclosing any sensible information of course, any info would be
appreciated.
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>
>
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.
Re: Team organization around Mesos cluster
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
You should really take a closer look at Apache Aurora. It is build with multi-user support in mind, and has quite a few features helping in that regard:
* namespaces per user and per environment
* authentication
* production quota and preemption
* ...
Best Regards,
Stephan
________________________________
From: Harry Metske <ha...@gmail.com>
Sent: Monday, November 30, 2015 6:40 PM
To: user@mesos.apache.org
Subject: Re: Team organization around Mesos cluster
We are in a similar stage and also have similar questions, starting with about 20 devops teams, but that could grow to over 50 teams.
I am a bit worried having all these teams work in a shared mesos cluster.
* they share the name-space for (marathon) applications
* share all the mesos slave machines ==> share the CPU, share the same diskspace, share the same memory, share the file hierarchy
* I am not aware of any means to authorize the use of resources in the cluster
* within marathon you can run a task with any userid, even root (if you run mesos slave with root)
* hard to audit (who did what in the cluster)
Having a couple of years experience with these same teams in a more classic environment (JEE AppServers), I would opt for a Mesos cluster per devops and take the "infrastructure waste" for granted.
In short the question is how to properly operate the cluster(s), and how "multi tenant proof" is it, right?
kind regards,
Harry
On 30 November 2015 at 18:01, David Greenberg <ds...@gmail.com>> wrote:
We have a three-tier model:
One team manages Mesos itself (from machine provioning to installing & configuring Mesos).
A small group of teams each manage the "approved" production-quality frameworks; these teams are clients of the team above.
Many teams use the frameworks, they are clients of the framework teams.
Thus one team provides infrastructure, several teams provide different platforms (services, batch computing, etc), and many other teams consume the platforms for their particular applications.
On Mon, Nov 30, 2015 at 11:55 AM aurelien.dehay@gmail.com<ma...@gmail.com> <au...@gmail.com>> wrote:
Hello.
I'm in the process of demonstrate and talk about mesos all around my company. Everybody is quite interested, by anytime we talk, they always raise the "operation" problem.
We are a quite big company (100k in France), we're doing operation and system management the "old way", with big operation team taking care of a lot of projects, with dozen of operating procedure for each task (from Apache restart to database restoration). Project team think that Mesos fits quite badly in this way of doing things, and wonder how "real people" running a Mesos cluster are doing.
Therefore, If you don't mind how you are doing things for operations, without disclosing any sensible information of course, any info would be appreciated.
Thanks.
Re: Team organization around Mesos cluster
Posted by Harry Metske <ha...@gmail.com>.
We are in a similar stage and also have similar questions, starting with
about 20 devops teams, but that could grow to over 50 teams.
I am a bit worried having all these teams work in a shared mesos cluster.
* they share the name-space for (marathon) applications
* share all the mesos slave machines ==> share the CPU, share the same
diskspace, share the same memory, share the file hierarchy
* I am not aware of any means to authorize the use of resources in the
cluster
* within marathon you can run a task with any userid, even root (if you run
mesos slave with root)
* hard to audit (who did what in the cluster)
Having a couple of years experience with these same teams in a more classic
environment (JEE AppServers), I would opt for a Mesos cluster per devops
and take the "infrastructure waste" for granted.
In short the question is how to properly operate the cluster(s), and how
"multi tenant proof" is it, right?
kind regards,
Harry
On 30 November 2015 at 18:01, David Greenberg <ds...@gmail.com>
wrote:
> We have a three-tier model:
>
> One team manages Mesos itself (from machine provioning to installing &
> configuring Mesos).
>
> A small group of teams each manage the "approved" production-quality
> frameworks; these teams are clients of the team above.
>
> Many teams use the frameworks, they are clients of the framework teams.
>
> Thus one team provides infrastructure, several teams provide different
> platforms (services, batch computing, etc), and many other teams consume
> the platforms for their particular applications.
>
> On Mon, Nov 30, 2015 at 11:55 AM aurelien.dehay@gmail.com <
> aurelien.dehay@gmail.com> wrote:
>
>> Hello.
>>
>>
>> I'm in the process of demonstrate and talk about mesos all around my
>> company. Everybody is quite interested, by anytime we talk, they always
>> raise the "operation" problem.
>>
>>
>> We are a quite big company (100k in France), we're doing operation and
>> system management the "old way", with big operation team taking care of a
>> lot of projects, with dozen of operating procedure for each task (from
>> Apache restart to database restoration). Project team think that Mesos fits
>> quite badly in this way of doing things, and wonder how "real people"
>> running a Mesos cluster are doing.
>>
>>
>> Therefore, If you don't mind how you are doing things for operations,
>> without disclosing any sensible information of course, any info would be
>> appreciated.
>>
>>
>> Thanks.
>>
>>
Re: Team organization around Mesos cluster
Posted by David Greenberg <ds...@gmail.com>.
We have a three-tier model:
One team manages Mesos itself (from machine provioning to installing &
configuring Mesos).
A small group of teams each manage the "approved" production-quality
frameworks; these teams are clients of the team above.
Many teams use the frameworks, they are clients of the framework teams.
Thus one team provides infrastructure, several teams provide different
platforms (services, batch computing, etc), and many other teams consume
the platforms for their particular applications.
On Mon, Nov 30, 2015 at 11:55 AM aurelien.dehay@gmail.com <
aurelien.dehay@gmail.com> wrote:
> Hello.
>
>
> I'm in the process of demonstrate and talk about mesos all around my
> company. Everybody is quite interested, by anytime we talk, they always
> raise the "operation" problem.
>
>
> We are a quite big company (100k in France), we're doing operation and
> system management the "old way", with big operation team taking care of a
> lot of projects, with dozen of operating procedure for each task (from
> Apache restart to database restoration). Project team think that Mesos fits
> quite badly in this way of doing things, and wonder how "real people"
> running a Mesos cluster are doing.
>
>
> Therefore, If you don't mind how you are doing things for operations,
> without disclosing any sensible information of course, any info would be
> appreciated.
>
>
> Thanks.
>
>