You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Daniel Gaston <Da...@Dal.Ca> on 2015/06/19 19:03:29 UTC
Thoughts and opinions in physically building a cluster
Hi Everyone,
I've looked through the archives and the web but still have some questions on this question.
1) If I was looking at building a small compute/HPC cluster is Mesos overkill in such a situation?
2) What is the minimum number of physical nodes? It seems from documentation and examples ideally this is something like 5, with 3 masters and say two slaves.
3) What are some other good resources in terms of doing this? Appropriate specs for individual nodes, particularly where you would likely want slave/compute nodes to be much beefier than Master nodes. What other equipment would you need, just nodes and switches?
4) Would it make sense to have a smaller number of physical nodes split up into virtual nodes or will this just make everything much more complex?
Any thoughts, opinions, or directions to resources is much appreciated!
Cheers,
Dan
Re: Thoughts and opinions in physically building a cluster
Posted by CCAAT <cc...@tampabay.rr.com>.
On 06/19/2015 01:45 PM, Dave Martens wrote:
> Thanks for all of these comments - I had similar questions.
>
> What is the minimum RAM for a master or a slave? I have heard that the
> Mesos slave software adds 1GB of RAM on top of what the slave's workload
> processing will require. I have read that 8GB is the min for a Mesos
> machine but it wasn't clear that this was an official/hard requirement.
There are probably published/standard numbers for the various distros
that the slave node is build upon (actual or virtual). Actually with a
robust (CI) infrastructure, these sort of resource metrics and the
various benchmarks should be revealed to the user community, routinely.
I'm not certain on what, if any, of this sort of data is being published.
If you tune (stip) the Operating System or the virtual image or the
kernel, then these numbers are most likely lower. I'm not sure much has
been published on "tuning" the OS, kernels or installs for mesos. HPC
offerings will surely be pushing the envelop of these and many more
related metrics for performance tuning to specialized classes of
problem, hardware specifics and other goals of performance tuning.
Most will run on "bloat_ware" but the smarter datacenters and HPC folks
will 'cut the pork' for those single digit performance gains. YMMV.
hth,
James
Re: Thoughts and opinions in physically building a cluster
Posted by Dave Martens <da...@gmail.com>.
Thanks for all of these comments - I had similar questions.
What is the minimum RAM for a master or a slave? I have heard that the
Mesos slave software adds 1GB of RAM on top of what the slave's workload
processing will require. I have read that 8GB is the min for a Mesos
machine but it wasn't clear that this was an official/hard requirement.
On Fri, Jun 19, 2015 at 11:31 AM, Eelco Maljaars | Maljaars IT <
eelco@maljaars-it.nl> wrote:
> Hi,
>
>
> exactly what I’ve been doing for a few smaller setups. I see this as the
> minimum ‘production’ setup. For testing or development I just run
> everything on a single node, sometimes even a VM.
>
> Regards,
>
> Eelco
>
>
> On 19 Jun 2015, at 20:23, Oliver Nicholas <bi...@uber.com> wrote:
>
>
> On Fri, Jun 19, 2015 at 11:22 AM, Brian Candler <b....@pobox.com>
> wrote:
>
>> On 19/06/2015 18:38, Oliver Nicholas wrote:
>>
>>> Unless you have some true HA requirements, it seems intuitively wasteful
>>> to have 3 masters and 2 slaves (unless the cost of 5 nodes is
>>> inconsequential to you and you hate the environment).
>>>
>> Any particular reason not to have three nodes which are acting both as
>> master and slaves?
>>
>
> Certainly seems reasonable to me!
>
> --
> *bigo* / oliver nicholas | staff engineer, infrastructure | uber
> technologies, inc.
>
>
>
Re: Thoughts and opinions in physically building a cluster
Posted by Eelco Maljaars | Maljaars IT <ee...@maljaars-it.nl>.
Hi,
exactly what I’ve been doing for a few smaller setups. I see this as the minimum ‘production’ setup. For testing or development I just run everything on a single node, sometimes even a VM.
Regards,
Eelco
> On 19 Jun 2015, at 20:23, Oliver Nicholas <bi...@uber.com> wrote:
>
>
> On Fri, Jun 19, 2015 at 11:22 AM, Brian Candler <b.candler@pobox.com <ma...@pobox.com>> wrote:
> On 19/06/2015 18:38, Oliver Nicholas wrote:
> Unless you have some true HA requirements, it seems intuitively wasteful to have 3 masters and 2 slaves (unless the cost of 5 nodes is inconsequential to you and you hate the environment).
> Any particular reason not to have three nodes which are acting both as master and slaves?
>
> Certainly seems reasonable to me!
>
> --
> bigo / oliver nicholas | staff engineer, infrastructure | uber technologies, inc.
Re: Thoughts and opinions in physically building a cluster
Posted by Oliver Nicholas <bi...@uber.com>.
On Fri, Jun 19, 2015 at 11:22 AM, Brian Candler <b....@pobox.com> wrote:
> On 19/06/2015 18:38, Oliver Nicholas wrote:
>
>> Unless you have some true HA requirements, it seems intuitively wasteful
>> to have 3 masters and 2 slaves (unless the cost of 5 nodes is
>> inconsequential to you and you hate the environment).
>>
> Any particular reason not to have three nodes which are acting both as
> master and slaves?
>
Certainly seems reasonable to me!
--
*bigo* / oliver nicholas | staff engineer, infrastructure | uber
technologies, inc.
Re: Thoughts and opinions in physically building a cluster
Posted by Daniel Gaston <Da...@Dal.Ca>.
Yes, I'm exploring many options at the moment but this would be an extremely small compute cluster (4 nodes max most likely to start). HA isn't required as it isn't serving up applications or the like. We would want to minimize downtime but if a node goes down for a few hours it isn't a big deal. So I'm just trying to figure everything out on the metal/architecture end of things. Setting up stuff on AWS and the like is fairly easy and straightforward as it is all virtualized, but physically organizing nodes and how to connect everything properly is a bit more challenging. Especially when you're not a networking guy.
________________________________________
From: rasputnik@gmail.com <ra...@gmail.com> on behalf of Dick Davies <di...@hellooperator.net>
Sent: June-25-15 1:46 PM
To: user@mesos.apache.org
Subject: Re: Thoughts and opinions in physically building a cluster
That doesn't sound too bad (it's a fairly typical setup e.g. on an Amazon VPC).
You probably want to avoid NAT or similar things between master and
slaves to avoid
a lot of LIBPROCESS_IP tricks so same switch sounds good.
Personally I quite like the master/slave distinction.
I wouldn't want a runaway set of tasks to bog down the masters and
operationally we'd alert
if we're starting to lose masters whereas the slaves are 'cattle' and
we can just spin up more as
they die if need be (it's a little more tricky to scale out masters
and zookeepers so they get treated
as though they were a bit less expendable).
I co-locate the zookeeper ensemble on the masters on smaller clusters
to save VM count,
but that's more personal taste than anything.
On 25 June 2015 at 17:12, Daniel Gaston <Da...@dal.ca> wrote:
> So this may be another relatively noob question, but when designing a mesos cluster, is it basically as simple as the nodes connected by a switch? Since any of the nodes can be "master nodes" or acting as both master and slave, I am guessing there is no need for another head node as you would have with a traditional cluster design. But would each of the nodes then have to be connected to the external/institutional network?
>
> My rough idea was for this small cluster to not be connected to the main institutional network but for my workstation to be connected to both the cluster's network as well as to the institutional network
>
>
> ________________________________________
> From: CCAAT <cc...@tampabay.rr.com>
> Sent: June-19-15 4:57 PM
> To: user@mesos.apache.org
> Cc: ccaat@tampabay.rr.com
> Subject: Re: Thoughts and opinions in physically building a cluster
>
> On 06/19/2015 01:28 PM, Daniel Gaston wrote:
>>
>> On 19/06/2015 18:38, Oliver Nicholas wrote:
>>> Unless you have some true HA requirements, it seems intuitively
>>> wasteful to have 3 masters and 2 slaves (unless the cost of 5 nodes is
>>> inconsequential to you and you hate the environment).
>> Any particular reason not to have three nodes which are acting both as
>> master and slaves?
>>
>> None at all. I'm not a cluster or networking guru, and have only played with mesos in
>> cloud-based settings so I wasn't sure how this would work. But it makes sense, that way
>> the 'standby' masters are still participating in the zookeeper quorum while still being
>> available to do real work as slave nodes.
>
> Daniel. There is no such thing as a 'cluster guru'. It's all 'seat of
> the pants' flying right now; so you are fine what you are doing and
> propose. If codes do not exist to meet your specific needs and goals,
> they can (should?) be created.
>
>
> I'm working on an architectural expansion Where nodes (virtual, actual
> or bare metal) migrate from master --> entrepreneur --> worker --> slave
> --> embedded (bare metal or specially attached hardware. I'm proposing
> to do all of this with the "Autonomy_Function" and decisions being made
> bottom_up as opposed to the current top_down dichotomy. I'm prolly going
> to have to 'fork codes' for a while to get things stable and then
> hope they are included; when other minds see the validity of the ideas.
>
>
> Surely one "box" can be set up as both master and slave. Moving slaves
> to masters, should be an automatic function and will prolly will be
> address in the future codes of mesos.
>
>
> PS: Keep pushing your ideas and do not take no for an answer!
> Mesos belongs to everybody.....
>
> hth,
> James
>
Re: Thoughts and opinions in physically building a cluster
Posted by Dick Davies <di...@hellooperator.net>.
That doesn't sound too bad (it's a fairly typical setup e.g. on an Amazon VPC).
You probably want to avoid NAT or similar things between master and
slaves to avoid
a lot of LIBPROCESS_IP tricks so same switch sounds good.
Personally I quite like the master/slave distinction.
I wouldn't want a runaway set of tasks to bog down the masters and
operationally we'd alert
if we're starting to lose masters whereas the slaves are 'cattle' and
we can just spin up more as
they die if need be (it's a little more tricky to scale out masters
and zookeepers so they get treated
as though they were a bit less expendable).
I co-locate the zookeeper ensemble on the masters on smaller clusters
to save VM count,
but that's more personal taste than anything.
On 25 June 2015 at 17:12, Daniel Gaston <Da...@dal.ca> wrote:
> So this may be another relatively noob question, but when designing a mesos cluster, is it basically as simple as the nodes connected by a switch? Since any of the nodes can be "master nodes" or acting as both master and slave, I am guessing there is no need for another head node as you would have with a traditional cluster design. But would each of the nodes then have to be connected to the external/institutional network?
>
> My rough idea was for this small cluster to not be connected to the main institutional network but for my workstation to be connected to both the cluster's network as well as to the institutional network
>
>
> ________________________________________
> From: CCAAT <cc...@tampabay.rr.com>
> Sent: June-19-15 4:57 PM
> To: user@mesos.apache.org
> Cc: ccaat@tampabay.rr.com
> Subject: Re: Thoughts and opinions in physically building a cluster
>
> On 06/19/2015 01:28 PM, Daniel Gaston wrote:
>>
>> On 19/06/2015 18:38, Oliver Nicholas wrote:
>>> Unless you have some true HA requirements, it seems intuitively
>>> wasteful to have 3 masters and 2 slaves (unless the cost of 5 nodes is
>>> inconsequential to you and you hate the environment).
>> Any particular reason not to have three nodes which are acting both as
>> master and slaves?
>>
>> None at all. I'm not a cluster or networking guru, and have only played with mesos in
>> cloud-based settings so I wasn't sure how this would work. But it makes sense, that way
>> the 'standby' masters are still participating in the zookeeper quorum while still being
>> available to do real work as slave nodes.
>
> Daniel. There is no such thing as a 'cluster guru'. It's all 'seat of
> the pants' flying right now; so you are fine what you are doing and
> propose. If codes do not exist to meet your specific needs and goals,
> they can (should?) be created.
>
>
> I'm working on an architectural expansion Where nodes (virtual, actual
> or bare metal) migrate from master --> entrepreneur --> worker --> slave
> --> embedded (bare metal or specially attached hardware. I'm proposing
> to do all of this with the "Autonomy_Function" and decisions being made
> bottom_up as opposed to the current top_down dichotomy. I'm prolly going
> to have to 'fork codes' for a while to get things stable and then
> hope they are included; when other minds see the validity of the ideas.
>
>
> Surely one "box" can be set up as both master and slave. Moving slaves
> to masters, should be an automatic function and will prolly will be
> address in the future codes of mesos.
>
>
> PS: Keep pushing your ideas and do not take no for an answer!
> Mesos belongs to everybody.....
>
> hth,
> James
>
Re: Thoughts and opinions in physically building a cluster
Posted by Daniel Gaston <Da...@Dal.Ca>.
So this may be another relatively noob question, but when designing a mesos cluster, is it basically as simple as the nodes connected by a switch? Since any of the nodes can be "master nodes" or acting as both master and slave, I am guessing there is no need for another head node as you would have with a traditional cluster design. But would each of the nodes then have to be connected to the external/institutional network?
My rough idea was for this small cluster to not be connected to the main institutional network but for my workstation to be connected to both the cluster's network as well as to the institutional network
________________________________________
From: CCAAT <cc...@tampabay.rr.com>
Sent: June-19-15 4:57 PM
To: user@mesos.apache.org
Cc: ccaat@tampabay.rr.com
Subject: Re: Thoughts and opinions in physically building a cluster
On 06/19/2015 01:28 PM, Daniel Gaston wrote:
>
> On 19/06/2015 18:38, Oliver Nicholas wrote:
>> Unless you have some true HA requirements, it seems intuitively
>> wasteful to have 3 masters and 2 slaves (unless the cost of 5 nodes is
>> inconsequential to you and you hate the environment).
> Any particular reason not to have three nodes which are acting both as
> master and slaves?
>
> None at all. I'm not a cluster or networking guru, and have only played with mesos in
> cloud-based settings so I wasn't sure how this would work. But it makes sense, that way
> the 'standby' masters are still participating in the zookeeper quorum while still being
> available to do real work as slave nodes.
Daniel. There is no such thing as a 'cluster guru'. It's all 'seat of
the pants' flying right now; so you are fine what you are doing and
propose. If codes do not exist to meet your specific needs and goals,
they can (should?) be created.
I'm working on an architectural expansion Where nodes (virtual, actual
or bare metal) migrate from master --> entrepreneur --> worker --> slave
--> embedded (bare metal or specially attached hardware. I'm proposing
to do all of this with the "Autonomy_Function" and decisions being made
bottom_up as opposed to the current top_down dichotomy. I'm prolly going
to have to 'fork codes' for a while to get things stable and then
hope they are included; when other minds see the validity of the ideas.
Surely one "box" can be set up as both master and slave. Moving slaves
to masters, should be an automatic function and will prolly will be
address in the future codes of mesos.
PS: Keep pushing your ideas and do not take no for an answer!
Mesos belongs to everybody.....
hth,
James
Re: Thoughts and opinions in physically building a cluster
Posted by CCAAT <cc...@tampabay.rr.com>.
On 06/19/2015 01:28 PM, Daniel Gaston wrote:
>
> On 19/06/2015 18:38, Oliver Nicholas wrote:
>> Unless you have some true HA requirements, it seems intuitively
>> wasteful to have 3 masters and 2 slaves (unless the cost of 5 nodes is
>> inconsequential to you and you hate the environment).
> Any particular reason not to have three nodes which are acting both as
> master and slaves?
>
> None at all. I'm not a cluster or networking guru, and have only played with mesos in
> cloud-based settings so I wasn't sure how this would work. But it makes sense, that way
> the 'standby' masters are still participating in the zookeeper quorum while still being
> available to do real work as slave nodes.
Daniel. There is no such thing as a 'cluster guru'. It's all 'seat of
the pants' flying right now; so you are fine what you are doing and
propose. If codes do not exist to meet your specific needs and goals,
they can (should?) be created.
I'm working on an architectural expansion Where nodes (virtual, actual
or bare metal) migrate from master --> entrepreneur --> worker --> slave
--> embedded (bare metal or specially attached hardware. I'm proposing
to do all of this with the "Autonomy_Function" and decisions being made
bottom_up as opposed to the current top_down dichotomy. I'm prolly going
to have to 'fork codes' for a while to get things stable and then
hope they are included; when other minds see the validity of the ideas.
Surely one "box" can be set up as both master and slave. Moving slaves
to masters, should be an automatic function and will prolly will be
address in the future codes of mesos.
PS: Keep pushing your ideas and do not take no for an answer!
Mesos belongs to everybody.....
hth,
James
Re: Thoughts and opinions in physically building a cluster
Posted by Daniel Gaston <Da...@Dal.Ca>.
On 19/06/2015 18:38, Oliver Nicholas wrote:
> Unless you have some true HA requirements, it seems intuitively
> wasteful to have 3 masters and 2 slaves (unless the cost of 5 nodes is
> inconsequential to you and you hate the environment).
Any particular reason not to have three nodes which are acting both as
master and slaves?
None at all. I'm not a cluster or networking guru, and have only played with mesos in cloud-based settings so I wasn't sure how this would work. But it makes sense, that way the 'standby' masters are still participating in the zookeeper quorum while still being available to do real work as slave nodes.
Re: Thoughts and opinions in physically building a cluster
Posted by Brian Candler <b....@pobox.com>.
On 19/06/2015 18:38, Oliver Nicholas wrote:
> Unless you have some true HA requirements, it seems intuitively
> wasteful to have 3 masters and 2 slaves (unless the cost of 5 nodes is
> inconsequential to you and you hate the environment).
Any particular reason not to have three nodes which are acting both as
master and slaves?
Re: Thoughts and opinions in physically building a cluster
Posted by Daniel Gaston <Da...@Dal.Ca>.
Thanks Oliver, a lot of great suggestions. One of the reasons I was interested in Mesos was the idea of it being more generalized. While this small HPC cluster will serve one primary job, it will also be used for research purposes. So being able to easily test out frameworks and not be 'locked in' to one way of doing things is appealing. Most jobs are relatively CPU/RAM heavy (and small file disk I/O unfortunately) but I already have a good handle on building individual compute servers that would handle that, so would be suitable slave nodes/compute clusters. HA would be nice in terms of ensuring turn-around times on workflows, but likely isn't a major issue in terms of if it is down for a few hours no one will lose any sleep or die. If the node could be brought back up reasonably it should be fine.
Re: Thoughts and opinions in physically building a cluster
Posted by Oliver Nicholas <bi...@uber.com>.
On Fri, Jun 19, 2015 at 10:03 AM, Daniel Gaston <Da...@dal.ca>
wrote:
> Hi Everyone,
>
> I've looked through the archives and the web but still have some questions
> on this question.
>
> 1) If I was looking at building a small compute/HPC cluster is Mesos
> overkill in such a situation?
>
>
> Mesos isn't overkill, though there may be platforms developed more
specifically for your use case (vs. Mesos which is extremely generalized).
> 2) What is the minimum number of physical nodes? It seems from
> documentation and examples ideally this is something like 5, with 3 masters
> and say two slaves.
>
Technically speaking, you can do it all with one node. It just depends
what properties you need. Having three masters (or any HA grouping, ie/ an
odd number greater than 1) is overkill if high availability isn't a
requirement - you can just have a single master node and live with the fact
that if it goes down, you can't schedule any new tasks until you bring it
back.
Unless you have some true HA requirements, it seems intuitively wasteful to
have 3 masters and 2 slaves (unless the cost of 5 nodes is inconsequential
to you and you hate the environment).
> 3) What are some other good resources in terms of doing this?
> Appropriate specs for individual nodes, particularly where you would likely
> want slave/compute nodes to be much beefier than Master nodes. What other
> equipment would you need, just nodes and switches?
>
Depends what your workloads look like. Mesos itself (both master and
slave) is very thin - under most circumstances it won't even need a whole
CPU core to itself. Remember, Mesos itself doesn't do any real work other
than coordination - it's the processes you use it to schedule/run that are
going to use up the physical resources.
So the question you ask yourself in this situation is which primary
resources does my workload use? Is it CPU heavy, memory heavy, maybe disk
or network I/O heavy? That's how you decide what machines to throw at it.
The question is more or less the same whether you use Mesos to schedule or
not. Identifying resource requirements should be possible both by
understanding what the process does, and by measuring it with standard unix
tools.
As for the second part of your question, you just need a set of computers
that can run modern Linux and talk to each other over TCP/IP. You probably
want them on a private network.
> 4) Would it make sense to have a smaller number of physical nodes split up
> into virtual nodes or will this just make everything much more complex?
>
This is probably not necessary. Mesos has native support for process
isolation via cgroups, which obviates one of the advantages of VMs.
Structurally, the whole *point* of Mesos is to abstract away the concept of
individual machines into pools of compute capacity, so you're kinda working
at cross purposes if you go down this road too far.
> Any thoughts, opinions, or directions to resources is much appreciated!
>
>
>
> Cheers,
> Dan
>
>
>
--
*bigo* / oliver nicholas | staff engineer, infrastructure | uber
technologies, inc.