cgroups isolation: at most vs at least

[Chris Burroughs <ch...@gmail.com>]

I have two related questions about the cpu and memory subsystems and at 
most vs at least semantics.

## cpu

As I understand it, by default cpu.shares is used to give tasks at least 
the specified cpu resources.  Optional with cgroups_enable_cfs both 
cpu.shares and cpu.cfs_period_us/cpu.cfs_quota_us to provide both at 
most and at least guarantees [1].  This is a per slave/cluster(?) 
setting as opposed to per task.

Since cgroups_enable_cfs potentially could leave cpu resources unused I 
presume it has a corresponding advantage, but I'm not sure what it is. 
When would one be preferred over the another?

## mem

Memory uses memory.limit_in_bytes to limit at most how much memory a 
task can use.  There does not seem to be a way to specify that a task 
should have at least a certain amount of memory.  This makes sense for 
RSS (since 'bursting' to RSS  > physical is not a desirable state and 
part of what these limits should avoid).  However, from the kernel docs 
my understanding is that memory.limit_in_bytes also limits how much page 
cache a task can use.  The page cache is a shared resource and keeping 
the kernel from per-fetching and caching as much as it can is a 
surprising choice to me.

Is there a best practice for frameworks to choose memory limits that 
include RSS + page cache?  In particular, I'm unsure how to reason about 
page cache use since a page is counted against whichever cgroup happened 
to access it first.


[1] 
http://mail-archives.apache.org/mod_mbox/mesos-user/201310.mbox/%3CCA%2B8RcoTbSY%3DKm2jKTVbCid-G76ytFA%2Bq_Zmzu5zRGC-wE206Xg%40mail.gmail.com%3E

[2] https://www.kernel.org/doc/Documentation/cgroups/memory.txt

Re: cgroups isolation: at most vs at least

[Chris Burroughs <ch...@gmail.com>]

On 06/23/2014 08:41 PM, Benjamin Mahler wrote:
>> Since cgroups_enable_cfs potentially could leave cpu resources unused I
>> presume it has a corresponding advantage, but I'm not sure what it is. When
>> would one be preferred over the another?
>>
>
> Setting the upper limit provides predictability. This is desired when
> running online workloads, like web servers.
>
> For example, let's say that you're happily running along with all your web
> services using 16 cores instead of 4. When the system is further
> constrained and you're forced down to 4 cores you will have a bad time. :)
>
> It is currently configured at the slave level, but of course it might be
> nice to provide different cpu isolation characteristics at the executor
> level in the future.
>

Thanks that makes a lot of sense.  I agree it might be useful to allow 
this to be a finer grained setting than the slave level though.  That 
was batch jobs could 'burst' while service jobs are still humming along 
in their tight bounds.


>> ## mem
>> Is there a best practice for frameworks to choose memory limits that
>> include RSS + page cache?  In particular, I'm unsure how to reason about
>> page cache use since a page is counted against whichever cgroup happened to
>> access it first.
>>
>
> The page cache accounting was a surprise to us as well, have you seen Ian's
> reply here:
>
> http://mail-archives.apache.org/mod_mbox/mesos-user/201406.mbox/%3CCAAJX7shQ_FB6qmvDBYJv5%2Bdh5VgG3WyaedBY_cQo1bY4cPvsDA%40mail.gmail.com%3E
>
> I believe we didn't see this on newer kernels.
>

That thread is very helpful, although I admit that "it depends on the 
kernel version" does not full me with hope.

The use case I am studying mesos for might be (slightly) atypical in 
that I don't have a separate DFS or database that tasks are fetching 
from.  Instead both batch and latency-sensitive tasks are reading and 
writing from the local file system.  In many case the same files are 
read by multiple tasks  In that environment I really don't see a 
sensible way to slice up upper bounds on page cache use without the 
static partitioning resulting in significant inefficiencies.

Re: cgroups isolation: at most vs at least

[Benjamin Mahler <be...@gmail.com>]

On Mon, Jun 23, 2014 at 9:24 AM, Chris Burroughs <ch...@gmail.com>
wrote:

> I have two related questions about the cpu and memory subsystems and at
> most vs at least semantics.
>
> ## cpu
>
> As I understand it, by default cpu.shares is used to give tasks at least
> the specified cpu resources.  Optional with cgroups_enable_cfs both
> cpu.shares and cpu.cfs_period_us/cpu.cfs_quota_us to provide both at most
> and at least guarantees [1].  This is a per slave/cluster(?) setting as
> opposed to per task.
>
> Since cgroups_enable_cfs potentially could leave cpu resources unused I
> presume it has a corresponding advantage, but I'm not sure what it is. When
> would one be preferred over the another?
>

Setting the upper limit provides predictability. This is desired when
running online workloads, like web servers.

For example, let's say that you're happily running along with all your web
services using 16 cores instead of 4. When the system is further
constrained and you're forced down to 4 cores you will have a bad time. :)

It is currently configured at the slave level, but of course it might be
nice to provide different cpu isolation characteristics at the executor
level in the future.


>
> ## mem
>
> Memory uses memory.limit_in_bytes to limit at most how much memory a task
> can use.  There does not seem to be a way to specify that a task should
> have at least a certain amount of memory.  This makes sense for RSS (since
> 'bursting' to RSS  > physical is not a desirable state and part of what
> these limits should avoid).  However, from the kernel docs my understanding
> is that memory.limit_in_bytes also limits how much page cache a task can
> use.  The page cache is a shared resource and keeping the kernel from
> per-fetching and caching as much as it can is a surprising choice to me.
>
> Is there a best practice for frameworks to choose memory limits that
> include RSS + page cache?  In particular, I'm unsure how to reason about
> page cache use since a page is counted against whichever cgroup happened to
> access it first.
>

The page cache accounting was a surprise to us as well, have you seen Ian's
reply here:

http://mail-archives.apache.org/mod_mbox/mesos-user/201406.mbox/%3CCAAJX7shQ_FB6qmvDBYJv5%2Bdh5VgG3WyaedBY_cQo1bY4cPvsDA%40mail.gmail.com%3E

I believe we didn't see this on newer kernels.


>
>
> [1] http://mail-archives.apache.org/mod_mbox/mesos-user/
> 201310.mbox/%3CCA%2B8RcoTbSY%3DKm2jKTVbCid-G76ytFA%2Bq_
> Zmzu5zRGC-wE206Xg%40mail.gmail.com%3E
>
> [2] https://www.kernel.org/doc/Documentation/cgroups/memory.txt
>