Re: UIMA DUCC limit max memory of node

[Daniel Baumartz <ba...@stud.uni-frankfurt.de>]

Hi Eddie,

ok, I will try to explain with more detail, maybe this is not how ducc  
is being used normally. We want to set up some nodes which are not  
exclusively used for ducc. For example, one of the nodes may have 100  
GB, but we want the usable memory for ducc to only be 50 GB, not all  
free memory. (We also want to limit the CPU usage, for example only  
use 32 of 64 cores, but we have not tried to set this up yet.)

We could not find any setting to achieve this behavior, so we tried  
using cgroups to limit the max usable memory for ducc. This did not  
work because ducc gets its memory info from /proc/meminfo which  
ignores the cgroups settings. After reading through the code it seems  
only setting "ducc.agent.node.metrics.fake.memory.size" (not setting  
up test mode) is doing something similar to what we want: "Comment  
from NodeMemInfoCollector.java: if running ducc in simulation mode  
skip memory adjustment. Report free memory = fakeMemorySize". But I am  
not sure if we can use this safely since it is for testing.

So we basically want to give ducc an upper limit of usable memory.

I hope it is a bit more clear what we want to achieve.

Thanks again,
Daniel


Zitat von Eddie Epstein <ea...@gmail.com>:

> Hi Daniel,
>
> For each node Ducc sums RSS for all "system" user processes and excludes
> that from Ducc usable memory on the node. System users are defined by a
> ducc.properties setting with default value:
> ducc.agent.node.metrics.sys.gid.max = 500
>
> Ducc's simulation mode is intended for creating a scaled out cluster of
> fake nodes for testing purposes.
>
> The only mechanism available for reserving additional memory is to have
> Ducc run some dummy process that stays up forever. This could be a Ducc
> service that is automatically started when Ducc starts. This could get
> complicated for a heterogeneous set of machines and/or Ducc classes.
>
> Can you be more precise of what features you are looking for limiting
> resource use of Ducc machines?
>
> Thanks,
> Eddie
>
>
> On Mon, Oct 31, 2016 at 10:03 AM, Daniel Baumartz <
> baumartz@stud.uni-frankfurt.de> wrote:
>
>> Hi,
>>
>> I am trying to set up nodes for Ducc that should not use all the memory on
>> the machine. I tried to limit the memory with cgroups, but it seems Ducc is
>> getting the memory info from /proc/meminfo which ignores the cgroups
>> settings.
>>
>> Did I miss an option to specify the max usable memory? Could I safely use
>> "ducc.agent.node.metrics.fake.memory.size" from the simulation settings?
>> Or is there a better way to do this?
>>
>> Thanks,
>> Daniel
>>
>>

Re: UIMA DUCC limit max memory of node

[Eddie Epstein <ea...@gmail.com>]

Hi,

You are right that ducc.agent.node.metrics.fake.memory.size will override
the agent's computation of total usable memory. This must be set as a java
property on the agent. To set this for all agents, add the following line
to site.ducc.properties in the resources folder and restart DUCC.
    ducc.agent.jvm.args    = -Xmx500M -Dducc.agent.node.metrics.fake.
memory.size=N
where N is in KB.

DUCC uses cgset -r cpu.shares=M to control a containers CPU. M is computed
as
   container-size-in-byes / total-memory-size-in-KB. So the maximum value
for M in a DUCC container is 1024.

cpu.shares controls CPU usage in a relative way. A container with
cpu.shares=1024 will potentially get 2x the CPU than a container with 512
shares. Note that if a container is using less than its share, other
containers will be allowed to get more than their share.

For newer OS, e.g. RHEL7, processes not put into a specific container are
put into the default container with cpu.shares = 1024. So if you break a
machine in half using fake.memory, and if DUCC were to fill its half up
with work, then the two halves of the box would have equal shares. Sounds
good for your scenario.

However, note that cpu.shares works for CPUs, not cores. so things may not
be so nice if hyperthreading is enabled. For example, consider a machine
with 32 cores and 2-way hyperthreading. A process burning 32 CPUs may
pretty much "max out" the machine even though there are 32 unused CPUs
available. To limit the DUCC half of a machine to only half the real
machine resources would require changing agent code to use the "*Ceiling
Enforcement Tunable Parameters*" which are absolute.

Eddie


On Tue, Nov 1, 2016 at 9:31 AM, Daniel Baumartz <
baumartz@stud.uni-frankfurt.de> wrote:

> Hi Eddie,
>
> ok, I will try to explain with more detail, maybe this is not how ducc is
> being used normally. We want to set up some nodes which are not exclusively
> used for ducc. For example, one of the nodes may have 100 GB, but we want
> the usable memory for ducc to only be 50 GB, not all free memory. (We also
> want to limit the CPU usage, for example only use 32 of 64 cores, but we
> have not tried to set this up yet.)
>
> We could not find any setting to achieve this behavior, so we tried using
> cgroups to limit the max usable memory for ducc. This did not work because
> ducc gets its memory info from /proc/meminfo which ignores the cgroups
> settings. After reading through the code it seems only setting
> "ducc.agent.node.metrics.fake.memory.size" (not setting up test mode) is
> doing something similar to what we want: "Comment from
> NodeMemInfoCollector.java: if running ducc in simulation mode skip memory
> adjustment. Report free memory = fakeMemorySize". But I am not sure if we
> can use this safely since it is for testing.
>
> So we basically want to give ducc an upper limit of usable memory.
>
> I hope it is a bit more clear what we want to achieve.
>
> Thanks again,
> Daniel
>
>
> Zitat von Eddie Epstein <ea...@gmail.com>:
>
>
> Hi Daniel,
>>
>> For each node Ducc sums RSS for all "system" user processes and excludes
>> that from Ducc usable memory on the node. System users are defined by a
>> ducc.properties setting with default value:
>> ducc.agent.node.metrics.sys.gid.max = 500
>>
>> Ducc's simulation mode is intended for creating a scaled out cluster of
>> fake nodes for testing purposes.
>>
>> The only mechanism available for reserving additional memory is to have
>> Ducc run some dummy process that stays up forever. This could be a Ducc
>> service that is automatically started when Ducc starts. This could get
>> complicated for a heterogeneous set of machines and/or Ducc classes.
>>
>> Can you be more precise of what features you are looking for limiting
>> resource use of Ducc machines?
>>
>> Thanks,
>> Eddie
>>
>>
>> On Mon, Oct 31, 2016 at 10:03 AM, Daniel Baumartz <
>> baumartz@stud.uni-frankfurt.de> wrote:
>>
>> Hi,
>>>
>>> I am trying to set up nodes for Ducc that should not use all the memory
>>> on
>>> the machine. I tried to limit the memory with cgroups, but it seems Ducc
>>> is
>>> getting the memory info from /proc/meminfo which ignores the cgroups
>>> settings.
>>>
>>> Did I miss an option to specify the max usable memory? Could I safely use
>>> "ducc.agent.node.metrics.fake.memory.size" from the simulation settings?
>>> Or is there a better way to do this?
>>>
>>> Thanks,
>>> Daniel
>>>
>>>
>>>
>
>
>