You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Ankan Mukherjee <an...@gmail.com> on 2015/12/11 19:44:25 UTC
Selectively enable/disable oom killer on tasks
Is there a way to selectively enable/disable oom killer on mesos tasks?
Re: Selectively enable/disable oom killer on tasks
Posted by Ian Downes <id...@twitter.com>.
And, you *may* want to consider enabling swap to get decidedly ungraceful
performance degradation rather than OOM killing.
On Fri, Dec 11, 2015 at 1:02 PM, Brenden Matthews <br...@diddyinc.com>
wrote:
> You may also want to consider giving your tasks more memory.
>
> On Fri, Dec 11, 2015 at 1:00 PM, Ankan Mukherjee <an...@gmail.com> wrote:
>
>> Thanks for clearing that up. I realize that going beyond allocated cpu
>> will get the container throttled (only.)
>>
>> We use both cgroups/cpu and cgroups/mem isolators. However, I'm looking
>> at avoiding OOM kills on some specific tasks (so this is really about mem.)
>>
>> I'm going try switching a couple of agents to not use cgroups/mem
>> isolator, and get those tasks to run on these machines somehow. I'm
>> thinking a text attribute or resource set on these agents will be one way
>> to go.
>>
>> Thanks again for your prompt response.
>>
>> On Fri, Dec 11, 2015 at 12:04 PM, Ian Downes <id...@twitter.com> wrote:
>>
>>> The OOM killer is for memory isolation. The kernel will kill processes
>>> when the container tries to allocate memory above its limit.
>>>
>>> "cgroups_enable_cfs" is a flag for cpu isolation and enables CPU
>>> bandwidth control which will throttle (but not kill) a container when it
>>> reaches its cpu limit. These are two different isolators which can each be
>>> enabled independently.
>>>
>>> Which are you wanting to use? Either way, my statement of "no" is true
>>> at the Mesos agent level. The agent does not support different container
>>> isolation configurations for the MesosContainerizer. But, you are correct
>>> that different Mesos agents in a cluster could have different
>>> configurations with some not enabling the cgroups/cpu or cgroups/mem
>>> isolators.
>>>
>>> On Fri, Dec 11, 2015 at 11:31 AM, Ankan Mukherjee <an...@gmail.com>
>>> wrote:
>>>
>>>> Thanks for your response.
>>>>
>>>> Yes, that's what we're using. I was looking to have instances of a
>>>> particular application running without the OOM killer. Looks like there's a request
>>>> for this <https://issues.apache.org/jira/browse/MESOS-2270> already.
>>>>
>>>> I believe it would still be possible to have a mix of mesos slaves with
>>>> "cgroups_enable_cfs" true and false. Is there a downside to doing this?
>>>>
>>>> On Fri, Dec 11, 2015 at 11:10 AM, Ian Downes <id...@twitter.com>
>>>> wrote:
>>>>
>>>>> No. Assuming you're using the MesosContainerizer and the cgroups/mem
>>>>> isolator then all containers will have the kernel OOM killer enabled.
>>>>>
>>>>> What behavior are you looking to achieve?
>>>>>
>>>>> On Fri, Dec 11, 2015 at 10:44 AM, Ankan Mukherjee <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Is there a way to selectively enable/disable oom killer on mesos
>>>>>> tasks?
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Selectively enable/disable oom killer on tasks
Posted by Brenden Matthews <br...@diddyinc.com>.
You may also want to consider giving your tasks more memory.
On Fri, Dec 11, 2015 at 1:00 PM, Ankan Mukherjee <an...@gmail.com> wrote:
> Thanks for clearing that up. I realize that going beyond allocated cpu
> will get the container throttled (only.)
>
> We use both cgroups/cpu and cgroups/mem isolators. However, I'm looking at
> avoiding OOM kills on some specific tasks (so this is really about mem.)
>
> I'm going try switching a couple of agents to not use cgroups/mem
> isolator, and get those tasks to run on these machines somehow. I'm
> thinking a text attribute or resource set on these agents will be one way
> to go.
>
> Thanks again for your prompt response.
>
> On Fri, Dec 11, 2015 at 12:04 PM, Ian Downes <id...@twitter.com> wrote:
>
>> The OOM killer is for memory isolation. The kernel will kill processes
>> when the container tries to allocate memory above its limit.
>>
>> "cgroups_enable_cfs" is a flag for cpu isolation and enables CPU
>> bandwidth control which will throttle (but not kill) a container when it
>> reaches its cpu limit. These are two different isolators which can each be
>> enabled independently.
>>
>> Which are you wanting to use? Either way, my statement of "no" is true at
>> the Mesos agent level. The agent does not support different container
>> isolation configurations for the MesosContainerizer. But, you are correct
>> that different Mesos agents in a cluster could have different
>> configurations with some not enabling the cgroups/cpu or cgroups/mem
>> isolators.
>>
>> On Fri, Dec 11, 2015 at 11:31 AM, Ankan Mukherjee <an...@gmail.com>
>> wrote:
>>
>>> Thanks for your response.
>>>
>>> Yes, that's what we're using. I was looking to have instances of a
>>> particular application running without the OOM killer. Looks like there's a request
>>> for this <https://issues.apache.org/jira/browse/MESOS-2270> already.
>>>
>>> I believe it would still be possible to have a mix of mesos slaves with
>>> "cgroups_enable_cfs" true and false. Is there a downside to doing this?
>>>
>>> On Fri, Dec 11, 2015 at 11:10 AM, Ian Downes <id...@twitter.com>
>>> wrote:
>>>
>>>> No. Assuming you're using the MesosContainerizer and the cgroups/mem
>>>> isolator then all containers will have the kernel OOM killer enabled.
>>>>
>>>> What behavior are you looking to achieve?
>>>>
>>>> On Fri, Dec 11, 2015 at 10:44 AM, Ankan Mukherjee <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> Is there a way to selectively enable/disable oom killer on mesos tasks?
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Selectively enable/disable oom killer on tasks
Posted by Ankan Mukherjee <an...@gmail.com>.
Thanks for clearing that up. I realize that going beyond allocated cpu will
get the container throttled (only.)
We use both cgroups/cpu and cgroups/mem isolators. However, I'm looking at
avoiding OOM kills on some specific tasks (so this is really about mem.)
I'm going try switching a couple of agents to not use cgroups/mem isolator,
and get those tasks to run on these machines somehow. I'm thinking a text
attribute or resource set on these agents will be one way to go.
Thanks again for your prompt response.
On Fri, Dec 11, 2015 at 12:04 PM, Ian Downes <id...@twitter.com> wrote:
> The OOM killer is for memory isolation. The kernel will kill processes
> when the container tries to allocate memory above its limit.
>
> "cgroups_enable_cfs" is a flag for cpu isolation and enables CPU
> bandwidth control which will throttle (but not kill) a container when it
> reaches its cpu limit. These are two different isolators which can each be
> enabled independently.
>
> Which are you wanting to use? Either way, my statement of "no" is true at
> the Mesos agent level. The agent does not support different container
> isolation configurations for the MesosContainerizer. But, you are correct
> that different Mesos agents in a cluster could have different
> configurations with some not enabling the cgroups/cpu or cgroups/mem
> isolators.
>
> On Fri, Dec 11, 2015 at 11:31 AM, Ankan Mukherjee <an...@gmail.com>
> wrote:
>
>> Thanks for your response.
>>
>> Yes, that's what we're using. I was looking to have instances of a
>> particular application running without the OOM killer. Looks like there's a request
>> for this <https://issues.apache.org/jira/browse/MESOS-2270> already.
>>
>> I believe it would still be possible to have a mix of mesos slaves with
>> "cgroups_enable_cfs" true and false. Is there a downside to doing this?
>>
>> On Fri, Dec 11, 2015 at 11:10 AM, Ian Downes <id...@twitter.com> wrote:
>>
>>> No. Assuming you're using the MesosContainerizer and the cgroups/mem
>>> isolator then all containers will have the kernel OOM killer enabled.
>>>
>>> What behavior are you looking to achieve?
>>>
>>> On Fri, Dec 11, 2015 at 10:44 AM, Ankan Mukherjee <an...@gmail.com>
>>> wrote:
>>>
>>>> Is there a way to selectively enable/disable oom killer on mesos tasks?
>>>>
>>>
>>>
>>
>
Re: Selectively enable/disable oom killer on tasks
Posted by Ian Downes <id...@twitter.com>.
The OOM killer is for memory isolation. The kernel will kill processes when
the container tries to allocate memory above its limit.
"cgroups_enable_cfs" is a flag for cpu isolation and enables CPU bandwidth
control which will throttle (but not kill) a container when it reaches its
cpu limit. These are two different isolators which can each be enabled
independently.
Which are you wanting to use? Either way, my statement of "no" is true at
the Mesos agent level. The agent does not support different container
isolation configurations for the MesosContainerizer. But, you are correct
that different Mesos agents in a cluster could have different
configurations with some not enabling the cgroups/cpu or cgroups/mem
isolators.
On Fri, Dec 11, 2015 at 11:31 AM, Ankan Mukherjee <an...@gmail.com> wrote:
> Thanks for your response.
>
> Yes, that's what we're using. I was looking to have instances of a
> particular application running without the OOM killer. Looks like there's a request
> for this <https://issues.apache.org/jira/browse/MESOS-2270> already.
>
> I believe it would still be possible to have a mix of mesos slaves with
> "cgroups_enable_cfs" true and false. Is there a downside to doing this?
>
> On Fri, Dec 11, 2015 at 11:10 AM, Ian Downes <id...@twitter.com> wrote:
>
>> No. Assuming you're using the MesosContainerizer and the cgroups/mem
>> isolator then all containers will have the kernel OOM killer enabled.
>>
>> What behavior are you looking to achieve?
>>
>> On Fri, Dec 11, 2015 at 10:44 AM, Ankan Mukherjee <an...@gmail.com>
>> wrote:
>>
>>> Is there a way to selectively enable/disable oom killer on mesos tasks?
>>>
>>
>>
>
Re: Selectively enable/disable oom killer on tasks
Posted by Ankan Mukherjee <an...@gmail.com>.
Thanks for your response.
Yes, that's what we're using. I was looking to have instances of a
particular application running without the OOM killer. Looks like
there's a request
for this <https://issues.apache.org/jira/browse/MESOS-2270> already.
I believe it would still be possible to have a mix of mesos slaves with
"cgroups_enable_cfs" true and false. Is there a downside to doing this?
On Fri, Dec 11, 2015 at 11:10 AM, Ian Downes <id...@twitter.com> wrote:
> No. Assuming you're using the MesosContainerizer and the cgroups/mem
> isolator then all containers will have the kernel OOM killer enabled.
>
> What behavior are you looking to achieve?
>
> On Fri, Dec 11, 2015 at 10:44 AM, Ankan Mukherjee <an...@gmail.com>
> wrote:
>
>> Is there a way to selectively enable/disable oom killer on mesos tasks?
>>
>
>
Re: Selectively enable/disable oom killer on tasks
Posted by Ian Downes <id...@twitter.com>.
No. Assuming you're using the MesosContainerizer and the cgroups/mem
isolator then all containers will have the kernel OOM killer enabled.
What behavior are you looking to achieve?
On Fri, Dec 11, 2015 at 10:44 AM, Ankan Mukherjee <an...@gmail.com> wrote:
> Is there a way to selectively enable/disable oom killer on mesos tasks?
>