You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Charles-François Natali <cf...@gmail.com> on 2020/02/20 17:34:18 UTC
Number of tasks per executor and resource limits
Hi,
Is there a way to force Mesos to use one executor per task?
The reason I would want to do that is for resources limits: for example,
when using cgroup to limit CPU and memory, IIUC the containeriser sets
limits corresponding to the sum of the resources allocated to the tasks
managed by the underlying executor.
Which means that for example if a task is allocated 1GB and another 2GB, if
they are started by the same executor, the collarbone containeriser will
limit the total memory for the two tasks (plus the executor) to 3GB. But,
unless I'm missing something, nothing prevents a process (assuming one
process per task with e.g. the command executor or a custom executor) from
using more than its limit.
Obviously it would be possible for the executor to enforce the individual
limits itself using cgroup - does the command/default executor do that? -
but in our case where we use a custom executor it would be quite painful.
Unless I'm missing something?
Thanks in advance for suggestions,
Charles
Re: Number of tasks per executor and resource limits
Posted by Charles-François Natali <cf...@gmail.com>.
Perfect, thanks!
On Fri, 21 Feb 2020, 00:55 Qian Zhang, <zh...@gmail.com> wrote:
> if we wanted to support multiple tasks per executor, could we leverage the
>> mesos containeriser to easily put each task in its own cgroup?
>
>
> If what your custom executor does is similar with default executor (i.e.,
> launch each task in a task group as a nested container by calling Mesos
> agent API `LAUNCH_CONTAINER`), then the answer is yes since we are going to
> support nested cgroups for nested containers (i.e., each nested container
> can have its own cgroups) in Mesos containerizer soon.
>
>
> Regards,
> Qian Zhang
>
>
> On Fri, Feb 21, 2020 at 4:46 AM Vinod Kone <vi...@apache.org> wrote:
>
>> Andrei, Qian: Can one of you answer the above question?
>>
>> On Thu, Feb 20, 2020 at 7:15 PM Charles-François Natali <
>> cf.natali@gmail.com> wrote:
>>
>>> Thanks for the quick reply!
>>>
>>> I think we're going to go for one executor per task for now, that's much
>>> simpler.
>>>
>>> Otherwise I was wondering - if we wanted to support multiple tasks per
>>> executor, could we leverage the mesos containeriser to easily put each task
>>> in its own cgroup?
>>>
>>>
>>>
>>> Is it just a matter of setting 'execu
>>> On Thu, 20 Feb 2020, 17:40 Vinod Kone, <vi...@apache.org> wrote:
>>>
>>>> Hi Charles,
>>>>
>>>> We are actually working on a new feature that puts each of the tasks
>>>> (of the default executor) in its own cgroup so that they can be
>>>> individually limited. See
>>>> https://issues.apache.org/jira/browse/MESOS-9916 . For custom
>>>> executor, you would be on your own to implement the same. Also, your custom
>>>> executor / scheduler should be able to limit one task per executor if you
>>>> so desire.
>>>>
>>>> On Thu, Feb 20, 2020 at 6:34 PM Charles-François Natali <
>>>> cf.natali@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Is there a way to force Mesos to use one executor per task?
>>>>> The reason I would want to do that is for resources limits: for
>>>>> example, when using cgroup to limit CPU and memory, IIUC the containeriser
>>>>> sets limits corresponding to the sum of the resources allocated to the
>>>>> tasks managed by the underlying executor.
>>>>>
>>>>> Which means that for example if a task is allocated 1GB and another
>>>>> 2GB, if they are started by the same executor, the collarbone containeriser
>>>>> will limit the total memory for the two tasks (plus the executor) to 3GB.
>>>>> But, unless I'm missing something, nothing prevents a process (assuming one
>>>>> process per task with e.g. the command executor or a custom executor) from
>>>>> using more than its limit.
>>>>>
>>>>> Obviously it would be possible for the executor to enforce the
>>>>> individual limits itself using cgroup - does the command/default executor
>>>>> do that? - but in our case where we use a custom executor it would be quite
>>>>> painful.
>>>>>
>>>>> Unless I'm missing something?
>>>>>
>>>>> Thanks in advance for suggestions,
>>>>>
>>>>> Charles
>>>>>
>>>>
Re: Number of tasks per executor and resource limits
Posted by Qian Zhang <zh...@gmail.com>.
>
> if we wanted to support multiple tasks per executor, could we leverage the
> mesos containeriser to easily put each task in its own cgroup?
If what your custom executor does is similar with default executor (i.e.,
launch each task in a task group as a nested container by calling Mesos
agent API `LAUNCH_CONTAINER`), then the answer is yes since we are going to
support nested cgroups for nested containers (i.e., each nested container
can have its own cgroups) in Mesos containerizer soon.
Regards,
Qian Zhang
On Fri, Feb 21, 2020 at 4:46 AM Vinod Kone <vi...@apache.org> wrote:
> Andrei, Qian: Can one of you answer the above question?
>
> On Thu, Feb 20, 2020 at 7:15 PM Charles-François Natali <
> cf.natali@gmail.com> wrote:
>
>> Thanks for the quick reply!
>>
>> I think we're going to go for one executor per task for now, that's much
>> simpler.
>>
>> Otherwise I was wondering - if we wanted to support multiple tasks per
>> executor, could we leverage the mesos containeriser to easily put each task
>> in its own cgroup?
>>
>>
>>
>> Is it just a matter of setting 'execu
>> On Thu, 20 Feb 2020, 17:40 Vinod Kone, <vi...@apache.org> wrote:
>>
>>> Hi Charles,
>>>
>>> We are actually working on a new feature that puts each of the tasks (of
>>> the default executor) in its own cgroup so that they can be individually
>>> limited. See https://issues.apache.org/jira/browse/MESOS-9916 . For
>>> custom executor, you would be on your own to implement the same. Also, your
>>> custom executor / scheduler should be able to limit one task per executor
>>> if you so desire.
>>>
>>> On Thu, Feb 20, 2020 at 6:34 PM Charles-François Natali <
>>> cf.natali@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Is there a way to force Mesos to use one executor per task?
>>>> The reason I would want to do that is for resources limits: for
>>>> example, when using cgroup to limit CPU and memory, IIUC the containeriser
>>>> sets limits corresponding to the sum of the resources allocated to the
>>>> tasks managed by the underlying executor.
>>>>
>>>> Which means that for example if a task is allocated 1GB and another
>>>> 2GB, if they are started by the same executor, the collarbone containeriser
>>>> will limit the total memory for the two tasks (plus the executor) to 3GB.
>>>> But, unless I'm missing something, nothing prevents a process (assuming one
>>>> process per task with e.g. the command executor or a custom executor) from
>>>> using more than its limit.
>>>>
>>>> Obviously it would be possible for the executor to enforce the
>>>> individual limits itself using cgroup - does the command/default executor
>>>> do that? - but in our case where we use a custom executor it would be quite
>>>> painful.
>>>>
>>>> Unless I'm missing something?
>>>>
>>>> Thanks in advance for suggestions,
>>>>
>>>> Charles
>>>>
>>>
Re: Number of tasks per executor and resource limits
Posted by Vinod Kone <vi...@apache.org>.
Andrei, Qian: Can one of you answer the above question?
On Thu, Feb 20, 2020 at 7:15 PM Charles-François Natali <cf...@gmail.com>
wrote:
> Thanks for the quick reply!
>
> I think we're going to go for one executor per task for now, that's much
> simpler.
>
> Otherwise I was wondering - if we wanted to support multiple tasks per
> executor, could we leverage the mesos containeriser to easily put each task
> in its own cgroup?
>
>
>
> Is it just a matter of setting 'execu
> On Thu, 20 Feb 2020, 17:40 Vinod Kone, <vi...@apache.org> wrote:
>
>> Hi Charles,
>>
>> We are actually working on a new feature that puts each of the tasks (of
>> the default executor) in its own cgroup so that they can be individually
>> limited. See https://issues.apache.org/jira/browse/MESOS-9916 . For
>> custom executor, you would be on your own to implement the same. Also, your
>> custom executor / scheduler should be able to limit one task per executor
>> if you so desire.
>>
>> On Thu, Feb 20, 2020 at 6:34 PM Charles-François Natali <
>> cf.natali@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Is there a way to force Mesos to use one executor per task?
>>> The reason I would want to do that is for resources limits: for example,
>>> when using cgroup to limit CPU and memory, IIUC the containeriser sets
>>> limits corresponding to the sum of the resources allocated to the tasks
>>> managed by the underlying executor.
>>>
>>> Which means that for example if a task is allocated 1GB and another 2GB,
>>> if they are started by the same executor, the collarbone containeriser will
>>> limit the total memory for the two tasks (plus the executor) to 3GB. But,
>>> unless I'm missing something, nothing prevents a process (assuming one
>>> process per task with e.g. the command executor or a custom executor) from
>>> using more than its limit.
>>>
>>> Obviously it would be possible for the executor to enforce the
>>> individual limits itself using cgroup - does the command/default executor
>>> do that? - but in our case where we use a custom executor it would be quite
>>> painful.
>>>
>>> Unless I'm missing something?
>>>
>>> Thanks in advance for suggestions,
>>>
>>> Charles
>>>
>>
Re: Number of tasks per executor and resource limits
Posted by Charles-François Natali <cf...@gmail.com>.
Thanks for the quick reply!
I think we're going to go for one executor per task for now, that's much
simpler.
Otherwise I was wondering - if we wanted to support multiple tasks per
executor, could we leverage the mesos containeriser to easily put each task
in its own cgroup?
Is it just a matter of setting 'execu
On Thu, 20 Feb 2020, 17:40 Vinod Kone, <vi...@apache.org> wrote:
> Hi Charles,
>
> We are actually working on a new feature that puts each of the tasks (of
> the default executor) in its own cgroup so that they can be individually
> limited. See https://issues.apache.org/jira/browse/MESOS-9916 . For
> custom executor, you would be on your own to implement the same. Also, your
> custom executor / scheduler should be able to limit one task per executor
> if you so desire.
>
> On Thu, Feb 20, 2020 at 6:34 PM Charles-François Natali <
> cf.natali@gmail.com> wrote:
>
>> Hi,
>>
>> Is there a way to force Mesos to use one executor per task?
>> The reason I would want to do that is for resources limits: for example,
>> when using cgroup to limit CPU and memory, IIUC the containeriser sets
>> limits corresponding to the sum of the resources allocated to the tasks
>> managed by the underlying executor.
>>
>> Which means that for example if a task is allocated 1GB and another 2GB,
>> if they are started by the same executor, the collarbone containeriser will
>> limit the total memory for the two tasks (plus the executor) to 3GB. But,
>> unless I'm missing something, nothing prevents a process (assuming one
>> process per task with e.g. the command executor or a custom executor) from
>> using more than its limit.
>>
>> Obviously it would be possible for the executor to enforce the individual
>> limits itself using cgroup - does the command/default executor do that? -
>> but in our case where we use a custom executor it would be quite painful.
>>
>> Unless I'm missing something?
>>
>> Thanks in advance for suggestions,
>>
>> Charles
>>
>
Re: Number of tasks per executor and resource limits
Posted by Vinod Kone <vi...@apache.org>.
Hi Charles,
We are actually working on a new feature that puts each of the tasks (of
the default executor) in its own cgroup so that they can be individually
limited. See https://issues.apache.org/jira/browse/MESOS-9916 . For custom
executor, you would be on your own to implement the same. Also, your custom
executor / scheduler should be able to limit one task per executor if you
so desire.
On Thu, Feb 20, 2020 at 6:34 PM Charles-François Natali <cf...@gmail.com>
wrote:
> Hi,
>
> Is there a way to force Mesos to use one executor per task?
> The reason I would want to do that is for resources limits: for example,
> when using cgroup to limit CPU and memory, IIUC the containeriser sets
> limits corresponding to the sum of the resources allocated to the tasks
> managed by the underlying executor.
>
> Which means that for example if a task is allocated 1GB and another 2GB,
> if they are started by the same executor, the collarbone containeriser will
> limit the total memory for the two tasks (plus the executor) to 3GB. But,
> unless I'm missing something, nothing prevents a process (assuming one
> process per task with e.g. the command executor or a custom executor) from
> using more than its limit.
>
> Obviously it would be possible for the executor to enforce the individual
> limits itself using cgroup - does the command/default executor do that? -
> but in our case where we use a custom executor it would be quite painful.
>
> Unless I'm missing something?
>
> Thanks in advance for suggestions,
>
> Charles
>