You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Dohyung Park <ty...@gmail.com> on 2015/09/21 07:34:15 UTC
Is there a limit of the number of tasks that can be launched by Mesos
on a slave?
I am currently using Mesos + Marathon with 4 slaves for the test.
When I launch a lot of tasks with command ping 8.8.8.8 or while true; do
sleep 1; done;,
at one point, slaves cannot launch a task any more. The states of tasks
that are newly launched are 'Failed'.
So I checked out stderr of sandbox, then it shows
"Failed to initialize, pthread_create"
I launched tasks with 0.00001 cpus and 0.00001 mems, so enough resources to
launch a task remained in slaves.
In a slave, ulimit -a shows
*# ulimit -a*
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1545932
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
*open files (-n) 65535*
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
*max user processes (-u) 1545932*
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Is there any limit of the number of tasks that can be launched by Mesos on
a slave?
Re: Is there a limit of the number of tasks that can be launched by
Mesos on a slave?
Posted by Ian Downes <id...@twitter.com>.
`man pthread_create` indicates either insufficient resources (memory) or a
kernel imposed limit from either RLIMIT_NPROC
or /proc/sys/kernel/threads-max.
On my system:
$ cat /proc/sys/kernel/threads-max
191114
which is a lot less than your `ulimit -u`.
How many tasks are you able to launch on a slave? How much memory is
available?
On Mon, Sep 21, 2015 at 4:10 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> FYI, the default executor will claim 32MB of RAM, so you'll either need to
> use a custom executor and specify less RAM, or pretend that your slaves
> have way more memory than they actually do.
>
> On Mon, Sep 21, 2015 at 11:11 AM, Joris Van Remoortere <
> joris@mesosphere.io> wrote:
>
>> > I launched tasks with 0.00001 cpus and 0.00001 mems
>> The executor and task is likely taking more than this amount of memory.
>> Can you check htop to see if you've run out of memory to create more
>> stacks to launch new threads?
>>
>> On Mon, Sep 21, 2015 at 1:34 AM, Dohyung Park <ty...@gmail.com> wrote:
>>
>>> I am currently using Mesos + Marathon with 4 slaves for the test.
>>>
>>> When I launch a lot of tasks with command ping 8.8.8.8 or while true;
>>> do sleep 1; done;,
>>>
>>> at one point, slaves cannot launch a task any more. The states of tasks
>>> that are newly launched are 'Failed'.
>>>
>>> So I checked out stderr of sandbox, then it shows
>>>
>>> "Failed to initialize, pthread_create"
>>>
>>> I launched tasks with 0.00001 cpus and 0.00001 mems, so enough resources
>>> to launch a task remained in slaves.
>>>
>>> In a slave, ulimit -a shows
>>>
>>> *# ulimit -a*
>>>
>>> core file size (blocks, -c) 0
>>> data seg size (kbytes, -d) unlimited
>>> scheduling priority (-e) 0
>>> file size (blocks, -f) unlimited
>>> pending signals (-i) 1545932
>>> max locked memory (kbytes, -l) 64
>>> max memory size (kbytes, -m) unlimited
>>> *open files (-n) 65535*
>>> pipe size (512 bytes, -p) 8
>>> POSIX message queues (bytes, -q) 819200
>>> real-time priority (-r) 0
>>> stack size (kbytes, -s) 8192
>>> cpu time (seconds, -t) unlimited
>>> *max user processes (-u) 1545932*
>>> virtual memory (kbytes, -v) unlimited
>>> file locks (-x) unlimited
>>>
>>> Is there any limit of the number of tasks that can be launched by Mesos
>>> on a slave?
>>>
>>
>>
>
Re: Is there a limit of the number of tasks that can be launched by
Mesos on a slave?
Posted by Adam Bordelon <ad...@mesosphere.io>.
FYI, the default executor will claim 32MB of RAM, so you'll either need to
use a custom executor and specify less RAM, or pretend that your slaves
have way more memory than they actually do.
On Mon, Sep 21, 2015 at 11:11 AM, Joris Van Remoortere <jo...@mesosphere.io>
wrote:
> > I launched tasks with 0.00001 cpus and 0.00001 mems
> The executor and task is likely taking more than this amount of memory.
> Can you check htop to see if you've run out of memory to create more
> stacks to launch new threads?
>
> On Mon, Sep 21, 2015 at 1:34 AM, Dohyung Park <ty...@gmail.com> wrote:
>
>> I am currently using Mesos + Marathon with 4 slaves for the test.
>>
>> When I launch a lot of tasks with command ping 8.8.8.8 or while true; do
>> sleep 1; done;,
>>
>> at one point, slaves cannot launch a task any more. The states of tasks
>> that are newly launched are 'Failed'.
>>
>> So I checked out stderr of sandbox, then it shows
>>
>> "Failed to initialize, pthread_create"
>>
>> I launched tasks with 0.00001 cpus and 0.00001 mems, so enough resources
>> to launch a task remained in slaves.
>>
>> In a slave, ulimit -a shows
>>
>> *# ulimit -a*
>>
>> core file size (blocks, -c) 0
>> data seg size (kbytes, -d) unlimited
>> scheduling priority (-e) 0
>> file size (blocks, -f) unlimited
>> pending signals (-i) 1545932
>> max locked memory (kbytes, -l) 64
>> max memory size (kbytes, -m) unlimited
>> *open files (-n) 65535*
>> pipe size (512 bytes, -p) 8
>> POSIX message queues (bytes, -q) 819200
>> real-time priority (-r) 0
>> stack size (kbytes, -s) 8192
>> cpu time (seconds, -t) unlimited
>> *max user processes (-u) 1545932*
>> virtual memory (kbytes, -v) unlimited
>> file locks (-x) unlimited
>>
>> Is there any limit of the number of tasks that can be launched by Mesos
>> on a slave?
>>
>
>
Re: Is there a limit of the number of tasks that can be launched by
Mesos on a slave?
Posted by Joris Van Remoortere <jo...@mesosphere.io>.
> I launched tasks with 0.00001 cpus and 0.00001 mems
The executor and task is likely taking more than this amount of memory.
Can you check htop to see if you've run out of memory to create more stacks
to launch new threads?
On Mon, Sep 21, 2015 at 1:34 AM, Dohyung Park <ty...@gmail.com> wrote:
> I am currently using Mesos + Marathon with 4 slaves for the test.
>
> When I launch a lot of tasks with command ping 8.8.8.8 or while true; do
> sleep 1; done;,
>
> at one point, slaves cannot launch a task any more. The states of tasks
> that are newly launched are 'Failed'.
>
> So I checked out stderr of sandbox, then it shows
>
> "Failed to initialize, pthread_create"
>
> I launched tasks with 0.00001 cpus and 0.00001 mems, so enough resources
> to launch a task remained in slaves.
>
> In a slave, ulimit -a shows
>
> *# ulimit -a*
>
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 1545932
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> *open files (-n) 65535*
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> *max user processes (-u) 1545932*
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> Is there any limit of the number of tasks that can be launched by Mesos on
> a slave?
>