You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by James Vanns <jv...@gmail.com> on 2017/09/25 12:24:05 UTC
Shedding light on libmesos and libprocess threading models?
Hi guys,
Can anyone shed some light on the threading models/setup used by Mesos
and/or libprocess? I've got a problem with mixing/competing thread pools! I
introduced some OpenMP code and of course now I get N**2 threads started
each time a different libprocess thread executes my OMP code by way of a
mesos scheduler framework callback. Well, that's what I'm guessing!
Anyway, I've come across LIBPROCESS_NUM_WORKER_THREADS and I can set that
to get a known #threads as workers - but my question is (this is now
curiosity more than anything) what are the remainder used for? Eg. If I
have a 4 core machine and indeed the above env var is set to 4 it appears
(without OMP) that libmesos or libprocess still spawn an additional 12
threads. So what are those 12 threads used for?
Oh - this is the (ancient) 0.28.3-2.0.1 release for Ubuntu 14.04 LTS, in
case that matters.
Cheers,
Jim
--
Senior Production Engineer
Industrial Light & Magic (ILM)
Re: Shedding light on libmesos and libprocess threading models?
Posted by James Vanns <jv...@gmail.com>.
Hi,
Sorry this has taken over a week to get back! I'm still getting loads of
threads as before and I've now double checked my Makefile and the output of
ldd. I am only explicitly linking against libmesos and libomp5. I see that
libmesos itself links against a ton of other libraries, which is fine, but
my problem still holds; it seems that in a function that makes use of an
OMP parallel for creates a thread pool of N every time it is called by a
different Mesos worker thread? Eg. I end up with 64 threads instead of 8. I
gather there is little I can do to prevent this? Not in any easy way!? I
guess it'll have to be some global variable and one-time explicit
initialisation of the (OMP) pool rather than a thread-local/stack variable,
which is how a more straight-forward use of OMP would work. Anyway, I guess
this isn't a Mesos problem - just a clash of worker pools! Just wanted some
insight into the (Mesos) thread model to be sure.
Cheers,
Jim
On 26 September 2017 at 11:18, James Vanns <jv...@gmail.com> wrote:
> Oh!? Hmm. I'll take a look - I wasn't explicitly linking against anything
> uncommon other than libomp and libmesos/libprocess. I'll see what ldd says!
>
> Jim
>
>
> On 26 September 2017 at 11:07, Benno Evers <be...@mesosphere.com> wrote:
>
>> Hi Jim,
>>
>> I think how the additional threads are used depends on what else your
>> binary is linked against, for example if I set
>> LIBPROCESS_NUM_WORKER_THREADS=4 on my local mesos-master, I get a process
>> with 7 threads, of which are:
>>
>> - 1 "main" thread
>> - 4 addional libprocess worker threads
>> - 1 background thread spawned by LevelDB
>> - 1 backgrount thread spawned by libev
>>
>> Best regards,
>> Benno
>>
>> On Mon, Sep 25, 2017 at 2:24 PM, James Vanns <jv...@gmail.com>
>> wrote:
>>
>> > Hi guys,
>> >
>> > Can anyone shed some light on the threading models/setup used by Mesos
>> > and/or libprocess? I've got a problem with mixing/competing thread
>> pools! I
>> > introduced some OpenMP code and of course now I get N**2 threads started
>> > each time a different libprocess thread executes my OMP code by way of a
>> > mesos scheduler framework callback. Well, that's what I'm guessing!
>> >
>> > Anyway, I've come across LIBPROCESS_NUM_WORKER_THREADS and I can set
>> that
>> > to get a known #threads as workers - but my question is (this is now
>> > curiosity more than anything) what are the remainder used for? Eg. If I
>> > have a 4 core machine and indeed the above env var is set to 4 it
>> appears
>> > (without OMP) that libmesos or libprocess still spawn an additional 12
>> > threads. So what are those 12 threads used for?
>> >
>> > Oh - this is the (ancient) 0.28.3-2.0.1 release for Ubuntu 14.04 LTS, in
>> > case that matters.
>> >
>> > Cheers,
>> >
>> > Jim
>> >
>> > --
>> > Senior Production Engineer
>> > Industrial Light & Magic (ILM)
>> >
>>
>>
>>
>> --
>> Benno Evers
>> Software Engineer, Mesosphere
>>
>
>
>
> --
> --
> Senior Code Pig
> Industrial Light & Magic
>
--
--
Senior Code Pig
Industrial Light & Magic
Re: Shedding light on libmesos and libprocess threading models?
Posted by James Vanns <jv...@gmail.com>.
Oh!? Hmm. I'll take a look - I wasn't explicitly linking against anything
uncommon other than libomp and libmesos/libprocess. I'll see what ldd says!
Jim
On 26 September 2017 at 11:07, Benno Evers <be...@mesosphere.com> wrote:
> Hi Jim,
>
> I think how the additional threads are used depends on what else your
> binary is linked against, for example if I set
> LIBPROCESS_NUM_WORKER_THREADS=4 on my local mesos-master, I get a process
> with 7 threads, of which are:
>
> - 1 "main" thread
> - 4 addional libprocess worker threads
> - 1 background thread spawned by LevelDB
> - 1 backgrount thread spawned by libev
>
> Best regards,
> Benno
>
> On Mon, Sep 25, 2017 at 2:24 PM, James Vanns <jv...@gmail.com> wrote:
>
> > Hi guys,
> >
> > Can anyone shed some light on the threading models/setup used by Mesos
> > and/or libprocess? I've got a problem with mixing/competing thread
> pools! I
> > introduced some OpenMP code and of course now I get N**2 threads started
> > each time a different libprocess thread executes my OMP code by way of a
> > mesos scheduler framework callback. Well, that's what I'm guessing!
> >
> > Anyway, I've come across LIBPROCESS_NUM_WORKER_THREADS and I can set that
> > to get a known #threads as workers - but my question is (this is now
> > curiosity more than anything) what are the remainder used for? Eg. If I
> > have a 4 core machine and indeed the above env var is set to 4 it appears
> > (without OMP) that libmesos or libprocess still spawn an additional 12
> > threads. So what are those 12 threads used for?
> >
> > Oh - this is the (ancient) 0.28.3-2.0.1 release for Ubuntu 14.04 LTS, in
> > case that matters.
> >
> > Cheers,
> >
> > Jim
> >
> > --
> > Senior Production Engineer
> > Industrial Light & Magic (ILM)
> >
>
>
>
> --
> Benno Evers
> Software Engineer, Mesosphere
>
--
--
Senior Code Pig
Industrial Light & Magic
Re: Shedding light on libmesos and libprocess threading models?
Posted by Benno Evers <be...@mesosphere.com>.
Hi Jim,
I think how the additional threads are used depends on what else your
binary is linked against, for example if I set
LIBPROCESS_NUM_WORKER_THREADS=4 on my local mesos-master, I get a process
with 7 threads, of which are:
- 1 "main" thread
- 4 addional libprocess worker threads
- 1 background thread spawned by LevelDB
- 1 backgrount thread spawned by libev
Best regards,
Benno
On Mon, Sep 25, 2017 at 2:24 PM, James Vanns <jv...@gmail.com> wrote:
> Hi guys,
>
> Can anyone shed some light on the threading models/setup used by Mesos
> and/or libprocess? I've got a problem with mixing/competing thread pools! I
> introduced some OpenMP code and of course now I get N**2 threads started
> each time a different libprocess thread executes my OMP code by way of a
> mesos scheduler framework callback. Well, that's what I'm guessing!
>
> Anyway, I've come across LIBPROCESS_NUM_WORKER_THREADS and I can set that
> to get a known #threads as workers - but my question is (this is now
> curiosity more than anything) what are the remainder used for? Eg. If I
> have a 4 core machine and indeed the above env var is set to 4 it appears
> (without OMP) that libmesos or libprocess still spawn an additional 12
> threads. So what are those 12 threads used for?
>
> Oh - this is the (ancient) 0.28.3-2.0.1 release for Ubuntu 14.04 LTS, in
> case that matters.
>
> Cheers,
>
> Jim
>
> --
> Senior Production Engineer
> Industrial Light & Magic (ILM)
>
--
Benno Evers
Software Engineer, Mesosphere