You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by deadbrain <je...@gmail.com> on 2017/11/07 13:28:35 UTC
Mesos , Java & hardware isolation
Hi all,
I expect to be at the right place for my question...
We are hosting Java micro services exposed as docker containers (Spring
Boot applications inside) inside Mesos....
We face 2 major problems:
- hardware isolation (number of cores seen by Java Virtual Machines)
makes our applications go crazy because JVM sees all cores and not the
only ones given in the Mesos config parameters as a consequence GC
heuristics are wrong and some config values in some libraries are false
too (Undertow web server spawns 8*number of cores as workers thread pool)...
In Linux since the 2.4.19 kernel we have the per process namespace
option to limit the hardware vision of any container... What would be
the solution on Mesos ?
- CPU starvation : some micro services do not seem to have the required
CPU time to work, what kind of advices could you give me to have a
better understanding regarding this problem....
Statement: we can see very long partial GC times (about 4 seconds) ,
standard times would be about 50ms....
Thanks for your help
Kind regards
Jerome
Re: Mesos , Java & hardware isolation
Posted by deadbrain <je...@gmail.com>.
Thanks for this detail Justin I just checked on mercurial and you are
right , we used a jdk 8u151 and I have seen a couple of cgroups fixes in
the u152
Thanks again
Regards
On 11/07/2017 05:06 PM, Justin Lee wrote:
> It may also depend on the version of Java you're using - depending on
> the version of Java that you're using, they may or may not properly
> pick up the cgroup settings vs. the total number of cores available.
>
> I don't know which version(s) of Java work, but I think the later
> versions have better support.
>
> Thanks,
> Justin Lee
>
> On Tue, Nov 7, 2017 at 10:18 AM, deadbrain <jerome.moliere@gmail.com
> <ma...@gmail.com>> wrote:
>
> Thanks Thodoris for you help I will need to check with people
> managing the infrastructure to see what parameters are passed to
> our agents...
>
> Thanks again for the tips
>
> Regards
>
>
>
> On 11/07/2017 04:10 PM, Thodoris Zois wrote:
>
> Hello,
>
> I don't really know if i can help you but i used Docker
> containers in Mesos with isolation and everything was perfect.
> So are you using Mesos containerizer? Did you enable
> cgroups/mem, cgroups/cpu on each agent? As it concerns CPU,
> cgroups do not limit the anount of CPU if its the only running
> application. In order to limit exactly CPU you need to enable
> cfs on each agent.
>
> Thodoris
>
> On 7 Nov 2017, at 15:28, deadbrain
> <jerome.moliere@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi all,
>
> I expect to be at the right place for my question...
>
> We are hosting Java micro services exposed as docker
> containers (Spring Boot applications inside) inside Mesos....
>
> We face 2 major problems:
>
> - hardware isolation (number of cores seen by Java Virtual
> Machines) makes our applications go crazy because JVM sees
> all cores and not the only ones given in the Mesos config
> parameters as a consequence GC heuristics are wrong and
> some config values in some libraries are false too
> (Undertow web server spawns 8*number of cores as workers
> thread pool)...
>
> In Linux since the 2.4.19 kernel we have the per process
> namespace option to limit the hardware vision of any
> container... What would be the solution on Mesos ?
>
>
> - CPU starvation : some micro services do not seem to have
> the required CPU time to work, what kind of advices could
> you give me to have a better understanding regarding this
> problem....
>
> Statement: we can see very long partial GC times (about 4
> seconds) , standard times would be about 50ms....
>
>
> Thanks for your help
>
> Kind regards
>
> Jerome
>
>
>
Re: Mesos , Java & hardware isolation
Posted by deadbrain <je...@gmail.com>.
Very interesting tip Tomek....
Regards
On 11/07/2017 08:25 PM, Tomek Janiszewski wrote:
>
> @Justin Take a look at this answer
> https://stackoverflow.com/a/22762558/1387612 we are using similar
> approach to change CPU/MEM seen by JVM running in a container.
>
>
> wt., 7.11.2017, 17:06 użytkownik Justin Lee <jlee@mesosphere.io
> <ma...@mesosphere.io>> napisał:
>
> It may also depend on the version of Java you're using - depending
> on the version of Java that you're using, they may or may not
> properly pick up the cgroup settings vs. the total number of cores
> available.
>
> I don't know which version(s) of Java work, but I think the later
> versions have better support.
>
> Thanks,
> Justin Lee
>
> On Tue, Nov 7, 2017 at 10:18 AM, deadbrain
> <jerome.moliere@gmail.com <ma...@gmail.com>> wrote:
>
> Thanks Thodoris for you help I will need to check with people
> managing the infrastructure to see what parameters are passed
> to our agents...
>
> Thanks again for the tips
>
> Regards
>
>
>
> On 11/07/2017 04:10 PM, Thodoris Zois wrote:
>
> Hello,
>
> I don't really know if i can help you but i used Docker
> containers in Mesos with isolation and everything was
> perfect. So are you using Mesos containerizer? Did you
> enable cgroups/mem, cgroups/cpu on each agent? As it
> concerns CPU, cgroups do not limit the anount of CPU if
> its the only running application. In order to limit
> exactly CPU you need to enable cfs on each agent.
>
> Thodoris
>
> On 7 Nov 2017, at 15:28, deadbrain
> <jerome.moliere@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi all,
>
> I expect to be at the right place for my question...
>
> We are hosting Java micro services exposed as docker
> containers (Spring Boot applications inside) inside
> Mesos....
>
> We face 2 major problems:
>
> - hardware isolation (number of cores seen by Java
> Virtual Machines) makes our applications go crazy
> because JVM sees all cores and not the only ones given
> in the Mesos config parameters as a consequence GC
> heuristics are wrong and some config values in some
> libraries are false too (Undertow web server spawns
> 8*number of cores as workers thread pool)...
>
> In Linux since the 2.4.19 kernel we have the per
> process namespace option to limit the hardware vision
> of any container... What would be the solution on Mesos ?
>
>
> - CPU starvation : some micro services do not seem to
> have the required CPU time to work, what kind of
> advices could you give me to have a better
> understanding regarding this problem....
>
> Statement: we can see very long partial GC times
> (about 4 seconds) , standard times would be about 50ms....
>
>
> Thanks for your help
>
> Kind regards
>
> Jerome
>
>
>
Re: Mesos , Java & hardware isolation
Posted by Tomek Janiszewski <ja...@gmail.com>.
@Justin Take a look at this answer
https://stackoverflow.com/a/22762558/1387612 we are using similar approach
to change CPU/MEM seen by JVM running in a container.
wt., 7.11.2017, 17:06 użytkownik Justin Lee <jl...@mesosphere.io> napisał:
> It may also depend on the version of Java you're using - depending on the
> version of Java that you're using, they may or may not properly pick up the
> cgroup settings vs. the total number of cores available.
>
> I don't know which version(s) of Java work, but I think the later versions
> have better support.
>
> Thanks,
> Justin Lee
>
> On Tue, Nov 7, 2017 at 10:18 AM, deadbrain <je...@gmail.com>
> wrote:
>
>> Thanks Thodoris for you help I will need to check with people managing
>> the infrastructure to see what parameters are passed to our agents...
>>
>> Thanks again for the tips
>>
>> Regards
>>
>>
>>
>> On 11/07/2017 04:10 PM, Thodoris Zois wrote:
>>
>>> Hello,
>>>
>>> I don't really know if i can help you but i used Docker containers in
>>> Mesos with isolation and everything was perfect. So are you using Mesos
>>> containerizer? Did you enable cgroups/mem, cgroups/cpu on each agent? As it
>>> concerns CPU, cgroups do not limit the anount of CPU if its the only
>>> running application. In order to limit exactly CPU you need to enable cfs
>>> on each agent.
>>>
>>> Thodoris
>>>
>>> On 7 Nov 2017, at 15:28, deadbrain <je...@gmail.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I expect to be at the right place for my question...
>>>>
>>>> We are hosting Java micro services exposed as docker containers (Spring
>>>> Boot applications inside) inside Mesos....
>>>>
>>>> We face 2 major problems:
>>>>
>>>> - hardware isolation (number of cores seen by Java Virtual Machines)
>>>> makes our applications go crazy because JVM sees all cores and not the only
>>>> ones given in the Mesos config parameters as a consequence GC heuristics
>>>> are wrong and some config values in some libraries are false too (Undertow
>>>> web server spawns 8*number of cores as workers thread pool)...
>>>>
>>>> In Linux since the 2.4.19 kernel we have the per process namespace
>>>> option to limit the hardware vision of any container... What would be the
>>>> solution on Mesos ?
>>>>
>>>>
>>>> - CPU starvation : some micro services do not seem to have the required
>>>> CPU time to work, what kind of advices could you give me to have a better
>>>> understanding regarding this problem....
>>>>
>>>> Statement: we can see very long partial GC times (about 4 seconds) ,
>>>> standard times would be about 50ms....
>>>>
>>>>
>>>> Thanks for your help
>>>>
>>>> Kind regards
>>>>
>>>> Jerome
>>>>
>>>>
>>
>
Re: Mesos , Java & hardware isolation
Posted by Justin Lee <jl...@mesosphere.io>.
It may also depend on the version of Java you're using - depending on the
version of Java that you're using, they may or may not properly pick up the
cgroup settings vs. the total number of cores available.
I don't know which version(s) of Java work, but I think the later versions
have better support.
Thanks,
Justin Lee
On Tue, Nov 7, 2017 at 10:18 AM, deadbrain <je...@gmail.com> wrote:
> Thanks Thodoris for you help I will need to check with people managing the
> infrastructure to see what parameters are passed to our agents...
>
> Thanks again for the tips
>
> Regards
>
>
>
> On 11/07/2017 04:10 PM, Thodoris Zois wrote:
>
>> Hello,
>>
>> I don't really know if i can help you but i used Docker containers in
>> Mesos with isolation and everything was perfect. So are you using Mesos
>> containerizer? Did you enable cgroups/mem, cgroups/cpu on each agent? As it
>> concerns CPU, cgroups do not limit the anount of CPU if its the only
>> running application. In order to limit exactly CPU you need to enable cfs
>> on each agent.
>>
>> Thodoris
>>
>> On 7 Nov 2017, at 15:28, deadbrain <je...@gmail.com> wrote:
>>>
>>> Hi all,
>>>
>>> I expect to be at the right place for my question...
>>>
>>> We are hosting Java micro services exposed as docker containers (Spring
>>> Boot applications inside) inside Mesos....
>>>
>>> We face 2 major problems:
>>>
>>> - hardware isolation (number of cores seen by Java Virtual Machines)
>>> makes our applications go crazy because JVM sees all cores and not the only
>>> ones given in the Mesos config parameters as a consequence GC heuristics
>>> are wrong and some config values in some libraries are false too (Undertow
>>> web server spawns 8*number of cores as workers thread pool)...
>>>
>>> In Linux since the 2.4.19 kernel we have the per process namespace
>>> option to limit the hardware vision of any container... What would be the
>>> solution on Mesos ?
>>>
>>>
>>> - CPU starvation : some micro services do not seem to have the required
>>> CPU time to work, what kind of advices could you give me to have a better
>>> understanding regarding this problem....
>>>
>>> Statement: we can see very long partial GC times (about 4 seconds) ,
>>> standard times would be about 50ms....
>>>
>>>
>>> Thanks for your help
>>>
>>> Kind regards
>>>
>>> Jerome
>>>
>>>
>
Re: Mesos , Java & hardware isolation
Posted by deadbrain <je...@gmail.com>.
Thanks Thodoris for you help I will need to check with people managing
the infrastructure to see what parameters are passed to our agents...
Thanks again for the tips
Regards
On 11/07/2017 04:10 PM, Thodoris Zois wrote:
> Hello,
>
> I don't really know if i can help you but i used Docker containers in Mesos with isolation and everything was perfect. So are you using Mesos containerizer? Did you enable cgroups/mem, cgroups/cpu on each agent? As it concerns CPU, cgroups do not limit the anount of CPU if its the only running application. In order to limit exactly CPU you need to enable cfs on each agent.
>
> Thodoris
>
>> On 7 Nov 2017, at 15:28, deadbrain <je...@gmail.com> wrote:
>>
>> Hi all,
>>
>> I expect to be at the right place for my question...
>>
>> We are hosting Java micro services exposed as docker containers (Spring Boot applications inside) inside Mesos....
>>
>> We face 2 major problems:
>>
>> - hardware isolation (number of cores seen by Java Virtual Machines) makes our applications go crazy because JVM sees all cores and not the only ones given in the Mesos config parameters as a consequence GC heuristics are wrong and some config values in some libraries are false too (Undertow web server spawns 8*number of cores as workers thread pool)...
>>
>> In Linux since the 2.4.19 kernel we have the per process namespace option to limit the hardware vision of any container... What would be the solution on Mesos ?
>>
>>
>> - CPU starvation : some micro services do not seem to have the required CPU time to work, what kind of advices could you give me to have a better understanding regarding this problem....
>>
>> Statement: we can see very long partial GC times (about 4 seconds) , standard times would be about 50ms....
>>
>>
>> Thanks for your help
>>
>> Kind regards
>>
>> Jerome
>>
Re: Mesos , Java & hardware isolation
Posted by Thodoris Zois <zo...@ics.forth.gr>.
Hello,
I don't really know if i can help you but i used Docker containers in Mesos with isolation and everything was perfect. So are you using Mesos containerizer? Did you enable cgroups/mem, cgroups/cpu on each agent? As it concerns CPU, cgroups do not limit the anount of CPU if its the only running application. In order to limit exactly CPU you need to enable cfs on each agent.
Thodoris
> On 7 Nov 2017, at 15:28, deadbrain <je...@gmail.com> wrote:
>
> Hi all,
>
> I expect to be at the right place for my question...
>
> We are hosting Java micro services exposed as docker containers (Spring Boot applications inside) inside Mesos....
>
> We face 2 major problems:
>
> - hardware isolation (number of cores seen by Java Virtual Machines) makes our applications go crazy because JVM sees all cores and not the only ones given in the Mesos config parameters as a consequence GC heuristics are wrong and some config values in some libraries are false too (Undertow web server spawns 8*number of cores as workers thread pool)...
>
> In Linux since the 2.4.19 kernel we have the per process namespace option to limit the hardware vision of any container... What would be the solution on Mesos ?
>
>
> - CPU starvation : some micro services do not seem to have the required CPU time to work, what kind of advices could you give me to have a better understanding regarding this problem....
>
> Statement: we can see very long partial GC times (about 4 seconds) , standard times would be about 50ms....
>
>
> Thanks for your help
>
> Kind regards
>
> Jerome
>