You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Li Shengmei <li...@ict.ac.cn> on 2013/05/31 13:05:58 UTC
About the cpu-cores and CPU usage
Hi, all
I modify the yarn-site.xml of nodemanager and set
yarn.nodemanager.resource.cpu-cores as "8". There are "16" cores in
nodemanager. I use "top" command to monitor the CPU usage while running
application. I found that all 16 cores are in using. Can anyone explain the
case?
Thanks
May
Re: About the cpu-cores and CPU usage
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Addting to Bobby's answer, YARN-3 did most of the work to support cgroups
cpu controller, YARN-600 (still open) is to do the final wiring. With this
you'll be able to enforce CPU quotas (in the form of virtual cores).
thx
On Fri, May 31, 2013 at 8:45 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
> The CPU config is not for physical cores it is for "virtual" cores, an
> abstract concept that is not very well documented. More accurately it is
> dividing the CPU on the box into 8 schedule-able chunks. The current Linux
> implementation does not limit the utilization within the box. If there is
> a single container that asked for 1 core running on a box it can use all
> 16 cores with no restrictions. If there were two 1 core containers
> running on the node, then both of them would be limited to about 8
> physical cores each. If you want to reserve CPU specifically for other
> processes running on the same box, I don't think that we currently support
> that. You probably want to file a JIRA against YARN describing your use
> case.
>
> --Bobby
>
> On 5/31/13 6:05 AM, "Li Shengmei" <li...@ict.ac.cn> wrote:
>
> >Hi, all
> >
> > I modify the yarn-site.xml of nodemanager and set
> >yarn.nodemanager.resource.cpu-cores as "8". There are "16" cores in
> >nodemanager. I use "top" command to monitor the CPU usage while running
> >application. I found that all 16 cores are in using. Can anyone explain
> >the
> >case?
> >
> >
> >
> >Thanks
> >
> >May
> >
>
>
--
Alejandro
Re: About the cpu-cores and CPU usage
Posted by Robert Evans <ev...@yahoo-inc.com>.
The CPU config is not for physical cores it is for "virtual" cores, an
abstract concept that is not very well documented. More accurately it is
dividing the CPU on the box into 8 schedule-able chunks. The current Linux
implementation does not limit the utilization within the box. If there is
a single container that asked for 1 core running on a box it can use all
16 cores with no restrictions. If there were two 1 core containers
running on the node, then both of them would be limited to about 8
physical cores each. If you want to reserve CPU specifically for other
processes running on the same box, I don't think that we currently support
that. You probably want to file a JIRA against YARN describing your use
case.
--Bobby
On 5/31/13 6:05 AM, "Li Shengmei" <li...@ict.ac.cn> wrote:
>Hi, all
>
> I modify the yarn-site.xml of nodemanager and set
>yarn.nodemanager.resource.cpu-cores as "8". There are "16" cores in
>nodemanager. I use "top" command to monitor the CPU usage while running
>application. I found that all 16 cores are in using. Can anyone explain
>the
>case?
>
>
>
>Thanks
>
>May
>