You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Naganarasimha G R (Naga)" <ga...@huawei.com> on 2014/11/07 11:37:33 UTC
One query related to cgroupsin 2.4.1
Hi All,
In 2.4.1 When there is multiple components like Hbase and yarn runnning on the same node can we restrict the cpu usage of of all YARN containers in that NM through cgroups
Assume a node has 10 cores and cpu vcores for this NM is configured to 7. so suppose 7 containers each requesting for 1 core is run does cgroup ensure that cpu usgae of all the containers
do not increase more than 700% by all the containers?
Basically I want to restrict the cpu usage of all the containers in a given node so that the system processes and other components like HDFS, hbase etc.. run well along with YARN
How to acheive this?
Regards,
Naganarasimha G R
Huawei Technologies Co., Ltd.,
Re: One query related to cgroupsin 2.4.1
Posted by Steve Loughran <st...@hortonworks.com>.
if HBase is installed outside YARN it knows nothing about it and doesn't
let you directly control it's uses.
If it is deployed in YARN using slider you can request limits your memory
and cores; cgroups limits them
cgroups does throttle CPU. you ask for 1 vcore and you get whatever
fraction of a core your admins say 1 core is. You never get any more.
RAM limits, if enabled are hard: ask for too much and your job is killed.
If the NM has been given a number of cores/RAM below the total of your
host, then yes, it will leave space for hbase or something else.
HDFS doesn't do any limits. you can restrict the number of threads it has
to handle requests, but you really don't want to do that as things fail
oddly, even on remote machines. If you've ever tried to debug an app
failing this way, you'll appreciate why high numbers are better.
There's no IO throttling in any containers. There's some proposals, but
the fact that HDFS is doing the IO complicates things. Possibly more
interesting is the idea that you can also reserve vSpindles, giving you
access to (non HDFS) disks in the system in a way that lets you balance IO
workloads.
On 7 November 2014 10:37, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi All,
>
> In 2.4.1 When there is multiple components like Hbase and yarn runnning on
> the same node can we restrict the cpu usage of of all YARN containers in
> that NM through cgroups
> Assume a node has 10 cores and cpu vcores for this NM is configured to 7.
> so suppose 7 containers each requesting for 1 core is run does cgroup
> ensure that cpu usgae of all the containers
> do not increase more than 700% by all the containers?
>
> Basically I want to restrict the cpu usage of all the containers in a
> given node so that the system processes and other components like HDFS,
> hbase etc.. run well along with YARN
> How to acheive this?
>
>
> Regards,
>
> Naganarasimha G R
>
> Huawei Technologies Co., Ltd.,
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: One query related to cgroupsin 2.4.1
Posted by Sufi Nawaz <su...@eaiti.com>.
Please advise how to remove myself from this list.
Thank you,
*Sufi Nawaz *Application Innovator
e: sufi@eaiti.com */ *w: www.eaiti.com
o: (571) 306-4683 */ *c: (940) 595-1285
On Fri, Nov 7, 2014 at 5:37 AM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi All,
>
> In 2.4.1 When there is multiple components like Hbase and yarn runnning
> on the same node can we restrict the cpu usage of of all YARN containers in
> that NM through cgroups
> Assume a node has 10 cores and cpu vcores for this NM is configured to 7.
> so suppose 7 containers each requesting for 1 core is run does cgroup
> ensure that cpu usgae of all the containers
> do not increase more than 700% by all the containers?
>
> Basically I want to restrict the cpu usage of all the containers in a
> given node so that the system processes and other components like HDFS,
> hbase etc.. run well along with YARN
> How to acheive this?
>
> Regards,
>
> Naganarasimha G R
>
> Huawei Technologies Co., Ltd.,
>
>
>
Re: One query related to cgroupsin 2.4.1
Posted by Sufi Nawaz <su...@eaiti.com>.
Please advise how to remove myself from this list.
Thank you,
*Sufi Nawaz *Application Innovator
e: sufi@eaiti.com */ *w: www.eaiti.com
o: (571) 306-4683 */ *c: (940) 595-1285
On Fri, Nov 7, 2014 at 5:37 AM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi All,
>
> In 2.4.1 When there is multiple components like Hbase and yarn runnning
> on the same node can we restrict the cpu usage of of all YARN containers in
> that NM through cgroups
> Assume a node has 10 cores and cpu vcores for this NM is configured to 7.
> so suppose 7 containers each requesting for 1 core is run does cgroup
> ensure that cpu usgae of all the containers
> do not increase more than 700% by all the containers?
>
> Basically I want to restrict the cpu usage of all the containers in a
> given node so that the system processes and other components like HDFS,
> hbase etc.. run well along with YARN
> How to acheive this?
>
> Regards,
>
> Naganarasimha G R
>
> Huawei Technologies Co., Ltd.,
>
>
>
Re: One query related to cgroupsin 2.4.1
Posted by Sufi Nawaz <su...@eaiti.com>.
Please advise how to remove myself from this list.
Thank you,
*Sufi Nawaz *Application Innovator
e: sufi@eaiti.com */ *w: www.eaiti.com
o: (571) 306-4683 */ *c: (940) 595-1285
On Fri, Nov 7, 2014 at 5:37 AM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi All,
>
> In 2.4.1 When there is multiple components like Hbase and yarn runnning
> on the same node can we restrict the cpu usage of of all YARN containers in
> that NM through cgroups
> Assume a node has 10 cores and cpu vcores for this NM is configured to 7.
> so suppose 7 containers each requesting for 1 core is run does cgroup
> ensure that cpu usgae of all the containers
> do not increase more than 700% by all the containers?
>
> Basically I want to restrict the cpu usage of all the containers in a
> given node so that the system processes and other components like HDFS,
> hbase etc.. run well along with YARN
> How to acheive this?
>
> Regards,
>
> Naganarasimha G R
>
> Huawei Technologies Co., Ltd.,
>
>
>
Re: One query related to cgroupsin 2.4.1
Posted by Sufi Nawaz <su...@eaiti.com>.
Please advise how to remove myself from this list.
Thank you,
*Sufi Nawaz *Application Innovator
e: sufi@eaiti.com */ *w: www.eaiti.com
o: (571) 306-4683 */ *c: (940) 595-1285
On Fri, Nov 7, 2014 at 5:37 AM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi All,
>
> In 2.4.1 When there is multiple components like Hbase and yarn runnning
> on the same node can we restrict the cpu usage of of all YARN containers in
> that NM through cgroups
> Assume a node has 10 cores and cpu vcores for this NM is configured to 7.
> so suppose 7 containers each requesting for 1 core is run does cgroup
> ensure that cpu usgae of all the containers
> do not increase more than 700% by all the containers?
>
> Basically I want to restrict the cpu usage of all the containers in a
> given node so that the system processes and other components like HDFS,
> hbase etc.. run well along with YARN
> How to acheive this?
>
> Regards,
>
> Naganarasimha G R
>
> Huawei Technologies Co., Ltd.,
>
>
>