You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by David Villegas <dv...@gmail.com> on 2015/11/17 17:12:03 UTC
CPU isolation with cgroups and RHEL 6
Hi all,
We are running some benchmarks (in the order of 1TB of data) while enabling
CPU isolation using cgroups in YARN containers. However, we are often
hitting kernel panics in RHEL 6 with kernel 2.6.32-504.el6.x86_64. We've
tried using RHEL 7 and so far we haven't hit the same issues.
Has anyone else seen this? I found some people are experiencing similar
problems (YARN-4048), so I wonder if there's any kind of consensus about
ditching RHEL / CentOS 6 for this type of setup and just supporting v7, or
if it is possible to make it work with some tweaking.
Regards,
David Villegas.
RE: CPU isolation with cgroups and RHEL 6
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi David,
We too have faced this many times and as alternative where OS cannot be upgraded we have CPU isolation based on CPUset which is not same as existing which based on period & quota. It doesnt guarantee the usage among containers is balanced efficiently but atleast will ensure the cpu usage of the yarn containers in a node doesnt exceed the number of cores assigned for it.
Regards,
Naga
________________________________
From: David Villegas [dville00@gmail.com]
Sent: Tuesday, November 17, 2015 21:42
To: user@hadoop.apache.org
Subject: CPU isolation with cgroups and RHEL 6
Hi all,
We are running some benchmarks (in the order of 1TB of data) while enabling CPU isolation using cgroups in YARN containers. However, we are often hitting kernel panics in RHEL 6 with kernel 2.6.32-504.el6.x86_64. We've tried using RHEL 7 and so far we haven't hit the same issues.
Has anyone else seen this? I found some people are experiencing similar problems (YARN-4048), so I wonder if there's any kind of consensus about ditching RHEL / CentOS 6 for this type of setup and just supporting v7, or if it is possible to make it work with some tweaking.
Regards,
David Villegas.
RE: CPU isolation with cgroups and RHEL 6
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi David,
We too have faced this many times and as alternative where OS cannot be upgraded we have CPU isolation based on CPUset which is not same as existing which based on period & quota. It doesnt guarantee the usage among containers is balanced efficiently but atleast will ensure the cpu usage of the yarn containers in a node doesnt exceed the number of cores assigned for it.
Regards,
Naga
________________________________
From: David Villegas [dville00@gmail.com]
Sent: Tuesday, November 17, 2015 21:42
To: user@hadoop.apache.org
Subject: CPU isolation with cgroups and RHEL 6
Hi all,
We are running some benchmarks (in the order of 1TB of data) while enabling CPU isolation using cgroups in YARN containers. However, we are often hitting kernel panics in RHEL 6 with kernel 2.6.32-504.el6.x86_64. We've tried using RHEL 7 and so far we haven't hit the same issues.
Has anyone else seen this? I found some people are experiencing similar problems (YARN-4048), so I wonder if there's any kind of consensus about ditching RHEL / CentOS 6 for this type of setup and just supporting v7, or if it is possible to make it work with some tweaking.
Regards,
David Villegas.
RE: CPU isolation with cgroups and RHEL 6
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi David,
We too have faced this many times and as alternative where OS cannot be upgraded we have CPU isolation based on CPUset which is not same as existing which based on period & quota. It doesnt guarantee the usage among containers is balanced efficiently but atleast will ensure the cpu usage of the yarn containers in a node doesnt exceed the number of cores assigned for it.
Regards,
Naga
________________________________
From: David Villegas [dville00@gmail.com]
Sent: Tuesday, November 17, 2015 21:42
To: user@hadoop.apache.org
Subject: CPU isolation with cgroups and RHEL 6
Hi all,
We are running some benchmarks (in the order of 1TB of data) while enabling CPU isolation using cgroups in YARN containers. However, we are often hitting kernel panics in RHEL 6 with kernel 2.6.32-504.el6.x86_64. We've tried using RHEL 7 and so far we haven't hit the same issues.
Has anyone else seen this? I found some people are experiencing similar problems (YARN-4048), so I wonder if there's any kind of consensus about ditching RHEL / CentOS 6 for this type of setup and just supporting v7, or if it is possible to make it work with some tweaking.
Regards,
David Villegas.
RE: CPU isolation with cgroups and RHEL 6
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi David,
We too have faced this many times and as alternative where OS cannot be upgraded we have CPU isolation based on CPUset which is not same as existing which based on period & quota. It doesnt guarantee the usage among containers is balanced efficiently but atleast will ensure the cpu usage of the yarn containers in a node doesnt exceed the number of cores assigned for it.
Regards,
Naga
________________________________
From: David Villegas [dville00@gmail.com]
Sent: Tuesday, November 17, 2015 21:42
To: user@hadoop.apache.org
Subject: CPU isolation with cgroups and RHEL 6
Hi all,
We are running some benchmarks (in the order of 1TB of data) while enabling CPU isolation using cgroups in YARN containers. However, we are often hitting kernel panics in RHEL 6 with kernel 2.6.32-504.el6.x86_64. We've tried using RHEL 7 and so far we haven't hit the same issues.
Has anyone else seen this? I found some people are experiencing similar problems (YARN-4048), so I wonder if there's any kind of consensus about ditching RHEL / CentOS 6 for this type of setup and just supporting v7, or if it is possible to make it work with some tweaking.
Regards,
David Villegas.