You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Gary Ogden <go...@gmail.com> on 2015/01/24 17:26:07 UTC
slave kernel panic with cgroups on a shared box
I'm using this:
3 Node Cluster 8GB 2 CPU each
CentOS 6.5 2.6.32-504.3.3.el6.x86_64
On each node we run Cassandra and Mesos where we run java spark jobs. This
is a testing environment so it's actually shared between test groups. So
we actually are running 3 instances of mesos-slave on each node.
(integration, qa and preprod). We want to ensure these sparks jobs don't
slow down Cassandra.
If I don't use cgroups, we don't get a kernel panic. No matter how I try to
configure cgroups, I still get the panic and reboot. Is there an issue
with having multiple slaves on the same machine? Here's the kernel panic
text:
<4>Process mesos-slave (pid: 19593, threadinfo ffff88023a224000, task
ffff8800aa1af540)
<4>Stack:
<4> ffff88023a225dd8 ffff8802395b6580 ffff88023a225e08 ffffffff810cdaa2
<4><d> ffff88008f740440 ffff8800aa0df938 0000000000000000 ffff8800aa0df950
<4><d> ffff88023a225e58 ffffffff810577e9 ffff88008f740440 0000000300000001
<4>Call Trace:
<4> [<ffffffff810cdaa2>] cgroup_event_wake+0x42/0x70
<4> [<ffffffff810577e9>] __wake_up_common+0x59/0x90
<4> [<ffffffff8105bd18>] __wake_up+0x48/0x70
<4> [<ffffffff811dad2d>] eventfd_release+0x2d/0x40
<4> [<ffffffff8118f8d5>] __fput+0xf5/0x210
<4> [<ffffffff8118fa15>] fput+0x25/0x30
<4> [<ffffffff8118ac6d>] filp_close+0x5d/0x90
<4> [<ffffffff8118ad45>] sys_close+0xa5/0x100
<4> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
<4>Code: 01 01 01 01 01 48 0f af c2 48 c1 e8 38 c3 90 90 90 90 90 90 90 90
90 90 90 90 90 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 8b 47 08 <4c> 8b 00
4c 39 c7 75
39 48 8b 03 4c 8b 40 08 4c 39 c3 75 4c 48
<1>RIP [<ffffffff8129e870>] list_del+0x10/0xa0
<4> RSP <ffff88023a225dc8>
Here's the cgconfig.conf:
mount {
cpu = /cgroup/cpu;
cpuacct = /cgroup/cpuacct;
memory = /cgroup/memory;
}
group cassandra {
cpu {
cpu.shares="800";
}
cpuacct {
cpuacct.usage="0";
}
memory {
memory.limit_in_bytes="5G";
memory.memsw.limit_in_bytes="5G";
}
}
group mesos {
cpu {
cpu.shares="200";
}
cpuacct {
cpuacct.usage="0";
}
memory {
memory.limit_in_bytes="1G";
memory.memsw.limit_in_bytes="1G";
}
}
Here's the cgrules.conf:
@mesos cpu,cpuacct,memory mesos
@cassandra cpu,cpuacct,memory cassandra
And here's how we start each slave:
cgexec -g cpu,cpuacct,memory:mesos /usr/sbin/mesos-slave
--isolation=cgroups/cpu,cgroups/mem --cgroups_limit_swap
--cgroups_hierarchy=/cgroup --resource
s="mem(*):256;cpus(*):1;ports(*):[20000-25000];disk(*):5000"
--gc_delay=2days --cgroups_root=mesos --log_dir=/var/log/mesos/int
--master=zk://intMesosMaster01:2
181,intMesosMaster02:2181,intMesosMaster03:2181/mesos --port=5150
--work_dir=/tmp/mesos/int
I've tried lots of different settings in cgroups, the startup of the
slaves, but nothing seems to matter. We've also disabled swap on these
boxes since Cassandra doesn't like swap.
Re: slave kernel panic with cgroups on a shared box
Posted by Benjamin Mahler <be...@gmail.com>.
It's a kernel bug from the looks of it, here's a similar stack trace:
https://bugs.centos.org/view.php?id=7538
But apparently they fixed it in the kernel you're running.
On Sat, Jan 24, 2015 at 8:26 AM, Gary Ogden <go...@gmail.com> wrote:
> I'm using this:
> 3 Node Cluster 8GB 2 CPU each
> CentOS 6.5 2.6.32-504.3.3.el6.x86_64
>
> On each node we run Cassandra and Mesos where we run java spark jobs.
> This is a testing environment so it's actually shared between test groups.
> So we actually are running 3 instances of mesos-slave on each node.
> (integration, qa and preprod). We want to ensure these sparks jobs don't
> slow down Cassandra.
>
> If I don't use cgroups, we don't get a kernel panic. No matter how I try
> to configure cgroups, I still get the panic and reboot. Is there an issue
> with having multiple slaves on the same machine? Here's the kernel panic
> text:
>
> <4>Process mesos-slave (pid: 19593, threadinfo ffff88023a224000, task
> ffff8800aa1af540)
> <4>Stack:
> <4> ffff88023a225dd8 ffff8802395b6580 ffff88023a225e08 ffffffff810cdaa2
> <4><d> ffff88008f740440 ffff8800aa0df938 0000000000000000 ffff8800aa0df950
> <4><d> ffff88023a225e58 ffffffff810577e9 ffff88008f740440 0000000300000001
> <4>Call Trace:
> <4> [<ffffffff810cdaa2>] cgroup_event_wake+0x42/0x70
> <4> [<ffffffff810577e9>] __wake_up_common+0x59/0x90
> <4> [<ffffffff8105bd18>] __wake_up+0x48/0x70
> <4> [<ffffffff811dad2d>] eventfd_release+0x2d/0x40
> <4> [<ffffffff8118f8d5>] __fput+0xf5/0x210
> <4> [<ffffffff8118fa15>] fput+0x25/0x30
> <4> [<ffffffff8118ac6d>] filp_close+0x5d/0x90
> <4> [<ffffffff8118ad45>] sys_close+0xa5/0x100
> <4> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> <4>Code: 01 01 01 01 01 48 0f af c2 48 c1 e8 38 c3 90 90 90 90 90 90 90 90
> 90 90 90 90 90 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 8b 47 08 <4c> 8b 00
> 4c 39 c7 75
> 39 48 8b 03 4c 8b 40 08 4c 39 c3 75 4c 48
> <1>RIP [<ffffffff8129e870>] list_del+0x10/0xa0
> <4> RSP <ffff88023a225dc8>
>
> Here's the cgconfig.conf:
>
> mount {
> cpu = /cgroup/cpu;
> cpuacct = /cgroup/cpuacct;
> memory = /cgroup/memory;
> }
>
> group cassandra {
> cpu {
> cpu.shares="800";
> }
> cpuacct {
> cpuacct.usage="0";
> }
> memory {
> memory.limit_in_bytes="5G";
> memory.memsw.limit_in_bytes="5G";
> }
> }
>
>
> group mesos {
> cpu {
> cpu.shares="200";
> }
> cpuacct {
> cpuacct.usage="0";
> }
> memory {
> memory.limit_in_bytes="1G";
> memory.memsw.limit_in_bytes="1G";
> }
> }
>
> Here's the cgrules.conf:
>
> @mesos cpu,cpuacct,memory mesos
> @cassandra cpu,cpuacct,memory cassandra
>
>
> And here's how we start each slave:
> cgexec -g cpu,cpuacct,memory:mesos /usr/sbin/mesos-slave
> --isolation=cgroups/cpu,cgroups/mem --cgroups_limit_swap
> --cgroups_hierarchy=/cgroup --resource
> s="mem(*):256;cpus(*):1;ports(*):[20000-25000];disk(*):5000"
> --gc_delay=2days --cgroups_root=mesos --log_dir=/var/log/mesos/int
> --master=zk://intMesosMaster01:2
> 181,intMesosMaster02:2181,intMesosMaster03:2181/mesos --port=5150
> --work_dir=/tmp/mesos/int
>
>
> I've tried lots of different settings in cgroups, the startup of the
> slaves, but nothing seems to matter. We've also disabled swap on these
> boxes since Cassandra doesn't like swap.
>
>
>
>