You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Artem Harutyunyan <ar...@mesosphere.io> on 2015/07/25 02:32:53 UTC
Re: Review Request 36627: Fixed cgroups oom killer and memory
pressure tests on Ubuntu 14.04.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36627/
-----------------------------------------------------------
(Updated July 24, 2015, 5:32 p.m.)
Review request for mesos, Benjamin Hindman and Joris Van Remoortere.
Summary (updated)
-----------------
Fixed cgroups oom killer and memory pressure tests on Ubuntu 14.04.
Bugs: MESOS-2660
https://issues.apache.org/jira/browse/MESOS-2660
Repository: mesos
Description
-------
See summary.
Diffs
-----
src/tests/cgroups_tests.cpp 9f5028f831a5a2fd4f7f0a0b7459c184ea9598fb
src/tests/memory_test_helper.cpp 8093e66c45cacc4bf0295a3448c128a1aa7c751a
Diff: https://reviews.apache.org/r/36627/diff/
Testing
-------
It seems there is still one more cgroups memory test failing on more test failing on my box. I'd like to fix that too and commit it together with this one.
sudo make check
Verified that the process actually gets killed by oom-killer:
```
# tail -f /var/log/syslog
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052405] lt-memory-test- invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052408] lt-memory-test- cpuset=/ mems_allowed=0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052411] CPU: 7 PID: 76599 Comm: lt-memory-test- Tainted: G OE 3.16.0-41-generic #57~14.04.1-Ubuntu
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] ffff88022efc1000 ffff8801fd2efc30 ffffffff81765721 ffff880231f10a30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052415] ffff8801fd2efcb8 ffffffff8175f2d5 ffff8802366f30c0 ffff8801e9405b00
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052416] ffff8801fd2efc80 ffffffff81165067 ffff880231f10ee8 ffff880231f10a30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052418] Call Trace:
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052424] [<ffffffff81765721>] dump_stack+0x45/0x56
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052426] [<ffffffff8175f2d5>] dump_header+0x7f/0x1f1
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052430] [<ffffffff81165067>] ? find_lock_task_mm+0x47/0xa0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052432] [<ffffffff811654e5>] oom_kill_process+0x205/0x360
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052434] [<ffffffff812eb975>] ? security_capable_noaudit+0x15/0x20
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052437] [<ffffffff811ca2e1>] mem_cgroup_oom_synchronize+0x581/0x5e0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052439] [<ffffffff811c97c0>] ? mem_cgroup_try_charge_mm+0xa0/0xa0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052440] [<ffffffff81165ce4>] pagefault_out_of_memory+0x14/0x80
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052442] [<ffffffff8175d97f>] mm_fault_error+0x67/0x140
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052445] [<ffffffff8105b28c>] __do_page_fault+0x4ec/0x560
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052449] [<ffffffff810a6208>] ? __enqueue_entity+0x78/0x80
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052450] [<ffffffff810a7f35>] ? set_next_entity+0x95/0xb0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052452] [<ffffffff81011627>] ? __switch_to+0x167/0x580
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052454] [<ffffffff8105b331>] do_page_fault+0x31/0x70
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052456] [<ffffffff8176fe68>] page_fault+0x28/0x30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052457] Task in /mesos_test killed as a result of limit of /mesos_test
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052459] memory: usage 65536kB, limit 65536kB, failcnt 24
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052460] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] Memory cgroup stats for /mesos_test: cache:0KB rss:65536KB rss_huge:63488KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:65536KB
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052467] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052574] [76599] 0 76599 82452 31766 122 1 0 lt-memory-test-
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052575] Memory cgroup out of memory: Kill process 76599 (lt-memory-test-) score 14 or sacrifice child
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052577] Killed process 76599 (lt-memory-test-) total-vm:329808kB, anon-rss:67988kB, file-rss:59076kB
```
Thanks,
Artem Harutyunyan
Re: Review Request 36627: Fixed cgroups oom killer and memory
pressure tests on Ubuntu 14.04.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36627/#review93018
-----------------------------------------------------------
Bad patch!
Reviews applied: [36618, 36627]
Failed command: ./support/apply-review.sh -n -r 36627
Error:
2015-07-25 00:48:22 URL:https://reviews.apache.org/r/36627/diff/raw/ [3244/3244] -> "36627.patch" [1]
error: src/tests/cgroups_tests.cpp: does not exist in index
error: src/tests/memory_test_helper.cpp: does not exist in index
Failed to apply patch
- Mesos ReviewBot
On July 25, 2015, 12:32 a.m., Artem Harutyunyan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36627/
> -----------------------------------------------------------
>
> (Updated July 25, 2015, 12:32 a.m.)
>
>
> Review request for mesos, Benjamin Hindman and Joris Van Remoortere.
>
>
> Bugs: MESOS-2660
> https://issues.apache.org/jira/browse/MESOS-2660
>
>
> Repository: mesos
>
>
> Description
> -------
>
> See summary.
>
>
> Diffs
> -----
>
> src/tests/cgroups_tests.cpp 9f5028f831a5a2fd4f7f0a0b7459c184ea9598fb
> src/tests/memory_test_helper.cpp 8093e66c45cacc4bf0295a3448c128a1aa7c751a
>
> Diff: https://reviews.apache.org/r/36627/diff/
>
>
> Testing
> -------
>
> It seems there is still one more cgroups memory test failing on more test failing on my box. I'd like to fix that too and commit it together with this one.
>
> sudo make check
>
> Verified that the process actually gets killed by oom-killer:
>
> ```
> # tail -f /var/log/syslog
>
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052405] lt-memory-test- invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052408] lt-memory-test- cpuset=/ mems_allowed=0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052411] CPU: 7 PID: 76599 Comm: lt-memory-test- Tainted: G OE 3.16.0-41-generic #57~14.04.1-Ubuntu
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] ffff88022efc1000 ffff8801fd2efc30 ffffffff81765721 ffff880231f10a30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052415] ffff8801fd2efcb8 ffffffff8175f2d5 ffff8802366f30c0 ffff8801e9405b00
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052416] ffff8801fd2efc80 ffffffff81165067 ffff880231f10ee8 ffff880231f10a30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052418] Call Trace:
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052424] [<ffffffff81765721>] dump_stack+0x45/0x56
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052426] [<ffffffff8175f2d5>] dump_header+0x7f/0x1f1
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052430] [<ffffffff81165067>] ? find_lock_task_mm+0x47/0xa0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052432] [<ffffffff811654e5>] oom_kill_process+0x205/0x360
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052434] [<ffffffff812eb975>] ? security_capable_noaudit+0x15/0x20
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052437] [<ffffffff811ca2e1>] mem_cgroup_oom_synchronize+0x581/0x5e0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052439] [<ffffffff811c97c0>] ? mem_cgroup_try_charge_mm+0xa0/0xa0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052440] [<ffffffff81165ce4>] pagefault_out_of_memory+0x14/0x80
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052442] [<ffffffff8175d97f>] mm_fault_error+0x67/0x140
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052445] [<ffffffff8105b28c>] __do_page_fault+0x4ec/0x560
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052449] [<ffffffff810a6208>] ? __enqueue_entity+0x78/0x80
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052450] [<ffffffff810a7f35>] ? set_next_entity+0x95/0xb0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052452] [<ffffffff81011627>] ? __switch_to+0x167/0x580
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052454] [<ffffffff8105b331>] do_page_fault+0x31/0x70
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052456] [<ffffffff8176fe68>] page_fault+0x28/0x30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052457] Task in /mesos_test killed as a result of limit of /mesos_test
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052459] memory: usage 65536kB, limit 65536kB, failcnt 24
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052460] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] Memory cgroup stats for /mesos_test: cache:0KB rss:65536KB rss_huge:63488KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:65536KB
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052467] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052574] [76599] 0 76599 82452 31766 122 1 0 lt-memory-test-
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052575] Memory cgroup out of memory: Kill process 76599 (lt-memory-test-) score 14 or sacrifice child
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052577] Killed process 76599 (lt-memory-test-) total-vm:329808kB, anon-rss:67988kB, file-rss:59076kB
>
>
> ```
>
>
> Thanks,
>
> Artem Harutyunyan
>
>
Re: Review Request 36627: Fixed cgroups oom killer and memory
pressure tests on Ubuntu 14.04.
Posted by Artem Harutyunyan <ar...@mesosphere.io>.
> On Aug. 1, 2015, 3:30 p.m., Benjamin Hindman wrote:
> > src/tests/containerizer/cgroups_tests.cpp, lines 1143-1146
> > <https://reviews.apache.org/r/36627/diff/5/?file=1024882#file1024882line1143>
> >
> > But doesn't this mean we might stay in an infinite loop forever? I'm assuming that somehow you figured out that things are just delayed but it eventually converges correctly, is that the case? Can you leave a comment on why not incrementing 'i' won't just make this an infinite loop?
The reasoning here is that counters will eventually become stable (that should happen on the second read) at which point the normal course of action (including incrementing the loop counter) should resume.
There is a comment about that a couple of lines above (lines 1120-1123):
```
// We need to know the readings are the same as last time to be
// sure they are stable, because the reading is not atomic. For
// example, the medium could turn positive after we read low to be
// 0, but this should be fixed by the next read immediately.
```
I will ammend this comment too.
> On Aug. 1, 2015, 3:30 p.m., Benjamin Hindman wrote:
> > src/tests/containerizer/memory_test_helper.cpp, line 93
> > <https://reviews.apache.org/r/36627/diff/5/?file=1024883#file1024883line93>
> >
> > Any specific reason to memset it to 0 instead of 1?
We thought that compiler might treat '0' as a special value that could affect optimization behaviour. It does not really seem to matter, I'll revert the change.
- Artem
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36627/#review93849
-----------------------------------------------------------
On July 29, 2015, 4:14 p.m., Artem Harutyunyan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36627/
> -----------------------------------------------------------
>
> (Updated July 29, 2015, 4:14 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Joris Van Remoortere.
>
>
> Bugs: MESOS-2660
> https://issues.apache.org/jira/browse/MESOS-2660
>
>
> Repository: mesos
>
>
> Description
> -------
>
> See summary.
>
>
> Diffs
> -----
>
> src/tests/containerizer/cgroups_tests.cpp caecd5dfa3fef33dba35cfc1b5934a11e2cc961a
> src/tests/containerizer/memory_test_helper.cpp 48a35632786963f484f66642b5c67afd4f7a89cc
>
> Diff: https://reviews.apache.org/r/36627/diff/
>
>
> Testing
> -------
>
> It seems there is still one more cgroups memory test failing on more test failing on my box. I'd like to fix that too and commit it together with this one.
>
> sudo make check
>
> Verified that the process actually gets killed by oom-killer:
>
> ```
> # tail -f /var/log/syslog
>
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052405] lt-memory-test- invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052408] lt-memory-test- cpuset=/ mems_allowed=0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052411] CPU: 7 PID: 76599 Comm: lt-memory-test- Tainted: G OE 3.16.0-41-generic #57~14.04.1-Ubuntu
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] ffff88022efc1000 ffff8801fd2efc30 ffffffff81765721 ffff880231f10a30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052415] ffff8801fd2efcb8 ffffffff8175f2d5 ffff8802366f30c0 ffff8801e9405b00
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052416] ffff8801fd2efc80 ffffffff81165067 ffff880231f10ee8 ffff880231f10a30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052418] Call Trace:
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052424] [<ffffffff81765721>] dump_stack+0x45/0x56
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052426] [<ffffffff8175f2d5>] dump_header+0x7f/0x1f1
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052430] [<ffffffff81165067>] ? find_lock_task_mm+0x47/0xa0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052432] [<ffffffff811654e5>] oom_kill_process+0x205/0x360
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052434] [<ffffffff812eb975>] ? security_capable_noaudit+0x15/0x20
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052437] [<ffffffff811ca2e1>] mem_cgroup_oom_synchronize+0x581/0x5e0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052439] [<ffffffff811c97c0>] ? mem_cgroup_try_charge_mm+0xa0/0xa0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052440] [<ffffffff81165ce4>] pagefault_out_of_memory+0x14/0x80
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052442] [<ffffffff8175d97f>] mm_fault_error+0x67/0x140
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052445] [<ffffffff8105b28c>] __do_page_fault+0x4ec/0x560
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052449] [<ffffffff810a6208>] ? __enqueue_entity+0x78/0x80
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052450] [<ffffffff810a7f35>] ? set_next_entity+0x95/0xb0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052452] [<ffffffff81011627>] ? __switch_to+0x167/0x580
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052454] [<ffffffff8105b331>] do_page_fault+0x31/0x70
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052456] [<ffffffff8176fe68>] page_fault+0x28/0x30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052457] Task in /mesos_test killed as a result of limit of /mesos_test
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052459] memory: usage 65536kB, limit 65536kB, failcnt 24
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052460] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] Memory cgroup stats for /mesos_test: cache:0KB rss:65536KB rss_huge:63488KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:65536KB
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052467] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052574] [76599] 0 76599 82452 31766 122 1 0 lt-memory-test-
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052575] Memory cgroup out of memory: Kill process 76599 (lt-memory-test-) score 14 or sacrifice child
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052577] Killed process 76599 (lt-memory-test-) total-vm:329808kB, anon-rss:67988kB, file-rss:59076kB
>
>
> ```
>
>
> Thanks,
>
> Artem Harutyunyan
>
>
Re: Review Request 36627: Fixed cgroups oom killer and memory
pressure tests on Ubuntu 14.04.
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36627/#review93849
-----------------------------------------------------------
Ship it!
src/tests/containerizer/cgroups_tests.cpp (lines 1143 - 1146)
<https://reviews.apache.org/r/36627/#comment148258>
But doesn't this mean we might stay in an infinite loop forever? I'm assuming that somehow you figured out that things are just delayed but it eventually converges correctly, is that the case? Can you leave a comment on why not incrementing 'i' won't just make this an infinite loop?
src/tests/containerizer/memory_test_helper.cpp (line 93)
<https://reviews.apache.org/r/36627/#comment148259>
Any specific reason to memset it to 0 instead of 1?
- Benjamin Hindman
On July 29, 2015, 11:14 p.m., Artem Harutyunyan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36627/
> -----------------------------------------------------------
>
> (Updated July 29, 2015, 11:14 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Joris Van Remoortere.
>
>
> Bugs: MESOS-2660
> https://issues.apache.org/jira/browse/MESOS-2660
>
>
> Repository: mesos
>
>
> Description
> -------
>
> See summary.
>
>
> Diffs
> -----
>
> src/tests/containerizer/cgroups_tests.cpp caecd5dfa3fef33dba35cfc1b5934a11e2cc961a
> src/tests/containerizer/memory_test_helper.cpp 48a35632786963f484f66642b5c67afd4f7a89cc
>
> Diff: https://reviews.apache.org/r/36627/diff/
>
>
> Testing
> -------
>
> It seems there is still one more cgroups memory test failing on more test failing on my box. I'd like to fix that too and commit it together with this one.
>
> sudo make check
>
> Verified that the process actually gets killed by oom-killer:
>
> ```
> # tail -f /var/log/syslog
>
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052405] lt-memory-test- invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052408] lt-memory-test- cpuset=/ mems_allowed=0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052411] CPU: 7 PID: 76599 Comm: lt-memory-test- Tainted: G OE 3.16.0-41-generic #57~14.04.1-Ubuntu
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] ffff88022efc1000 ffff8801fd2efc30 ffffffff81765721 ffff880231f10a30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052415] ffff8801fd2efcb8 ffffffff8175f2d5 ffff8802366f30c0 ffff8801e9405b00
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052416] ffff8801fd2efc80 ffffffff81165067 ffff880231f10ee8 ffff880231f10a30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052418] Call Trace:
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052424] [<ffffffff81765721>] dump_stack+0x45/0x56
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052426] [<ffffffff8175f2d5>] dump_header+0x7f/0x1f1
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052430] [<ffffffff81165067>] ? find_lock_task_mm+0x47/0xa0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052432] [<ffffffff811654e5>] oom_kill_process+0x205/0x360
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052434] [<ffffffff812eb975>] ? security_capable_noaudit+0x15/0x20
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052437] [<ffffffff811ca2e1>] mem_cgroup_oom_synchronize+0x581/0x5e0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052439] [<ffffffff811c97c0>] ? mem_cgroup_try_charge_mm+0xa0/0xa0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052440] [<ffffffff81165ce4>] pagefault_out_of_memory+0x14/0x80
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052442] [<ffffffff8175d97f>] mm_fault_error+0x67/0x140
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052445] [<ffffffff8105b28c>] __do_page_fault+0x4ec/0x560
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052449] [<ffffffff810a6208>] ? __enqueue_entity+0x78/0x80
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052450] [<ffffffff810a7f35>] ? set_next_entity+0x95/0xb0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052452] [<ffffffff81011627>] ? __switch_to+0x167/0x580
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052454] [<ffffffff8105b331>] do_page_fault+0x31/0x70
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052456] [<ffffffff8176fe68>] page_fault+0x28/0x30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052457] Task in /mesos_test killed as a result of limit of /mesos_test
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052459] memory: usage 65536kB, limit 65536kB, failcnt 24
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052460] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] Memory cgroup stats for /mesos_test: cache:0KB rss:65536KB rss_huge:63488KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:65536KB
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052467] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052574] [76599] 0 76599 82452 31766 122 1 0 lt-memory-test-
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052575] Memory cgroup out of memory: Kill process 76599 (lt-memory-test-) score 14 or sacrifice child
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052577] Killed process 76599 (lt-memory-test-) total-vm:329808kB, anon-rss:67988kB, file-rss:59076kB
>
>
> ```
>
>
> Thanks,
>
> Artem Harutyunyan
>
>
Re: Review Request 36627: Fixed cgroups oom killer and memory
pressure tests on Ubuntu 14.04.
Posted by Artem Harutyunyan <ar...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36627/
-----------------------------------------------------------
(Updated Aug. 1, 2015, 3:52 p.m.)
Review request for mesos, Benjamin Hindman and Joris Van Remoortere.
Changes
-------
Addressed comments.
Bugs: MESOS-2660
https://issues.apache.org/jira/browse/MESOS-2660
Repository: mesos
Description
-------
See summary.
Diffs (updated)
-----
src/tests/containerizer/cgroups_tests.cpp ea54ab8f691e4d4d87224a815ee055e7bb84dfe3
src/tests/containerizer/memory_test_helper.cpp 48a35632786963f484f66642b5c67afd4f7a89cc
Diff: https://reviews.apache.org/r/36627/diff/
Testing
-------
It seems there is still one more cgroups memory test failing on more test failing on my box. I'd like to fix that too and commit it together with this one.
sudo make check
Verified that the process actually gets killed by oom-killer:
```
# tail -f /var/log/syslog
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052405] lt-memory-test- invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052408] lt-memory-test- cpuset=/ mems_allowed=0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052411] CPU: 7 PID: 76599 Comm: lt-memory-test- Tainted: G OE 3.16.0-41-generic #57~14.04.1-Ubuntu
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] ffff88022efc1000 ffff8801fd2efc30 ffffffff81765721 ffff880231f10a30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052415] ffff8801fd2efcb8 ffffffff8175f2d5 ffff8802366f30c0 ffff8801e9405b00
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052416] ffff8801fd2efc80 ffffffff81165067 ffff880231f10ee8 ffff880231f10a30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052418] Call Trace:
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052424] [<ffffffff81765721>] dump_stack+0x45/0x56
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052426] [<ffffffff8175f2d5>] dump_header+0x7f/0x1f1
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052430] [<ffffffff81165067>] ? find_lock_task_mm+0x47/0xa0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052432] [<ffffffff811654e5>] oom_kill_process+0x205/0x360
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052434] [<ffffffff812eb975>] ? security_capable_noaudit+0x15/0x20
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052437] [<ffffffff811ca2e1>] mem_cgroup_oom_synchronize+0x581/0x5e0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052439] [<ffffffff811c97c0>] ? mem_cgroup_try_charge_mm+0xa0/0xa0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052440] [<ffffffff81165ce4>] pagefault_out_of_memory+0x14/0x80
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052442] [<ffffffff8175d97f>] mm_fault_error+0x67/0x140
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052445] [<ffffffff8105b28c>] __do_page_fault+0x4ec/0x560
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052449] [<ffffffff810a6208>] ? __enqueue_entity+0x78/0x80
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052450] [<ffffffff810a7f35>] ? set_next_entity+0x95/0xb0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052452] [<ffffffff81011627>] ? __switch_to+0x167/0x580
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052454] [<ffffffff8105b331>] do_page_fault+0x31/0x70
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052456] [<ffffffff8176fe68>] page_fault+0x28/0x30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052457] Task in /mesos_test killed as a result of limit of /mesos_test
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052459] memory: usage 65536kB, limit 65536kB, failcnt 24
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052460] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] Memory cgroup stats for /mesos_test: cache:0KB rss:65536KB rss_huge:63488KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:65536KB
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052467] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052574] [76599] 0 76599 82452 31766 122 1 0 lt-memory-test-
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052575] Memory cgroup out of memory: Kill process 76599 (lt-memory-test-) score 14 or sacrifice child
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052577] Killed process 76599 (lt-memory-test-) total-vm:329808kB, anon-rss:67988kB, file-rss:59076kB
```
Thanks,
Artem Harutyunyan
Re: Review Request 36627: Fixed cgroups oom killer and memory
pressure tests on Ubuntu 14.04.
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36627/#review93536
-----------------------------------------------------------
Patch looks great!
Reviews applied: [36618, 36627]
All tests passed.
- Mesos ReviewBot
On July 29, 2015, 11:14 p.m., Artem Harutyunyan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36627/
> -----------------------------------------------------------
>
> (Updated July 29, 2015, 11:14 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Joris Van Remoortere.
>
>
> Bugs: MESOS-2660
> https://issues.apache.org/jira/browse/MESOS-2660
>
>
> Repository: mesos
>
>
> Description
> -------
>
> See summary.
>
>
> Diffs
> -----
>
> src/tests/containerizer/cgroups_tests.cpp caecd5dfa3fef33dba35cfc1b5934a11e2cc961a
> src/tests/containerizer/memory_test_helper.cpp 48a35632786963f484f66642b5c67afd4f7a89cc
>
> Diff: https://reviews.apache.org/r/36627/diff/
>
>
> Testing
> -------
>
> It seems there is still one more cgroups memory test failing on more test failing on my box. I'd like to fix that too and commit it together with this one.
>
> sudo make check
>
> Verified that the process actually gets killed by oom-killer:
>
> ```
> # tail -f /var/log/syslog
>
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052405] lt-memory-test- invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052408] lt-memory-test- cpuset=/ mems_allowed=0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052411] CPU: 7 PID: 76599 Comm: lt-memory-test- Tainted: G OE 3.16.0-41-generic #57~14.04.1-Ubuntu
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] ffff88022efc1000 ffff8801fd2efc30 ffffffff81765721 ffff880231f10a30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052415] ffff8801fd2efcb8 ffffffff8175f2d5 ffff8802366f30c0 ffff8801e9405b00
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052416] ffff8801fd2efc80 ffffffff81165067 ffff880231f10ee8 ffff880231f10a30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052418] Call Trace:
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052424] [<ffffffff81765721>] dump_stack+0x45/0x56
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052426] [<ffffffff8175f2d5>] dump_header+0x7f/0x1f1
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052430] [<ffffffff81165067>] ? find_lock_task_mm+0x47/0xa0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052432] [<ffffffff811654e5>] oom_kill_process+0x205/0x360
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052434] [<ffffffff812eb975>] ? security_capable_noaudit+0x15/0x20
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052437] [<ffffffff811ca2e1>] mem_cgroup_oom_synchronize+0x581/0x5e0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052439] [<ffffffff811c97c0>] ? mem_cgroup_try_charge_mm+0xa0/0xa0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052440] [<ffffffff81165ce4>] pagefault_out_of_memory+0x14/0x80
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052442] [<ffffffff8175d97f>] mm_fault_error+0x67/0x140
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052445] [<ffffffff8105b28c>] __do_page_fault+0x4ec/0x560
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052449] [<ffffffff810a6208>] ? __enqueue_entity+0x78/0x80
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052450] [<ffffffff810a7f35>] ? set_next_entity+0x95/0xb0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052452] [<ffffffff81011627>] ? __switch_to+0x167/0x580
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052454] [<ffffffff8105b331>] do_page_fault+0x31/0x70
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052456] [<ffffffff8176fe68>] page_fault+0x28/0x30
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052457] Task in /mesos_test killed as a result of limit of /mesos_test
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052459] memory: usage 65536kB, limit 65536kB, failcnt 24
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052460] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] Memory cgroup stats for /mesos_test: cache:0KB rss:65536KB rss_huge:63488KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:65536KB
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052467] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052574] [76599] 0 76599 82452 31766 122 1 0 lt-memory-test-
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052575] Memory cgroup out of memory: Kill process 76599 (lt-memory-test-) score 14 or sacrifice child
> Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052577] Killed process 76599 (lt-memory-test-) total-vm:329808kB, anon-rss:67988kB, file-rss:59076kB
>
>
> ```
>
>
> Thanks,
>
> Artem Harutyunyan
>
>
Re: Review Request 36627: Fixed cgroups oom killer and memory
pressure tests on Ubuntu 14.04.
Posted by Artem Harutyunyan <ar...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36627/
-----------------------------------------------------------
(Updated July 29, 2015, 4:14 p.m.)
Review request for mesos, Benjamin Hindman and Joris Van Remoortere.
Changes
-------
Remove some extra whitespace.
Bugs: MESOS-2660
https://issues.apache.org/jira/browse/MESOS-2660
Repository: mesos
Description
-------
See summary.
Diffs (updated)
-----
src/tests/containerizer/cgroups_tests.cpp caecd5dfa3fef33dba35cfc1b5934a11e2cc961a
src/tests/containerizer/memory_test_helper.cpp 48a35632786963f484f66642b5c67afd4f7a89cc
Diff: https://reviews.apache.org/r/36627/diff/
Testing
-------
It seems there is still one more cgroups memory test failing on more test failing on my box. I'd like to fix that too and commit it together with this one.
sudo make check
Verified that the process actually gets killed by oom-killer:
```
# tail -f /var/log/syslog
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052405] lt-memory-test- invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052408] lt-memory-test- cpuset=/ mems_allowed=0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052411] CPU: 7 PID: 76599 Comm: lt-memory-test- Tainted: G OE 3.16.0-41-generic #57~14.04.1-Ubuntu
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] ffff88022efc1000 ffff8801fd2efc30 ffffffff81765721 ffff880231f10a30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052415] ffff8801fd2efcb8 ffffffff8175f2d5 ffff8802366f30c0 ffff8801e9405b00
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052416] ffff8801fd2efc80 ffffffff81165067 ffff880231f10ee8 ffff880231f10a30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052418] Call Trace:
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052424] [<ffffffff81765721>] dump_stack+0x45/0x56
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052426] [<ffffffff8175f2d5>] dump_header+0x7f/0x1f1
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052430] [<ffffffff81165067>] ? find_lock_task_mm+0x47/0xa0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052432] [<ffffffff811654e5>] oom_kill_process+0x205/0x360
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052434] [<ffffffff812eb975>] ? security_capable_noaudit+0x15/0x20
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052437] [<ffffffff811ca2e1>] mem_cgroup_oom_synchronize+0x581/0x5e0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052439] [<ffffffff811c97c0>] ? mem_cgroup_try_charge_mm+0xa0/0xa0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052440] [<ffffffff81165ce4>] pagefault_out_of_memory+0x14/0x80
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052442] [<ffffffff8175d97f>] mm_fault_error+0x67/0x140
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052445] [<ffffffff8105b28c>] __do_page_fault+0x4ec/0x560
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052449] [<ffffffff810a6208>] ? __enqueue_entity+0x78/0x80
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052450] [<ffffffff810a7f35>] ? set_next_entity+0x95/0xb0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052452] [<ffffffff81011627>] ? __switch_to+0x167/0x580
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052454] [<ffffffff8105b331>] do_page_fault+0x31/0x70
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052456] [<ffffffff8176fe68>] page_fault+0x28/0x30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052457] Task in /mesos_test killed as a result of limit of /mesos_test
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052459] memory: usage 65536kB, limit 65536kB, failcnt 24
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052460] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] Memory cgroup stats for /mesos_test: cache:0KB rss:65536KB rss_huge:63488KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:65536KB
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052467] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052574] [76599] 0 76599 82452 31766 122 1 0 lt-memory-test-
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052575] Memory cgroup out of memory: Kill process 76599 (lt-memory-test-) score 14 or sacrifice child
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052577] Killed process 76599 (lt-memory-test-) total-vm:329808kB, anon-rss:67988kB, file-rss:59076kB
```
Thanks,
Artem Harutyunyan
Re: Review Request 36627: Fixed cgroups oom killer and memory
pressure tests on Ubuntu 14.04.
Posted by Artem Harutyunyan <ar...@mesosphere.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36627/
-----------------------------------------------------------
(Updated July 29, 2015, 4:11 p.m.)
Review request for mesos, Benjamin Hindman and Joris Van Remoortere.
Changes
-------
Rebasing.
Bugs: MESOS-2660
https://issues.apache.org/jira/browse/MESOS-2660
Repository: mesos
Description
-------
See summary.
Diffs (updated)
-----
src/tests/containerizer/cgroups_tests.cpp caecd5dfa3fef33dba35cfc1b5934a11e2cc961a
src/tests/containerizer/memory_test_helper.cpp 48a35632786963f484f66642b5c67afd4f7a89cc
Diff: https://reviews.apache.org/r/36627/diff/
Testing
-------
It seems there is still one more cgroups memory test failing on more test failing on my box. I'd like to fix that too and commit it together with this one.
sudo make check
Verified that the process actually gets killed by oom-killer:
```
# tail -f /var/log/syslog
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052405] lt-memory-test- invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052408] lt-memory-test- cpuset=/ mems_allowed=0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052411] CPU: 7 PID: 76599 Comm: lt-memory-test- Tainted: G OE 3.16.0-41-generic #57~14.04.1-Ubuntu
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052413] ffff88022efc1000 ffff8801fd2efc30 ffffffff81765721 ffff880231f10a30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052415] ffff8801fd2efcb8 ffffffff8175f2d5 ffff8802366f30c0 ffff8801e9405b00
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052416] ffff8801fd2efc80 ffffffff81165067 ffff880231f10ee8 ffff880231f10a30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052418] Call Trace:
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052424] [<ffffffff81765721>] dump_stack+0x45/0x56
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052426] [<ffffffff8175f2d5>] dump_header+0x7f/0x1f1
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052430] [<ffffffff81165067>] ? find_lock_task_mm+0x47/0xa0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052432] [<ffffffff811654e5>] oom_kill_process+0x205/0x360
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052434] [<ffffffff812eb975>] ? security_capable_noaudit+0x15/0x20
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052437] [<ffffffff811ca2e1>] mem_cgroup_oom_synchronize+0x581/0x5e0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052439] [<ffffffff811c97c0>] ? mem_cgroup_try_charge_mm+0xa0/0xa0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052440] [<ffffffff81165ce4>] pagefault_out_of_memory+0x14/0x80
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052442] [<ffffffff8175d97f>] mm_fault_error+0x67/0x140
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052445] [<ffffffff8105b28c>] __do_page_fault+0x4ec/0x560
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052449] [<ffffffff810a6208>] ? __enqueue_entity+0x78/0x80
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052450] [<ffffffff810a7f35>] ? set_next_entity+0x95/0xb0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052452] [<ffffffff81011627>] ? __switch_to+0x167/0x580
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052454] [<ffffffff8105b331>] do_page_fault+0x31/0x70
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052456] [<ffffffff8176fe68>] page_fault+0x28/0x30
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052457] Task in /mesos_test killed as a result of limit of /mesos_test
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052459] memory: usage 65536kB, limit 65536kB, failcnt 24
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052460] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052461] Memory cgroup stats for /mesos_test: cache:0KB rss:65536KB rss_huge:63488KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:65536KB
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052467] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052574] [76599] 0 76599 82452 31766 122 1 0 lt-memory-test-
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052575] Memory cgroup out of memory: Kill process 76599 (lt-memory-test-) score 14 or sacrifice child
Jul 22 14:56:00 harutyunyan-virtual-machine kernel: [17440.052577] Killed process 76599 (lt-memory-test-) total-vm:329808kB, anon-rss:67988kB, file-rss:59076kB
```
Thanks,
Artem Harutyunyan