You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Vinod Kone <vi...@gmail.com> on 2012/11/29 01:02:41 UTC
Review Request: Properly deallocates exited executor's resources"
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/
-----------------------------------------------------------
Review request for mesos, Benjamin Hindman and Ben Mahler.
Description
-------
Tested by modifying balloon framework test to start the slave cpusets enabled.
Will be writing more tests soon.
Diffs
-----
src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
Diff: https://reviews.apache.org/r/8255/diff/
Testing
-------
udo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cgroup*
Thanks,
Vinod Kone
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Vinod Kone <vi...@gmail.com>.
> On Nov. 29, 2012, 3:07 a.m., Ben Mahler wrote:
> > src/slave/cgroups_isolation_module.cpp, line 940
> > <https://reviews.apache.org/r/8255/diff/1/?file=231391#file231391line940>
> >
> > You need to store the result of the shrink and apply it to the cpus (similarly to how cpusetChanged() works).
done
> On Nov. 29, 2012, 3:07 a.m., Ben Mahler wrote:
> > src/slave/cgroups_isolation_module.cpp, line 707
> > <https://reviews.apache.org/r/8255/diff/1/?file=231391#file231391line707>
> >
> > Can you get this all in one log line? Meaning, build up a string and do one LOG call?
> >
> > Otherwise it will be a little tricky to read interspersed with other logging.
done
> On Nov. 29, 2012, 3:07 a.m., Ben Mahler wrote:
> > src/slave/cgroups_isolation_module.cpp, line 695
> > <https://reviews.apache.org/r/8255/diff/1/?file=231391#file231391line695>
> >
> > Instead of printCpusets(): logCpusets()?
> >
> > But even better:
> > LOG(INFO) << cpusetsState(); // where cpusetsState() returns the string.
> >
> > I'm seeing now that if we had Cpuset as a tree, this stuff could be a bit cleaner, but that's a bigger change so let's forget it for now.
sure.
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/#review13837
-----------------------------------------------------------
On Nov. 29, 2012, 12:02 a.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8255/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2012, 12:02 a.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Tested by modifying balloon framework test to start the slave cpusets enabled.
>
> Will be writing more tests soon.
>
>
> Diffs
> -----
>
> src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
> src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
> src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
>
> Diff: https://reviews.apache.org/r/8255/diff/
>
>
> Testing
> -------
>
> udo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cgroup*
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/#review13837
-----------------------------------------------------------
Thanks for catching this! Glad to see it wasn't a bug in grow/shrink :)
src/slave/cgroups_isolation_module.cpp
<https://reviews.apache.org/r/8255/#comment29569>
Instead of printCpusets(): logCpusets()?
But even better:
LOG(INFO) << cpusetsState(); // where cpusetsState() returns the string.
I'm seeing now that if we had Cpuset as a tree, this stuff could be a bit cleaner, but that's a bigger change so let's forget it for now.
src/slave/cgroups_isolation_module.cpp
<https://reviews.apache.org/r/8255/#comment29570>
CHECK_NOT_NULL instead
src/slave/cgroups_isolation_module.cpp
<https://reviews.apache.org/r/8255/#comment29568>
Can you get this all in one log line? Meaning, build up a string and do one LOG call?
Otherwise it will be a little tricky to read interspersed with other logging.
src/slave/cgroups_isolation_module.cpp
<https://reviews.apache.org/r/8255/#comment29567>
You need to store the result of the shrink and apply it to the cpus (similarly to how cpusetChanged() works).
- Ben Mahler
On Nov. 29, 2012, 12:02 a.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8255/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2012, 12:02 a.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Tested by modifying balloon framework test to start the slave cpusets enabled.
>
> Will be writing more tests soon.
>
>
> Diffs
> -----
>
> src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
> src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
> src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
>
> Diff: https://reviews.apache.org/r/8255/diff/
>
>
> Testing
> -------
>
> udo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cgroup*
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Vinod Kone <vi...@gmail.com>.
> On Dec. 11, 2012, 11:22 p.m., Benjamin Hindman wrote:
> > src/slave/cgroups_isolation_module.cpp, line 701
> > <https://reviews.apache.org/r/8255/diff/3/?file=231632#file231632line701>
> >
> > Can we make this an 'operator << (std::ostream&, const list<CgroupInfo*>& infos)' and then do 'stringify(infos)' where ever we call 'cpusetsState()'?
done. had to move the operator overload inside the class definition, to circumvent the compiler warning about CgroupInfo being private (despite making the operator a friend).
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/#review14324
-----------------------------------------------------------
On Nov. 29, 2012, 7:09 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8255/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2012, 7:09 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Tested by modifying balloon framework test to start the slave cpusets enabled.
>
> Will be writing more tests soon.
>
>
> Diffs
> -----
>
> src/Makefile.am 8dbb799c6203b2860e29860dcadc152c7d29755b
> src/examples/cpuset_executor.cpp PRE-CREATION
> src/examples/cpuset_framework.cpp PRE-CREATION
> src/master/master.hpp 9b9b550adbc501f90ab280da74f2ba9852c19ff7
> src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
> src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
> src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
> src/tests/cgroups_isolation_tests.cpp 2685e22eeaf0b764b84e2a6e33e73f75db0515b6
> src/tests/cpuset_framework_test.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8255/diff/
>
>
> Testing
> -------
>
> [vinod@smfd-aki-27-sr1:~/mesos/build] $ sudo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cpuset*
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.089644 52201 process.cpp:1404] libprocess is initialized on 10.35.12.124:36552 for 16 cpus
> I1129 09:53:40.090000 52201 logging.cpp:179] Logging to STDERR
> Source directory: /home/vinod/mesos
> Build directory: /home/vinod/mesos/build
> Note: Google Test filter = *Cpuset*
> [==========] Running 6 tests from 2 test cases.
> [----------] Global test environment set-up.
> [----------] 1 test from CgroupsCpusetIsolationTest
> [ RUN ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework
> Using temporary directory '/tmp/CgroupsCpusetIsolationTest_ROOT_CGROUPS_CpusetFramework_vgnCzO'
> Launched master at 52261
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.140386 52261 process.cpp:1404] libprocess is initialized on 10.35.12.124:5432 for 16 cpus
> I1129 09:53:40.140688 52261 logging.cpp:179] Logging to STDERR
> I1129 09:53:40.140991 52261 main.cpp:116] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:40.141032 52261 main.cpp:117] Starting Mesos master
> I1129 09:53:40.141351 52283 master.cpp:306] Master started on 10.35.12.124:5432
> I1129 09:53:40.141427 52283 master.cpp:321] Master ID: 201211290953-2081170186-5432-52261
> W1129 09:53:40.141677 52282 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:40.141690 52288 hierarchical_allocator_process.hpp:234] Initializing hierarchical allocator process with master : master@10.35.12.124:5432
> I1129 09:53:40.142582 52283 master.cpp:559] Elected as master!
> I1129 09:53:41.143687 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:41.143728 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 50.20us
> Launched slave at 52293
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:42.142441 52293 process.cpp:1404] libprocess is initialized on 10.35.12.124:58922 for 16 cpus
> I1129 09:53:42.142765 52293 logging.cpp:179] Logging to STDERR
> I1129 09:53:42.143033 52293 main.cpp:122] Creating "cgroups" isolation module
> I1129 09:53:42.143414 52293 main.cpp:130] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:42.143451 52293 main.cpp:131] Starting Mesos slave
> I1129 09:53:42.143728 52309 slave.cpp:166] Slave started on 1)@10.35.12.124:58922
> I1129 09:53:42.143776 52309 slave.cpp:167] Slave resources: cpus=2; mem=1024
> I1129 09:53:42.144351 52321 cgroups_isolation_module.cpp:219] Using /cgroup as cgroups hierarchy root
> I1129 09:53:42.144350 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.144395 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 52.87us
> I1129 09:53:42.145091 52309 slave.cpp:368] New master detected at master@127.0.0.1:5432
> I1129 09:53:42.145892 52278 master.cpp:924] Attempting to register slave on smfd-aki-27-sr1.devel.twitter.com at slave(1)@10.35.12.124:58922
> I1129 09:53:42.145926 52278 master.cpp:1159] Master now considering a slave at smfd-aki-27-sr1.devel.twitter.com:58922 as active
> I1129 09:53:42.145949 52278 master.cpp:1729] Adding slave 201211290953-2081170186-5432-52261-0 at smfd-aki-27-sr1.devel.twitter.com with cpus=2; mem=1024
> I1129 09:53:42.146421 52278 hierarchical_allocator_process.hpp:393] Added slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) with cpus=2; mem=1024 (and cpus=2; mem=1024 available)
> I1129 09:53:42.146469 52278 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.146488 52278 hierarchical_allocator_process.hpp:616] Performed allocation for slave 201211290953-2081170186-5432-52261-0 in 19.15us
> I1129 09:53:42.146875 52318 slave.cpp:388] Registered with master; given slave ID 201211290953-2081170186-5432-52261-0
> I1129 09:53:42.147235 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:42.161064 52321 cgroups_isolation_module.cpp:315] Removing orphaned cgroup 'mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b'
> I1129 09:53:42.163172 52310 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.163269 52310 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b after 1 attempts
> I1129 09:53:42.163727 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:0, core:0, socket:0)
> I1129 09:53:42.163769 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:1, core:1, socket:0)
> I1129 09:53:42.164930 52314 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.165045 52314 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.179460 52312 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:43.146617 52289 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:43.146682 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 73.24us
> I1129 09:53:44.147394 52291 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:44.147459 52291 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 80.86us
> Initialized scheduler with 4 tasks
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:44.266501 52325 process.cpp:1404] libprocess is initialized on 10.35.12.124:39959 for 16 cpus
> I1129 09:53:44.266835 52325 logging.cpp:179] Logging to STDERR
> I1129 09:53:44.268498 52348 sched.cpp:181] New master at master@127.0.0.1:5432
> I1129 09:53:44.269433 52289 master.cpp:602] Registering framework 201211290953-2081170186-5432-52261-0000 at scheduler(1)@10.35.12.124:39959
> I1129 09:53:44.269762 52289 hierarchical_allocator_process.hpp:266] Added framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269809 52289 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.269863 52289 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269948 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 151.08us
> I1129 09:53:44.269989 52349 sched.cpp:216] Framework registered with 201211290953-2081170186-5432-52261-0000
> Registered
> I1129 09:53:44.270150 52288 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.270256 52288 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.270597 52350 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 1
> Starting task 2
> I1129 09:53:44.271181 52283 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-0 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.271360 52283 master.hpp:285] Adding task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271396 52283 master.cpp:1580] Launching task 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271618 52283 master.hpp:285] Adding task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271663 52283 master.cpp:1580] Launching task 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271772 52283 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271889 52309 slave.cpp:477] Got assigned task 1 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.273299 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.273581 52309 slave.cpp:477] Got assigned task 2 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.275192 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.275202 52321 cgroups_isolation_module.cpp:441] Launching 1 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c
> I1129 09:53:44.275431 52309 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.275568 52310 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.277354 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.277768 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.277865 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.278280 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.278712 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.279321 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52361
> I1129 09:53:44.280704 52321 cgroups_isolation_module.cpp:441] Launching 2 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b
> I1129 09:53:44.293046 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.293432 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.293527 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.293939 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294385 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294860 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52364
> I1129 09:53:44.532757 52323 slave.cpp:775] Got registration for executor '1' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533000 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.533148 52323 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533500 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533584 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.534265 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.536386 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.536618 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.536713 52318 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.536906 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.537102 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.537147 52318 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537211 52318 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.537238 52289 master.hpp:297] Removing task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537382 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537376 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.537499 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.537716 52351 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537915 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.537931 52322 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538000 52318 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538142 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.538341 52319 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.539729 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.594710 52313 slave.cpp:775] Got registration for executor '2' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.594856 52312 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.595021 52313 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595425 52312 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595496 52312 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.596107 52312 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.598393 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598541 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.598624 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.598870 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598999 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599064 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.599083 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.599196 52280 master.hpp:297] Removing task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599321 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599367 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.599483 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.599690 52347 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599812 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599819 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.599839 52323 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.600172 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.600360 52321 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.623421 52324 process.cpp:872] Socket closed while receiving
> W1129 09:53:45.143110 52280 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:45.147429 52308 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147513 52308 cgroups_isolation_module.cpp:543] Killing executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147531 52314 slave.cpp:1024] Executor '1' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:45.148080 52321 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c for removal
> I1129 09:53:45.148236 52321 gc.cpp:125] Scheduling GC removal event to fire after 7.00days
> I1129 09:53:45.148641 52308 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.148715 52308 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:45.148777 52308 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.148810 52308 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149700 52309 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149817 52309 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc after 1 attempts
> I1129 09:53:45.151449 52317 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.151527 52317 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.147889 52276 hierarchical_allocator_process.hpp:665] No resources available to allocate!
> I1129 09:53:45.167976 52276 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 20.11ms
> I1129 09:53:45.147966 52279 master.cpp:1123] Executor 1 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:45.168719 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.174438 52310 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:46.149343 52309 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149416 52309 cgroups_isolation_module.cpp:543] Killing executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149461 52311 slave.cpp:1024] Executor '2' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:46.149678 52311 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b for removal
> I1129 09:53:46.149778 52279 master.cpp:1123] Executor 2 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:46.149931 52283 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150512 52309 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150578 52309 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:46.150712 52309 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.150753 52309 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151849 52319 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151981 52319 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789 after 1 attempts
> I1129 09:53:46.153518 52309 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.153612 52309 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.168972 52284 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169070 52284 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169142 52284 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 192.05us
> I1129 09:53:46.169292 52284 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169360 52284 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169692 52353 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 3
> Starting task 4
> I1129 09:53:46.170192 52288 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-1 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170331 52288 master.hpp:285] Adding task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170371 52288 master.cpp:1580] Launching task 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170524 52288 master.hpp:285] Adding task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170580 52288 master.cpp:1580] Launching task 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170658 52314 slave.cpp:477] Got assigned task 3 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170697 52288 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.171835 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.172061 52314 slave.cpp:477] Got assigned task 4 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.173218 52323 cgroups_isolation_module.cpp:441] Launching 3 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_3_tag_9d1040aa-d38a-4702-971d-41d88052555c
> I1129 09:53:46.173284 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098
> I1129 09:53:46.173493 52314 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.173651 52313 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> I1129 09:53:46.175964 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.176360 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.176440 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.176902 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177340 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177829 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52439
> I1129 09:53:46.179039 52323 cgroups_isolation_module.cpp:441] Launching 4 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_4_tag_aee80a43-b55a-4d91-9885-0be61dd1da88
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36
> I1129 09:53:46.193050 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.193454 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.193539 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.194010 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194481 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194953 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52442
> I1129 09:53:46.195163 52323 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.242967 52310 slave.cpp:775] Got registration for executor '3' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243160 52322 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.243348 52310 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243744 52322 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243871 52322 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.244596 52322 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247010 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247195 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247309 52320 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.247477 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247652 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247760 52320 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247808 52291 master.hpp:297] Removing task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.247818 52320 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.247889 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.247928 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247969 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.248142 52354 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.248278 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248327 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.248522 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248623 52320 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264420 52319 slave.cpp:775] Got registration for executor '4' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264575 52316 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.264660 52319 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265015 52316 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265108 52316 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.265822 52316 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.266053 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.266679 52314 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267032 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267441 52355 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.267706 52310 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.267735 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.267887 52311 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.267972 52310 slave.cpp:752] Got acknowledgement of status update for task 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268178 52281 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.268337 52281 master.hpp:297] Removing task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268324 52311 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268412 52311 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.268451 52281 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268563 52352 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.268631 52352 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268764 52352 sched.cpp:421] Stopping framework '201211290953-2081170186-5432-52261-0000'
> I1129 09:53:46.268933 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.269044 52311 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269086 52289 master.cpp:730] Asked to unregister framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269214 52291 hierarchical_allocator_process.hpp:357] Deactivated framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269321 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269405 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269484 52280 hierarchical_allocator_process.hpp:310] Removed framework 201211290953-2081170186-5432-52261-0000
> W1129 09:53:46.269382 52320 slave.cpp:671] Ignoring shutdown framework message from master@10.35.12.124:5432because it is not from the registered master (master@127.0.0.1:5432)
> I1129 09:53:46.270210 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.270334 52292 process.cpp:872] Socket closed while receiving
> /home/vinod/mesos/support/atexit.sh: line 11: 52261 Terminated ${MESOS_BUILD_DIR}/src/mesos-master --port=5432
> /home/vinod/mesos/support/atexit.sh: line 11: 52293 Terminated ${MESOS_BUILD_DIR}/src/mesos-slave --master=localhost:5432 --isolation=cgroups --cgroups_hierarchy_root=${HIERARCHY} --cgroups_subsystems="cpuset,memory,freezer" --resources="cpus:2;mem:1024"
> [ OK ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework (6215 ms)
> [----------] 1 test from CgroupsCpusetIsolationTest (6215 ms total)
>
> [----------] 5 tests from CgroupsCpusetTest
> [ RUN ] CgroupsCpusetTest.OneCPUOneCpuset
> [ OK ] CgroupsCpusetTest.OneCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.OneCPUManyCpusets
> [ OK ] CgroupsCpusetTest.OneCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUOneCpuset
> [ OK ] CgroupsCpusetTest.ManyCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUManyCpusets
> [ OK ] CgroupsCpusetTest.ManyCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.IntegerAllocations
> [ OK ] CgroupsCpusetTest.IntegerAllocations (0 ms)
> [----------] 5 tests from CgroupsCpusetTest (0 ms total)
>
> [----------] Global test environment tear-down
> [==========] 6 tests from 2 test cases ran. (6216 ms total)
> [ PASSED ] 6 tests.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/#review14324
-----------------------------------------------------------
src/examples/cpuset_executor.cpp
<https://reviews.apache.org/r/8255/#comment30498>
Why this exit(0)?
src/examples/cpuset_framework.cpp
<https://reviews.apache.org/r/8255/#comment30499>
Kill newline.
src/examples/cpuset_framework.cpp
<https://reviews.apache.org/r/8255/#comment30500>
Never use this variable, might not even need to pass it in (can just pass in numTasks).
src/slave/cgroups_isolation_module.cpp
<https://reviews.apache.org/r/8255/#comment30502>
Can we make this an 'operator << (std::ostream&, const list<CgroupInfo*>& infos)' and then do 'stringify(infos)' where ever we call 'cpusetsState()'?
src/slave/cgroups_isolation_module.cpp
<https://reviews.apache.org/r/8255/#comment30501>
Consider foreachkey, foreachvalue as discussed offline (rather than needing the typedef).
- Benjamin Hindman
On Nov. 29, 2012, 7:09 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8255/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2012, 7:09 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Tested by modifying balloon framework test to start the slave cpusets enabled.
>
> Will be writing more tests soon.
>
>
> Diffs
> -----
>
> src/Makefile.am 8dbb799c6203b2860e29860dcadc152c7d29755b
> src/examples/cpuset_executor.cpp PRE-CREATION
> src/examples/cpuset_framework.cpp PRE-CREATION
> src/master/master.hpp 9b9b550adbc501f90ab280da74f2ba9852c19ff7
> src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
> src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
> src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
> src/tests/cgroups_isolation_tests.cpp 2685e22eeaf0b764b84e2a6e33e73f75db0515b6
> src/tests/cpuset_framework_test.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8255/diff/
>
>
> Testing
> -------
>
> [vinod@smfd-aki-27-sr1:~/mesos/build] $ sudo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cpuset*
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.089644 52201 process.cpp:1404] libprocess is initialized on 10.35.12.124:36552 for 16 cpus
> I1129 09:53:40.090000 52201 logging.cpp:179] Logging to STDERR
> Source directory: /home/vinod/mesos
> Build directory: /home/vinod/mesos/build
> Note: Google Test filter = *Cpuset*
> [==========] Running 6 tests from 2 test cases.
> [----------] Global test environment set-up.
> [----------] 1 test from CgroupsCpusetIsolationTest
> [ RUN ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework
> Using temporary directory '/tmp/CgroupsCpusetIsolationTest_ROOT_CGROUPS_CpusetFramework_vgnCzO'
> Launched master at 52261
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.140386 52261 process.cpp:1404] libprocess is initialized on 10.35.12.124:5432 for 16 cpus
> I1129 09:53:40.140688 52261 logging.cpp:179] Logging to STDERR
> I1129 09:53:40.140991 52261 main.cpp:116] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:40.141032 52261 main.cpp:117] Starting Mesos master
> I1129 09:53:40.141351 52283 master.cpp:306] Master started on 10.35.12.124:5432
> I1129 09:53:40.141427 52283 master.cpp:321] Master ID: 201211290953-2081170186-5432-52261
> W1129 09:53:40.141677 52282 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:40.141690 52288 hierarchical_allocator_process.hpp:234] Initializing hierarchical allocator process with master : master@10.35.12.124:5432
> I1129 09:53:40.142582 52283 master.cpp:559] Elected as master!
> I1129 09:53:41.143687 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:41.143728 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 50.20us
> Launched slave at 52293
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:42.142441 52293 process.cpp:1404] libprocess is initialized on 10.35.12.124:58922 for 16 cpus
> I1129 09:53:42.142765 52293 logging.cpp:179] Logging to STDERR
> I1129 09:53:42.143033 52293 main.cpp:122] Creating "cgroups" isolation module
> I1129 09:53:42.143414 52293 main.cpp:130] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:42.143451 52293 main.cpp:131] Starting Mesos slave
> I1129 09:53:42.143728 52309 slave.cpp:166] Slave started on 1)@10.35.12.124:58922
> I1129 09:53:42.143776 52309 slave.cpp:167] Slave resources: cpus=2; mem=1024
> I1129 09:53:42.144351 52321 cgroups_isolation_module.cpp:219] Using /cgroup as cgroups hierarchy root
> I1129 09:53:42.144350 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.144395 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 52.87us
> I1129 09:53:42.145091 52309 slave.cpp:368] New master detected at master@127.0.0.1:5432
> I1129 09:53:42.145892 52278 master.cpp:924] Attempting to register slave on smfd-aki-27-sr1.devel.twitter.com at slave(1)@10.35.12.124:58922
> I1129 09:53:42.145926 52278 master.cpp:1159] Master now considering a slave at smfd-aki-27-sr1.devel.twitter.com:58922 as active
> I1129 09:53:42.145949 52278 master.cpp:1729] Adding slave 201211290953-2081170186-5432-52261-0 at smfd-aki-27-sr1.devel.twitter.com with cpus=2; mem=1024
> I1129 09:53:42.146421 52278 hierarchical_allocator_process.hpp:393] Added slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) with cpus=2; mem=1024 (and cpus=2; mem=1024 available)
> I1129 09:53:42.146469 52278 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.146488 52278 hierarchical_allocator_process.hpp:616] Performed allocation for slave 201211290953-2081170186-5432-52261-0 in 19.15us
> I1129 09:53:42.146875 52318 slave.cpp:388] Registered with master; given slave ID 201211290953-2081170186-5432-52261-0
> I1129 09:53:42.147235 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:42.161064 52321 cgroups_isolation_module.cpp:315] Removing orphaned cgroup 'mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b'
> I1129 09:53:42.163172 52310 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.163269 52310 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b after 1 attempts
> I1129 09:53:42.163727 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:0, core:0, socket:0)
> I1129 09:53:42.163769 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:1, core:1, socket:0)
> I1129 09:53:42.164930 52314 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.165045 52314 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.179460 52312 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:43.146617 52289 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:43.146682 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 73.24us
> I1129 09:53:44.147394 52291 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:44.147459 52291 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 80.86us
> Initialized scheduler with 4 tasks
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:44.266501 52325 process.cpp:1404] libprocess is initialized on 10.35.12.124:39959 for 16 cpus
> I1129 09:53:44.266835 52325 logging.cpp:179] Logging to STDERR
> I1129 09:53:44.268498 52348 sched.cpp:181] New master at master@127.0.0.1:5432
> I1129 09:53:44.269433 52289 master.cpp:602] Registering framework 201211290953-2081170186-5432-52261-0000 at scheduler(1)@10.35.12.124:39959
> I1129 09:53:44.269762 52289 hierarchical_allocator_process.hpp:266] Added framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269809 52289 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.269863 52289 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269948 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 151.08us
> I1129 09:53:44.269989 52349 sched.cpp:216] Framework registered with 201211290953-2081170186-5432-52261-0000
> Registered
> I1129 09:53:44.270150 52288 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.270256 52288 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.270597 52350 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 1
> Starting task 2
> I1129 09:53:44.271181 52283 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-0 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.271360 52283 master.hpp:285] Adding task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271396 52283 master.cpp:1580] Launching task 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271618 52283 master.hpp:285] Adding task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271663 52283 master.cpp:1580] Launching task 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271772 52283 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271889 52309 slave.cpp:477] Got assigned task 1 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.273299 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.273581 52309 slave.cpp:477] Got assigned task 2 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.275192 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.275202 52321 cgroups_isolation_module.cpp:441] Launching 1 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c
> I1129 09:53:44.275431 52309 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.275568 52310 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.277354 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.277768 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.277865 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.278280 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.278712 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.279321 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52361
> I1129 09:53:44.280704 52321 cgroups_isolation_module.cpp:441] Launching 2 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b
> I1129 09:53:44.293046 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.293432 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.293527 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.293939 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294385 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294860 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52364
> I1129 09:53:44.532757 52323 slave.cpp:775] Got registration for executor '1' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533000 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.533148 52323 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533500 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533584 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.534265 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.536386 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.536618 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.536713 52318 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.536906 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.537102 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.537147 52318 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537211 52318 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.537238 52289 master.hpp:297] Removing task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537382 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537376 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.537499 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.537716 52351 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537915 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.537931 52322 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538000 52318 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538142 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.538341 52319 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.539729 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.594710 52313 slave.cpp:775] Got registration for executor '2' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.594856 52312 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.595021 52313 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595425 52312 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595496 52312 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.596107 52312 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.598393 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598541 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.598624 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.598870 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598999 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599064 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.599083 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.599196 52280 master.hpp:297] Removing task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599321 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599367 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.599483 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.599690 52347 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599812 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599819 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.599839 52323 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.600172 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.600360 52321 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.623421 52324 process.cpp:872] Socket closed while receiving
> W1129 09:53:45.143110 52280 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:45.147429 52308 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147513 52308 cgroups_isolation_module.cpp:543] Killing executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147531 52314 slave.cpp:1024] Executor '1' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:45.148080 52321 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c for removal
> I1129 09:53:45.148236 52321 gc.cpp:125] Scheduling GC removal event to fire after 7.00days
> I1129 09:53:45.148641 52308 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.148715 52308 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:45.148777 52308 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.148810 52308 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149700 52309 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149817 52309 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc after 1 attempts
> I1129 09:53:45.151449 52317 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.151527 52317 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.147889 52276 hierarchical_allocator_process.hpp:665] No resources available to allocate!
> I1129 09:53:45.167976 52276 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 20.11ms
> I1129 09:53:45.147966 52279 master.cpp:1123] Executor 1 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:45.168719 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.174438 52310 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:46.149343 52309 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149416 52309 cgroups_isolation_module.cpp:543] Killing executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149461 52311 slave.cpp:1024] Executor '2' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:46.149678 52311 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b for removal
> I1129 09:53:46.149778 52279 master.cpp:1123] Executor 2 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:46.149931 52283 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150512 52309 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150578 52309 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:46.150712 52309 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.150753 52309 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151849 52319 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151981 52319 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789 after 1 attempts
> I1129 09:53:46.153518 52309 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.153612 52309 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.168972 52284 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169070 52284 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169142 52284 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 192.05us
> I1129 09:53:46.169292 52284 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169360 52284 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169692 52353 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 3
> Starting task 4
> I1129 09:53:46.170192 52288 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-1 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170331 52288 master.hpp:285] Adding task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170371 52288 master.cpp:1580] Launching task 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170524 52288 master.hpp:285] Adding task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170580 52288 master.cpp:1580] Launching task 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170658 52314 slave.cpp:477] Got assigned task 3 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170697 52288 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.171835 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.172061 52314 slave.cpp:477] Got assigned task 4 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.173218 52323 cgroups_isolation_module.cpp:441] Launching 3 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_3_tag_9d1040aa-d38a-4702-971d-41d88052555c
> I1129 09:53:46.173284 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098
> I1129 09:53:46.173493 52314 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.173651 52313 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> I1129 09:53:46.175964 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.176360 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.176440 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.176902 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177340 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177829 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52439
> I1129 09:53:46.179039 52323 cgroups_isolation_module.cpp:441] Launching 4 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_4_tag_aee80a43-b55a-4d91-9885-0be61dd1da88
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36
> I1129 09:53:46.193050 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.193454 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.193539 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.194010 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194481 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194953 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52442
> I1129 09:53:46.195163 52323 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.242967 52310 slave.cpp:775] Got registration for executor '3' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243160 52322 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.243348 52310 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243744 52322 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243871 52322 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.244596 52322 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247010 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247195 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247309 52320 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.247477 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247652 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247760 52320 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247808 52291 master.hpp:297] Removing task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.247818 52320 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.247889 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.247928 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247969 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.248142 52354 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.248278 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248327 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.248522 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248623 52320 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264420 52319 slave.cpp:775] Got registration for executor '4' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264575 52316 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.264660 52319 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265015 52316 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265108 52316 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.265822 52316 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.266053 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.266679 52314 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267032 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267441 52355 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.267706 52310 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.267735 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.267887 52311 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.267972 52310 slave.cpp:752] Got acknowledgement of status update for task 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268178 52281 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.268337 52281 master.hpp:297] Removing task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268324 52311 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268412 52311 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.268451 52281 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268563 52352 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.268631 52352 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268764 52352 sched.cpp:421] Stopping framework '201211290953-2081170186-5432-52261-0000'
> I1129 09:53:46.268933 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.269044 52311 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269086 52289 master.cpp:730] Asked to unregister framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269214 52291 hierarchical_allocator_process.hpp:357] Deactivated framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269321 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269405 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269484 52280 hierarchical_allocator_process.hpp:310] Removed framework 201211290953-2081170186-5432-52261-0000
> W1129 09:53:46.269382 52320 slave.cpp:671] Ignoring shutdown framework message from master@10.35.12.124:5432because it is not from the registered master (master@127.0.0.1:5432)
> I1129 09:53:46.270210 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.270334 52292 process.cpp:872] Socket closed while receiving
> /home/vinod/mesos/support/atexit.sh: line 11: 52261 Terminated ${MESOS_BUILD_DIR}/src/mesos-master --port=5432
> /home/vinod/mesos/support/atexit.sh: line 11: 52293 Terminated ${MESOS_BUILD_DIR}/src/mesos-slave --master=localhost:5432 --isolation=cgroups --cgroups_hierarchy_root=${HIERARCHY} --cgroups_subsystems="cpuset,memory,freezer" --resources="cpus:2;mem:1024"
> [ OK ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework (6215 ms)
> [----------] 1 test from CgroupsCpusetIsolationTest (6215 ms total)
>
> [----------] 5 tests from CgroupsCpusetTest
> [ RUN ] CgroupsCpusetTest.OneCPUOneCpuset
> [ OK ] CgroupsCpusetTest.OneCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.OneCPUManyCpusets
> [ OK ] CgroupsCpusetTest.OneCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUOneCpuset
> [ OK ] CgroupsCpusetTest.ManyCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUManyCpusets
> [ OK ] CgroupsCpusetTest.ManyCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.IntegerAllocations
> [ OK ] CgroupsCpusetTest.IntegerAllocations (0 ms)
> [----------] 5 tests from CgroupsCpusetTest (0 ms total)
>
> [----------] Global test environment tear-down
> [==========] 6 tests from 2 test cases ran. (6216 ms total)
> [ PASSED ] 6 tests.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/
-----------------------------------------------------------
(Updated Jan. 3, 2013, 11:28 p.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
benh's comments and rebased off trunk
Description
-------
Tested by modifying balloon framework test to start the slave cpusets enabled.
Will be writing more tests soon.
Diffs (updated)
-----
src/Makefile.am c94736df660a25b58dc47c07d9c56c3c26152a66
src/examples/balloon_framework.cpp 35bc0fe712edbf6130e10e93286c5278625be096
src/examples/cpuset_executor.cpp PRE-CREATION
src/examples/cpuset_framework.cpp PRE-CREATION
src/examples/java/TestFramework.java 8417394487a80b439e7d9897c83f0b2c1eb17ff4
src/examples/python/test_framework.py f436865b32c555a593e7cbb2c35b8cdae439932a
src/examples/test_framework.cpp b9ab692414e4df64f176fc1ecd05f24ff089bde0
src/slave/cgroups_isolation_module.hpp 669efa14ba2603764aa68ae19a44e79dbfdec192
src/slave/cgroups_isolation_module.cpp 0f2975d1adf874dba4d0a539eb5c99233cef6e6b
src/slave/constants.hpp ddf02570caf3793106b3c48e158a5bb48c1ae80c
src/slave/constants.cpp 1735a6b55a93e6537a5a119e5345961f3d84a000
src/slave/slave.hpp e9f7b659ca2860501840b3d01e69915ebd162039
src/slave/slave.cpp 9755b46f97173d6fcc9ab1fd63e0e4814b3bc018
src/tests/allocator_tests.cpp 44d72ad2b746b1b7032f8243817a6002100a7503
src/tests/allocator_zookeeper_tests.cpp 5f83dd71aa3cac1dd3727b65af087f651f961392
src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
src/tests/cgroups_isolation_tests.cpp 41d6ac9d6331bcfd1e57d403e13a1d27cd5d14bc
src/tests/cpuset_framework_test.sh PRE-CREATION
src/tests/fault_tolerance_tests.cpp 7c3d155479a4f7fda5c14f62909eea5f5a1a92df
src/tests/master_tests.cpp 948ab5dff34eeba1f3ce593a864ddf282c8b19ed
src/tests/resource_offers_tests.cpp 5981191597e673c47be331420b71be66456694c6
src/tests/utils.hpp be457117515ee727af101370b26bf9188afb8f45
Diff: https://reviews.apache.org/r/8255/diff/
Testing
-------
[vinod@smfd-aki-27-sr1:~/mesos/build] $ sudo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cpuset*
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:40.089644 52201 process.cpp:1404] libprocess is initialized on 10.35.12.124:36552 for 16 cpus
I1129 09:53:40.090000 52201 logging.cpp:179] Logging to STDERR
Source directory: /home/vinod/mesos
Build directory: /home/vinod/mesos/build
Note: Google Test filter = *Cpuset*
[==========] Running 6 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from CgroupsCpusetIsolationTest
[ RUN ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework
Using temporary directory '/tmp/CgroupsCpusetIsolationTest_ROOT_CGROUPS_CpusetFramework_vgnCzO'
Launched master at 52261
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:40.140386 52261 process.cpp:1404] libprocess is initialized on 10.35.12.124:5432 for 16 cpus
I1129 09:53:40.140688 52261 logging.cpp:179] Logging to STDERR
I1129 09:53:40.140991 52261 main.cpp:116] Build: 2012-11-02 23:22:34 by vinod
I1129 09:53:40.141032 52261 main.cpp:117] Starting Mesos master
I1129 09:53:40.141351 52283 master.cpp:306] Master started on 10.35.12.124:5432
I1129 09:53:40.141427 52283 master.cpp:321] Master ID: 201211290953-2081170186-5432-52261
W1129 09:53:40.141677 52282 master.cpp:79] No whitelist given. Advertising offers for all slaves
I1129 09:53:40.141690 52288 hierarchical_allocator_process.hpp:234] Initializing hierarchical allocator process with master : master@10.35.12.124:5432
I1129 09:53:40.142582 52283 master.cpp:559] Elected as master!
I1129 09:53:41.143687 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:41.143728 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 50.20us
Launched slave at 52293
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:42.142441 52293 process.cpp:1404] libprocess is initialized on 10.35.12.124:58922 for 16 cpus
I1129 09:53:42.142765 52293 logging.cpp:179] Logging to STDERR
I1129 09:53:42.143033 52293 main.cpp:122] Creating "cgroups" isolation module
I1129 09:53:42.143414 52293 main.cpp:130] Build: 2012-11-02 23:22:34 by vinod
I1129 09:53:42.143451 52293 main.cpp:131] Starting Mesos slave
I1129 09:53:42.143728 52309 slave.cpp:166] Slave started on 1)@10.35.12.124:58922
I1129 09:53:42.143776 52309 slave.cpp:167] Slave resources: cpus=2; mem=1024
I1129 09:53:42.144351 52321 cgroups_isolation_module.cpp:219] Using /cgroup as cgroups hierarchy root
I1129 09:53:42.144350 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:42.144395 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 52.87us
I1129 09:53:42.145091 52309 slave.cpp:368] New master detected at master@127.0.0.1:5432
I1129 09:53:42.145892 52278 master.cpp:924] Attempting to register slave on smfd-aki-27-sr1.devel.twitter.com at slave(1)@10.35.12.124:58922
I1129 09:53:42.145926 52278 master.cpp:1159] Master now considering a slave at smfd-aki-27-sr1.devel.twitter.com:58922 as active
I1129 09:53:42.145949 52278 master.cpp:1729] Adding slave 201211290953-2081170186-5432-52261-0 at smfd-aki-27-sr1.devel.twitter.com with cpus=2; mem=1024
I1129 09:53:42.146421 52278 hierarchical_allocator_process.hpp:393] Added slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) with cpus=2; mem=1024 (and cpus=2; mem=1024 available)
I1129 09:53:42.146469 52278 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:42.146488 52278 hierarchical_allocator_process.hpp:616] Performed allocation for slave 201211290953-2081170186-5432-52261-0 in 19.15us
I1129 09:53:42.146875 52318 slave.cpp:388] Registered with master; given slave ID 201211290953-2081170186-5432-52261-0
I1129 09:53:42.147235 52292 process.cpp:872] Socket closed while receiving
I1129 09:53:42.161064 52321 cgroups_isolation_module.cpp:315] Removing orphaned cgroup 'mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b'
I1129 09:53:42.163172 52310 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:42.163269 52310 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b after 1 attempts
I1129 09:53:42.163727 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:0, core:0, socket:0)
I1129 09:53:42.163769 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:1, core:1, socket:0)
I1129 09:53:42.164930 52314 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:42.165045 52314 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:42.179460 52312 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:43.146617 52289 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:43.146682 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 73.24us
I1129 09:53:44.147394 52291 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:44.147459 52291 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 80.86us
Initialized scheduler with 4 tasks
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:44.266501 52325 process.cpp:1404] libprocess is initialized on 10.35.12.124:39959 for 16 cpus
I1129 09:53:44.266835 52325 logging.cpp:179] Logging to STDERR
I1129 09:53:44.268498 52348 sched.cpp:181] New master at master@127.0.0.1:5432
I1129 09:53:44.269433 52289 master.cpp:602] Registering framework 201211290953-2081170186-5432-52261-0000 at scheduler(1)@10.35.12.124:39959
I1129 09:53:44.269762 52289 hierarchical_allocator_process.hpp:266] Added framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.269809 52289 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.269863 52289 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.269948 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 151.08us
I1129 09:53:44.269989 52349 sched.cpp:216] Framework registered with 201211290953-2081170186-5432-52261-0000
Registered
I1129 09:53:44.270150 52288 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.270256 52288 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.270597 52350 sched.cpp:281] Received 1 offers
Resource offers received: 1
Starting task 1
Starting task 2
I1129 09:53:44.271181 52283 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-0 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.271360 52283 master.hpp:285] Adding task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.271396 52283 master.cpp:1580] Launching task 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:44.271618 52283 master.hpp:285] Adding task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.271663 52283 master.cpp:1580] Launching task 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:44.271772 52283 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.271889 52309 slave.cpp:477] Got assigned task 1 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.273299 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
I1129 09:53:44.273581 52309 slave.cpp:477] Got assigned task 2 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.275192 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
I1129 09:53:44.275202 52321 cgroups_isolation_module.cpp:441] Launching 1 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c
I1129 09:53:44.275431 52309 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
I1129 09:53:44.275568 52310 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
I1129 09:53:44.277354 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.277768 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.277865 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.278280 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.278712 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.279321 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52361
I1129 09:53:44.280704 52321 cgroups_isolation_module.cpp:441] Launching 2 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b
I1129 09:53:44.293046 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.293432 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.293527 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.293939 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.294385 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.294860 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52364
I1129 09:53:44.532757 52323 slave.cpp:775] Got registration for executor '1' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.533000 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:44.533148 52323 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.533500 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.533584 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.534265 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.536386 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.536618 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.536713 52318 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.536906 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.537102 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.537147 52318 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.537211 52318 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.537238 52289 master.hpp:297] Removing task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.537382 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.537376 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:44.537499 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:44.537716 52351 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.537915 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.537931 52322 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.538000 52318 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.538142 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.538341 52319 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.539729 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.594710 52313 slave.cpp:775] Got registration for executor '2' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.594856 52312 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:44.595021 52313 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.595425 52312 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.595496 52312 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.596107 52312 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.598393 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.598541 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.598624 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.598870 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.598999 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.599064 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.599083 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.599196 52280 master.hpp:297] Removing task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.599321 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.599367 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:44.599483 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:44.599690 52347 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.599812 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.599819 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.599839 52323 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.600172 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.600360 52321 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.623421 52324 process.cpp:872] Socket closed while receiving
W1129 09:53:45.143110 52280 master.cpp:79] No whitelist given. Advertising offers for all slaves
I1129 09:53:45.147429 52308 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.147513 52308 cgroups_isolation_module.cpp:543] Killing executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.147531 52314 slave.cpp:1024] Executor '1' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
I1129 09:53:45.148080 52321 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c for removal
I1129 09:53:45.148236 52321 gc.cpp:125] Scheduling GC removal event to fire after 7.00days
I1129 09:53:45.148641 52308 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.148715 52308 cgroups_isolation_module.cpp:959] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { }
******************************************************************
I1129 09:53:45.148777 52308 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.148810 52308 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.149700 52309 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.149817 52309 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc after 1 attempts
I1129 09:53:45.151449 52317 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.151527 52317 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.147889 52276 hierarchical_allocator_process.hpp:665] No resources available to allocate!
I1129 09:53:45.167976 52276 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 20.11ms
I1129 09:53:45.147966 52279 master.cpp:1123] Executor 1 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
I1129 09:53:45.168719 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.174438 52310 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:46.149343 52309 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.149416 52309 cgroups_isolation_module.cpp:543] Killing executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.149461 52311 slave.cpp:1024] Executor '2' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
I1129 09:53:46.149678 52311 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b for removal
I1129 09:53:46.149778 52279 master.cpp:1123] Executor 2 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
I1129 09:53:46.149931 52283 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.150512 52309 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.150578 52309 cgroups_isolation_module.cpp:959] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { }
******************************************************************
I1129 09:53:46.150712 52309 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.150753 52309 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.151849 52319 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.151981 52319 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789 after 1 attempts
I1129 09:53:46.153518 52309 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.153612 52309 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.168972 52284 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.169070 52284 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.169142 52284 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 192.05us
I1129 09:53:46.169292 52284 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.169360 52284 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.169692 52353 sched.cpp:281] Received 1 offers
Resource offers received: 1
Starting task 3
Starting task 4
I1129 09:53:46.170192 52288 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-1 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.170331 52288 master.hpp:285] Adding task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.170371 52288 master.cpp:1580] Launching task 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:46.170524 52288 master.hpp:285] Adding task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.170580 52288 master.cpp:1580] Launching task 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:46.170658 52314 slave.cpp:477] Got assigned task 3 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.170697 52288 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.171835 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
I1129 09:53:46.172061 52314 slave.cpp:477] Got assigned task 4 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.173218 52323 cgroups_isolation_module.cpp:441] Launching 3 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_3_tag_9d1040aa-d38a-4702-971d-41d88052555c
I1129 09:53:46.173284 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098
I1129 09:53:46.173493 52314 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
I1129 09:53:46.173651 52313 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
I1129 09:53:46.175964 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.176360 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.176440 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.176902 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.177340 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.177829 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52439
I1129 09:53:46.179039 52323 cgroups_isolation_module.cpp:441] Launching 4 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_4_tag_aee80a43-b55a-4d91-9885-0be61dd1da88
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36
I1129 09:53:46.193050 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.193454 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.193539 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.194010 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.194481 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.194953 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52442
I1129 09:53:46.195163 52323 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.242967 52310 slave.cpp:775] Got registration for executor '3' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.243160 52322 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:46.243348 52310 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.243744 52322 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.243871 52322 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.244596 52322 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.247010 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.247195 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.247309 52320 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.247477 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.247652 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.247760 52320 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.247808 52291 master.hpp:297] Removing task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.247818 52320 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.247889 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:46.247928 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.247969 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:46.248142 52354 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.248278 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.248327 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.248522 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.248623 52320 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.264420 52319 slave.cpp:775] Got registration for executor '4' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.264575 52316 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:46.264660 52319 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.265015 52316 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.265108 52316 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.265822 52316 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.266053 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.266679 52314 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.267032 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.267441 52355 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:46.267706 52310 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.267735 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.267887 52311 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.267972 52310 slave.cpp:752] Got acknowledgement of status update for task 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.268178 52281 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.268337 52281 master.hpp:297] Removing task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.268324 52311 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.268412 52311 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.268451 52281 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.268563 52352 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:46.268631 52352 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.268764 52352 sched.cpp:421] Stopping framework '201211290953-2081170186-5432-52261-0000'
I1129 09:53:46.268933 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.269044 52311 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269086 52289 master.cpp:730] Asked to unregister framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269214 52291 hierarchical_allocator_process.hpp:357] Deactivated framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269321 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269405 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269484 52280 hierarchical_allocator_process.hpp:310] Removed framework 201211290953-2081170186-5432-52261-0000
W1129 09:53:46.269382 52320 slave.cpp:671] Ignoring shutdown framework message from master@10.35.12.124:5432because it is not from the registered master (master@127.0.0.1:5432)
I1129 09:53:46.270210 52292 process.cpp:872] Socket closed while receiving
I1129 09:53:46.270334 52292 process.cpp:872] Socket closed while receiving
/home/vinod/mesos/support/atexit.sh: line 11: 52261 Terminated ${MESOS_BUILD_DIR}/src/mesos-master --port=5432
/home/vinod/mesos/support/atexit.sh: line 11: 52293 Terminated ${MESOS_BUILD_DIR}/src/mesos-slave --master=localhost:5432 --isolation=cgroups --cgroups_hierarchy_root=${HIERARCHY} --cgroups_subsystems="cpuset,memory,freezer" --resources="cpus:2;mem:1024"
[ OK ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework (6215 ms)
[----------] 1 test from CgroupsCpusetIsolationTest (6215 ms total)
[----------] 5 tests from CgroupsCpusetTest
[ RUN ] CgroupsCpusetTest.OneCPUOneCpuset
[ OK ] CgroupsCpusetTest.OneCPUOneCpuset (0 ms)
[ RUN ] CgroupsCpusetTest.OneCPUManyCpusets
[ OK ] CgroupsCpusetTest.OneCPUManyCpusets (0 ms)
[ RUN ] CgroupsCpusetTest.ManyCPUOneCpuset
[ OK ] CgroupsCpusetTest.ManyCPUOneCpuset (0 ms)
[ RUN ] CgroupsCpusetTest.ManyCPUManyCpusets
[ OK ] CgroupsCpusetTest.ManyCPUManyCpusets (0 ms)
[ RUN ] CgroupsCpusetTest.IntegerAllocations
[ OK ] CgroupsCpusetTest.IntegerAllocations (0 ms)
[----------] 5 tests from CgroupsCpusetTest (0 ms total)
[----------] Global test environment tear-down
[==========] 6 tests from 2 test cases ran. (6216 ms total)
[ PASSED ] 6 tests.
Thanks,
Vinod Kone
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Ben Mahler <be...@gmail.com>.
> On Nov. 30, 2012, 12:59 a.m., Ben Mahler wrote:
> > src/slave/cgroups_isolation_module.cpp, line 706
> > <https://reviews.apache.org/r/8255/diff/3/?file=231632#file231632line706>
> >
> > This might come out a little nicer: s/ \t /\t\t
>
> Vinod Kone wrote:
> are you basing this on current executor id and framework id lengths in my test output? because, we can't guarantee their lengths are going to look the same.
Fair enough, either way, can you use only tabs or only spaces?
> On Nov. 30, 2012, 12:59 a.m., Ben Mahler wrote:
> > src/slave/cgroups_isolation_module.cpp, line 943
> > <https://reviews.apache.org/r/8255/diff/3/?file=231632#file231632line943>
> >
> > What are the semantics of destroyWaited()? And is the window safe to have?
>
> Vinod Kone wrote:
> it is just the asynchronous callback that gets called when the cgroup gets destroyed/failed.
>
> i THINK it is safe, but can't say for sure without looking at how it manifests in reality. my assumption is based on the fact that the old cgroup that is being in the process of getting destroyed wouldn't be burning cpus. from what we have seen so far, the reason this delay is non-trivial is because some processes get stuck in a weird state (S? D? blocking on kernel code?). but i don't seem to recall seeing processes in running state. BenH/Toby can correct me?
>
> Nonetheless, the right thing to do here, that the isolation module shouldn't inform the slave (and hence the master) about the terminated executor until 'destroyWaited' is called.
> We don't do that currently because we (I?) are not sure how to get the exit status of the executor, which we need to send as part of 'executorTerminated' message. There is already a TODO for this in the code. Once we do this, it doesn't matter whether the isolation module does the deallocation in unregisterCgroupsInfo() or destroyWaited() because the master wouldn't offer those resources until destroyWaited() is called.
>
> Makes sense?
Alright, that helps, but I still don't quite get the overall flow here, I'll just ask you in person later.
- Ben
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/#review13878
-----------------------------------------------------------
On Nov. 29, 2012, 7:09 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8255/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2012, 7:09 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Tested by modifying balloon framework test to start the slave cpusets enabled.
>
> Will be writing more tests soon.
>
>
> Diffs
> -----
>
> src/Makefile.am 8dbb799c6203b2860e29860dcadc152c7d29755b
> src/examples/cpuset_executor.cpp PRE-CREATION
> src/examples/cpuset_framework.cpp PRE-CREATION
> src/master/master.hpp 9b9b550adbc501f90ab280da74f2ba9852c19ff7
> src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
> src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
> src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
> src/tests/cgroups_isolation_tests.cpp 2685e22eeaf0b764b84e2a6e33e73f75db0515b6
> src/tests/cpuset_framework_test.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8255/diff/
>
>
> Testing
> -------
>
> [vinod@smfd-aki-27-sr1:~/mesos/build] $ sudo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cpuset*
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.089644 52201 process.cpp:1404] libprocess is initialized on 10.35.12.124:36552 for 16 cpus
> I1129 09:53:40.090000 52201 logging.cpp:179] Logging to STDERR
> Source directory: /home/vinod/mesos
> Build directory: /home/vinod/mesos/build
> Note: Google Test filter = *Cpuset*
> [==========] Running 6 tests from 2 test cases.
> [----------] Global test environment set-up.
> [----------] 1 test from CgroupsCpusetIsolationTest
> [ RUN ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework
> Using temporary directory '/tmp/CgroupsCpusetIsolationTest_ROOT_CGROUPS_CpusetFramework_vgnCzO'
> Launched master at 52261
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.140386 52261 process.cpp:1404] libprocess is initialized on 10.35.12.124:5432 for 16 cpus
> I1129 09:53:40.140688 52261 logging.cpp:179] Logging to STDERR
> I1129 09:53:40.140991 52261 main.cpp:116] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:40.141032 52261 main.cpp:117] Starting Mesos master
> I1129 09:53:40.141351 52283 master.cpp:306] Master started on 10.35.12.124:5432
> I1129 09:53:40.141427 52283 master.cpp:321] Master ID: 201211290953-2081170186-5432-52261
> W1129 09:53:40.141677 52282 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:40.141690 52288 hierarchical_allocator_process.hpp:234] Initializing hierarchical allocator process with master : master@10.35.12.124:5432
> I1129 09:53:40.142582 52283 master.cpp:559] Elected as master!
> I1129 09:53:41.143687 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:41.143728 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 50.20us
> Launched slave at 52293
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:42.142441 52293 process.cpp:1404] libprocess is initialized on 10.35.12.124:58922 for 16 cpus
> I1129 09:53:42.142765 52293 logging.cpp:179] Logging to STDERR
> I1129 09:53:42.143033 52293 main.cpp:122] Creating "cgroups" isolation module
> I1129 09:53:42.143414 52293 main.cpp:130] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:42.143451 52293 main.cpp:131] Starting Mesos slave
> I1129 09:53:42.143728 52309 slave.cpp:166] Slave started on 1)@10.35.12.124:58922
> I1129 09:53:42.143776 52309 slave.cpp:167] Slave resources: cpus=2; mem=1024
> I1129 09:53:42.144351 52321 cgroups_isolation_module.cpp:219] Using /cgroup as cgroups hierarchy root
> I1129 09:53:42.144350 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.144395 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 52.87us
> I1129 09:53:42.145091 52309 slave.cpp:368] New master detected at master@127.0.0.1:5432
> I1129 09:53:42.145892 52278 master.cpp:924] Attempting to register slave on smfd-aki-27-sr1.devel.twitter.com at slave(1)@10.35.12.124:58922
> I1129 09:53:42.145926 52278 master.cpp:1159] Master now considering a slave at smfd-aki-27-sr1.devel.twitter.com:58922 as active
> I1129 09:53:42.145949 52278 master.cpp:1729] Adding slave 201211290953-2081170186-5432-52261-0 at smfd-aki-27-sr1.devel.twitter.com with cpus=2; mem=1024
> I1129 09:53:42.146421 52278 hierarchical_allocator_process.hpp:393] Added slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) with cpus=2; mem=1024 (and cpus=2; mem=1024 available)
> I1129 09:53:42.146469 52278 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.146488 52278 hierarchical_allocator_process.hpp:616] Performed allocation for slave 201211290953-2081170186-5432-52261-0 in 19.15us
> I1129 09:53:42.146875 52318 slave.cpp:388] Registered with master; given slave ID 201211290953-2081170186-5432-52261-0
> I1129 09:53:42.147235 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:42.161064 52321 cgroups_isolation_module.cpp:315] Removing orphaned cgroup 'mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b'
> I1129 09:53:42.163172 52310 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.163269 52310 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b after 1 attempts
> I1129 09:53:42.163727 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:0, core:0, socket:0)
> I1129 09:53:42.163769 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:1, core:1, socket:0)
> I1129 09:53:42.164930 52314 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.165045 52314 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.179460 52312 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:43.146617 52289 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:43.146682 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 73.24us
> I1129 09:53:44.147394 52291 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:44.147459 52291 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 80.86us
> Initialized scheduler with 4 tasks
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:44.266501 52325 process.cpp:1404] libprocess is initialized on 10.35.12.124:39959 for 16 cpus
> I1129 09:53:44.266835 52325 logging.cpp:179] Logging to STDERR
> I1129 09:53:44.268498 52348 sched.cpp:181] New master at master@127.0.0.1:5432
> I1129 09:53:44.269433 52289 master.cpp:602] Registering framework 201211290953-2081170186-5432-52261-0000 at scheduler(1)@10.35.12.124:39959
> I1129 09:53:44.269762 52289 hierarchical_allocator_process.hpp:266] Added framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269809 52289 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.269863 52289 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269948 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 151.08us
> I1129 09:53:44.269989 52349 sched.cpp:216] Framework registered with 201211290953-2081170186-5432-52261-0000
> Registered
> I1129 09:53:44.270150 52288 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.270256 52288 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.270597 52350 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 1
> Starting task 2
> I1129 09:53:44.271181 52283 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-0 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.271360 52283 master.hpp:285] Adding task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271396 52283 master.cpp:1580] Launching task 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271618 52283 master.hpp:285] Adding task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271663 52283 master.cpp:1580] Launching task 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271772 52283 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271889 52309 slave.cpp:477] Got assigned task 1 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.273299 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.273581 52309 slave.cpp:477] Got assigned task 2 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.275192 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.275202 52321 cgroups_isolation_module.cpp:441] Launching 1 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c
> I1129 09:53:44.275431 52309 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.275568 52310 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.277354 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.277768 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.277865 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.278280 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.278712 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.279321 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52361
> I1129 09:53:44.280704 52321 cgroups_isolation_module.cpp:441] Launching 2 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b
> I1129 09:53:44.293046 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.293432 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.293527 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.293939 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294385 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294860 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52364
> I1129 09:53:44.532757 52323 slave.cpp:775] Got registration for executor '1' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533000 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.533148 52323 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533500 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533584 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.534265 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.536386 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.536618 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.536713 52318 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.536906 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.537102 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.537147 52318 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537211 52318 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.537238 52289 master.hpp:297] Removing task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537382 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537376 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.537499 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.537716 52351 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537915 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.537931 52322 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538000 52318 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538142 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.538341 52319 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.539729 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.594710 52313 slave.cpp:775] Got registration for executor '2' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.594856 52312 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.595021 52313 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595425 52312 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595496 52312 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.596107 52312 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.598393 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598541 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.598624 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.598870 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598999 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599064 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.599083 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.599196 52280 master.hpp:297] Removing task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599321 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599367 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.599483 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.599690 52347 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599812 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599819 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.599839 52323 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.600172 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.600360 52321 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.623421 52324 process.cpp:872] Socket closed while receiving
> W1129 09:53:45.143110 52280 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:45.147429 52308 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147513 52308 cgroups_isolation_module.cpp:543] Killing executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147531 52314 slave.cpp:1024] Executor '1' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:45.148080 52321 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c for removal
> I1129 09:53:45.148236 52321 gc.cpp:125] Scheduling GC removal event to fire after 7.00days
> I1129 09:53:45.148641 52308 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.148715 52308 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:45.148777 52308 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.148810 52308 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149700 52309 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149817 52309 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc after 1 attempts
> I1129 09:53:45.151449 52317 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.151527 52317 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.147889 52276 hierarchical_allocator_process.hpp:665] No resources available to allocate!
> I1129 09:53:45.167976 52276 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 20.11ms
> I1129 09:53:45.147966 52279 master.cpp:1123] Executor 1 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:45.168719 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.174438 52310 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:46.149343 52309 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149416 52309 cgroups_isolation_module.cpp:543] Killing executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149461 52311 slave.cpp:1024] Executor '2' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:46.149678 52311 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b for removal
> I1129 09:53:46.149778 52279 master.cpp:1123] Executor 2 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:46.149931 52283 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150512 52309 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150578 52309 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:46.150712 52309 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.150753 52309 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151849 52319 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151981 52319 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789 after 1 attempts
> I1129 09:53:46.153518 52309 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.153612 52309 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.168972 52284 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169070 52284 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169142 52284 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 192.05us
> I1129 09:53:46.169292 52284 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169360 52284 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169692 52353 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 3
> Starting task 4
> I1129 09:53:46.170192 52288 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-1 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170331 52288 master.hpp:285] Adding task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170371 52288 master.cpp:1580] Launching task 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170524 52288 master.hpp:285] Adding task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170580 52288 master.cpp:1580] Launching task 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170658 52314 slave.cpp:477] Got assigned task 3 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170697 52288 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.171835 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.172061 52314 slave.cpp:477] Got assigned task 4 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.173218 52323 cgroups_isolation_module.cpp:441] Launching 3 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_3_tag_9d1040aa-d38a-4702-971d-41d88052555c
> I1129 09:53:46.173284 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098
> I1129 09:53:46.173493 52314 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.173651 52313 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> I1129 09:53:46.175964 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.176360 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.176440 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.176902 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177340 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177829 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52439
> I1129 09:53:46.179039 52323 cgroups_isolation_module.cpp:441] Launching 4 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_4_tag_aee80a43-b55a-4d91-9885-0be61dd1da88
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36
> I1129 09:53:46.193050 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.193454 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.193539 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.194010 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194481 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194953 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52442
> I1129 09:53:46.195163 52323 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.242967 52310 slave.cpp:775] Got registration for executor '3' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243160 52322 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.243348 52310 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243744 52322 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243871 52322 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.244596 52322 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247010 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247195 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247309 52320 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.247477 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247652 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247760 52320 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247808 52291 master.hpp:297] Removing task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.247818 52320 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.247889 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.247928 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247969 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.248142 52354 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.248278 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248327 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.248522 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248623 52320 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264420 52319 slave.cpp:775] Got registration for executor '4' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264575 52316 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.264660 52319 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265015 52316 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265108 52316 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.265822 52316 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.266053 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.266679 52314 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267032 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267441 52355 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.267706 52310 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.267735 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.267887 52311 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.267972 52310 slave.cpp:752] Got acknowledgement of status update for task 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268178 52281 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.268337 52281 master.hpp:297] Removing task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268324 52311 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268412 52311 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.268451 52281 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268563 52352 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.268631 52352 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268764 52352 sched.cpp:421] Stopping framework '201211290953-2081170186-5432-52261-0000'
> I1129 09:53:46.268933 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.269044 52311 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269086 52289 master.cpp:730] Asked to unregister framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269214 52291 hierarchical_allocator_process.hpp:357] Deactivated framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269321 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269405 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269484 52280 hierarchical_allocator_process.hpp:310] Removed framework 201211290953-2081170186-5432-52261-0000
> W1129 09:53:46.269382 52320 slave.cpp:671] Ignoring shutdown framework message from master@10.35.12.124:5432because it is not from the registered master (master@127.0.0.1:5432)
> I1129 09:53:46.270210 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.270334 52292 process.cpp:872] Socket closed while receiving
> /home/vinod/mesos/support/atexit.sh: line 11: 52261 Terminated ${MESOS_BUILD_DIR}/src/mesos-master --port=5432
> /home/vinod/mesos/support/atexit.sh: line 11: 52293 Terminated ${MESOS_BUILD_DIR}/src/mesos-slave --master=localhost:5432 --isolation=cgroups --cgroups_hierarchy_root=${HIERARCHY} --cgroups_subsystems="cpuset,memory,freezer" --resources="cpus:2;mem:1024"
> [ OK ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework (6215 ms)
> [----------] 1 test from CgroupsCpusetIsolationTest (6215 ms total)
>
> [----------] 5 tests from CgroupsCpusetTest
> [ RUN ] CgroupsCpusetTest.OneCPUOneCpuset
> [ OK ] CgroupsCpusetTest.OneCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.OneCPUManyCpusets
> [ OK ] CgroupsCpusetTest.OneCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUOneCpuset
> [ OK ] CgroupsCpusetTest.ManyCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUManyCpusets
> [ OK ] CgroupsCpusetTest.ManyCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.IntegerAllocations
> [ OK ] CgroupsCpusetTest.IntegerAllocations (0 ms)
> [----------] 5 tests from CgroupsCpusetTest (0 ms total)
>
> [----------] Global test environment tear-down
> [==========] 6 tests from 2 test cases ran. (6216 ms total)
> [ PASSED ] 6 tests.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Vinod Kone <vi...@gmail.com>.
> On Nov. 30, 2012, 12:59 a.m., Ben Mahler wrote:
> > Just to confirm, this isn't testing the actual isolation, correct?
> > It's more work, but we could use the getcpu call to verify exclusivity of cores (if doing integer allocations).
I was mainly testing whether we were successfully allocating and de-allocating resources and hence launching tasks.
But, I agree we could/should more tests.
> On Nov. 30, 2012, 12:59 a.m., Ben Mahler wrote:
> > src/slave/cgroups_isolation_module.cpp, line 706
> > <https://reviews.apache.org/r/8255/diff/3/?file=231632#file231632line706>
> >
> > This might come out a little nicer: s/ \t /\t\t
are you basing this on current executor id and framework id lengths in my test output? because, we can't guarantee their lengths are going to look the same.
> On Nov. 30, 2012, 12:59 a.m., Ben Mahler wrote:
> > src/slave/cgroups_isolation_module.cpp, line 943
> > <https://reviews.apache.org/r/8255/diff/3/?file=231632#file231632line943>
> >
> > What are the semantics of destroyWaited()? And is the window safe to have?
it is just the asynchronous callback that gets called when the cgroup gets destroyed/failed.
i THINK it is safe, but can't say for sure without looking at how it manifests in reality. my assumption is based on the fact that the old cgroup that is being in the process of getting destroyed wouldn't be burning cpus. from what we have seen so far, the reason this delay is non-trivial is because some processes get stuck in a weird state (S? D? blocking on kernel code?). but i don't seem to recall seeing processes in running state. BenH/Toby can correct me?
Nonetheless, the right thing to do here, that the isolation module shouldn't inform the slave (and hence the master) about the terminated executor until 'destroyWaited' is called.
We don't do that currently because we (I?) are not sure how to get the exit status of the executor, which we need to send as part of 'executorTerminated' message. There is already a TODO for this in the code. Once we do this, it doesn't matter whether the isolation module does the deallocation in unregisterCgroupsInfo() or destroyWaited() because the master wouldn't offer those resources until destroyWaited() is called.
Makes sense?
- Vinod
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/#review13878
-----------------------------------------------------------
On Nov. 29, 2012, 7:09 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8255/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2012, 7:09 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Tested by modifying balloon framework test to start the slave cpusets enabled.
>
> Will be writing more tests soon.
>
>
> Diffs
> -----
>
> src/Makefile.am 8dbb799c6203b2860e29860dcadc152c7d29755b
> src/examples/cpuset_executor.cpp PRE-CREATION
> src/examples/cpuset_framework.cpp PRE-CREATION
> src/master/master.hpp 9b9b550adbc501f90ab280da74f2ba9852c19ff7
> src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
> src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
> src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
> src/tests/cgroups_isolation_tests.cpp 2685e22eeaf0b764b84e2a6e33e73f75db0515b6
> src/tests/cpuset_framework_test.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8255/diff/
>
>
> Testing
> -------
>
> [vinod@smfd-aki-27-sr1:~/mesos/build] $ sudo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cpuset*
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.089644 52201 process.cpp:1404] libprocess is initialized on 10.35.12.124:36552 for 16 cpus
> I1129 09:53:40.090000 52201 logging.cpp:179] Logging to STDERR
> Source directory: /home/vinod/mesos
> Build directory: /home/vinod/mesos/build
> Note: Google Test filter = *Cpuset*
> [==========] Running 6 tests from 2 test cases.
> [----------] Global test environment set-up.
> [----------] 1 test from CgroupsCpusetIsolationTest
> [ RUN ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework
> Using temporary directory '/tmp/CgroupsCpusetIsolationTest_ROOT_CGROUPS_CpusetFramework_vgnCzO'
> Launched master at 52261
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.140386 52261 process.cpp:1404] libprocess is initialized on 10.35.12.124:5432 for 16 cpus
> I1129 09:53:40.140688 52261 logging.cpp:179] Logging to STDERR
> I1129 09:53:40.140991 52261 main.cpp:116] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:40.141032 52261 main.cpp:117] Starting Mesos master
> I1129 09:53:40.141351 52283 master.cpp:306] Master started on 10.35.12.124:5432
> I1129 09:53:40.141427 52283 master.cpp:321] Master ID: 201211290953-2081170186-5432-52261
> W1129 09:53:40.141677 52282 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:40.141690 52288 hierarchical_allocator_process.hpp:234] Initializing hierarchical allocator process with master : master@10.35.12.124:5432
> I1129 09:53:40.142582 52283 master.cpp:559] Elected as master!
> I1129 09:53:41.143687 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:41.143728 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 50.20us
> Launched slave at 52293
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:42.142441 52293 process.cpp:1404] libprocess is initialized on 10.35.12.124:58922 for 16 cpus
> I1129 09:53:42.142765 52293 logging.cpp:179] Logging to STDERR
> I1129 09:53:42.143033 52293 main.cpp:122] Creating "cgroups" isolation module
> I1129 09:53:42.143414 52293 main.cpp:130] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:42.143451 52293 main.cpp:131] Starting Mesos slave
> I1129 09:53:42.143728 52309 slave.cpp:166] Slave started on 1)@10.35.12.124:58922
> I1129 09:53:42.143776 52309 slave.cpp:167] Slave resources: cpus=2; mem=1024
> I1129 09:53:42.144351 52321 cgroups_isolation_module.cpp:219] Using /cgroup as cgroups hierarchy root
> I1129 09:53:42.144350 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.144395 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 52.87us
> I1129 09:53:42.145091 52309 slave.cpp:368] New master detected at master@127.0.0.1:5432
> I1129 09:53:42.145892 52278 master.cpp:924] Attempting to register slave on smfd-aki-27-sr1.devel.twitter.com at slave(1)@10.35.12.124:58922
> I1129 09:53:42.145926 52278 master.cpp:1159] Master now considering a slave at smfd-aki-27-sr1.devel.twitter.com:58922 as active
> I1129 09:53:42.145949 52278 master.cpp:1729] Adding slave 201211290953-2081170186-5432-52261-0 at smfd-aki-27-sr1.devel.twitter.com with cpus=2; mem=1024
> I1129 09:53:42.146421 52278 hierarchical_allocator_process.hpp:393] Added slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) with cpus=2; mem=1024 (and cpus=2; mem=1024 available)
> I1129 09:53:42.146469 52278 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.146488 52278 hierarchical_allocator_process.hpp:616] Performed allocation for slave 201211290953-2081170186-5432-52261-0 in 19.15us
> I1129 09:53:42.146875 52318 slave.cpp:388] Registered with master; given slave ID 201211290953-2081170186-5432-52261-0
> I1129 09:53:42.147235 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:42.161064 52321 cgroups_isolation_module.cpp:315] Removing orphaned cgroup 'mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b'
> I1129 09:53:42.163172 52310 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.163269 52310 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b after 1 attempts
> I1129 09:53:42.163727 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:0, core:0, socket:0)
> I1129 09:53:42.163769 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:1, core:1, socket:0)
> I1129 09:53:42.164930 52314 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.165045 52314 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.179460 52312 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:43.146617 52289 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:43.146682 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 73.24us
> I1129 09:53:44.147394 52291 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:44.147459 52291 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 80.86us
> Initialized scheduler with 4 tasks
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:44.266501 52325 process.cpp:1404] libprocess is initialized on 10.35.12.124:39959 for 16 cpus
> I1129 09:53:44.266835 52325 logging.cpp:179] Logging to STDERR
> I1129 09:53:44.268498 52348 sched.cpp:181] New master at master@127.0.0.1:5432
> I1129 09:53:44.269433 52289 master.cpp:602] Registering framework 201211290953-2081170186-5432-52261-0000 at scheduler(1)@10.35.12.124:39959
> I1129 09:53:44.269762 52289 hierarchical_allocator_process.hpp:266] Added framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269809 52289 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.269863 52289 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269948 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 151.08us
> I1129 09:53:44.269989 52349 sched.cpp:216] Framework registered with 201211290953-2081170186-5432-52261-0000
> Registered
> I1129 09:53:44.270150 52288 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.270256 52288 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.270597 52350 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 1
> Starting task 2
> I1129 09:53:44.271181 52283 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-0 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.271360 52283 master.hpp:285] Adding task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271396 52283 master.cpp:1580] Launching task 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271618 52283 master.hpp:285] Adding task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271663 52283 master.cpp:1580] Launching task 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271772 52283 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271889 52309 slave.cpp:477] Got assigned task 1 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.273299 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.273581 52309 slave.cpp:477] Got assigned task 2 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.275192 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.275202 52321 cgroups_isolation_module.cpp:441] Launching 1 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c
> I1129 09:53:44.275431 52309 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.275568 52310 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.277354 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.277768 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.277865 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.278280 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.278712 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.279321 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52361
> I1129 09:53:44.280704 52321 cgroups_isolation_module.cpp:441] Launching 2 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b
> I1129 09:53:44.293046 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.293432 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.293527 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.293939 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294385 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294860 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52364
> I1129 09:53:44.532757 52323 slave.cpp:775] Got registration for executor '1' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533000 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.533148 52323 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533500 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533584 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.534265 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.536386 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.536618 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.536713 52318 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.536906 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.537102 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.537147 52318 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537211 52318 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.537238 52289 master.hpp:297] Removing task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537382 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537376 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.537499 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.537716 52351 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537915 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.537931 52322 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538000 52318 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538142 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.538341 52319 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.539729 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.594710 52313 slave.cpp:775] Got registration for executor '2' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.594856 52312 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.595021 52313 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595425 52312 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595496 52312 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.596107 52312 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.598393 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598541 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.598624 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.598870 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598999 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599064 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.599083 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.599196 52280 master.hpp:297] Removing task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599321 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599367 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.599483 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.599690 52347 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599812 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599819 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.599839 52323 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.600172 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.600360 52321 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.623421 52324 process.cpp:872] Socket closed while receiving
> W1129 09:53:45.143110 52280 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:45.147429 52308 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147513 52308 cgroups_isolation_module.cpp:543] Killing executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147531 52314 slave.cpp:1024] Executor '1' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:45.148080 52321 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c for removal
> I1129 09:53:45.148236 52321 gc.cpp:125] Scheduling GC removal event to fire after 7.00days
> I1129 09:53:45.148641 52308 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.148715 52308 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:45.148777 52308 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.148810 52308 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149700 52309 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149817 52309 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc after 1 attempts
> I1129 09:53:45.151449 52317 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.151527 52317 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.147889 52276 hierarchical_allocator_process.hpp:665] No resources available to allocate!
> I1129 09:53:45.167976 52276 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 20.11ms
> I1129 09:53:45.147966 52279 master.cpp:1123] Executor 1 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:45.168719 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.174438 52310 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:46.149343 52309 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149416 52309 cgroups_isolation_module.cpp:543] Killing executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149461 52311 slave.cpp:1024] Executor '2' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:46.149678 52311 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b for removal
> I1129 09:53:46.149778 52279 master.cpp:1123] Executor 2 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:46.149931 52283 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150512 52309 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150578 52309 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:46.150712 52309 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.150753 52309 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151849 52319 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151981 52319 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789 after 1 attempts
> I1129 09:53:46.153518 52309 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.153612 52309 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.168972 52284 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169070 52284 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169142 52284 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 192.05us
> I1129 09:53:46.169292 52284 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169360 52284 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169692 52353 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 3
> Starting task 4
> I1129 09:53:46.170192 52288 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-1 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170331 52288 master.hpp:285] Adding task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170371 52288 master.cpp:1580] Launching task 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170524 52288 master.hpp:285] Adding task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170580 52288 master.cpp:1580] Launching task 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170658 52314 slave.cpp:477] Got assigned task 3 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170697 52288 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.171835 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.172061 52314 slave.cpp:477] Got assigned task 4 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.173218 52323 cgroups_isolation_module.cpp:441] Launching 3 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_3_tag_9d1040aa-d38a-4702-971d-41d88052555c
> I1129 09:53:46.173284 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098
> I1129 09:53:46.173493 52314 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.173651 52313 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> I1129 09:53:46.175964 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.176360 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.176440 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.176902 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177340 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177829 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52439
> I1129 09:53:46.179039 52323 cgroups_isolation_module.cpp:441] Launching 4 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_4_tag_aee80a43-b55a-4d91-9885-0be61dd1da88
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36
> I1129 09:53:46.193050 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.193454 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.193539 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.194010 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194481 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194953 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52442
> I1129 09:53:46.195163 52323 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.242967 52310 slave.cpp:775] Got registration for executor '3' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243160 52322 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.243348 52310 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243744 52322 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243871 52322 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.244596 52322 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247010 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247195 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247309 52320 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.247477 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247652 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247760 52320 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247808 52291 master.hpp:297] Removing task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.247818 52320 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.247889 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.247928 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247969 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.248142 52354 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.248278 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248327 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.248522 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248623 52320 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264420 52319 slave.cpp:775] Got registration for executor '4' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264575 52316 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.264660 52319 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265015 52316 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265108 52316 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.265822 52316 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.266053 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.266679 52314 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267032 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267441 52355 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.267706 52310 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.267735 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.267887 52311 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.267972 52310 slave.cpp:752] Got acknowledgement of status update for task 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268178 52281 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.268337 52281 master.hpp:297] Removing task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268324 52311 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268412 52311 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.268451 52281 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268563 52352 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.268631 52352 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268764 52352 sched.cpp:421] Stopping framework '201211290953-2081170186-5432-52261-0000'
> I1129 09:53:46.268933 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.269044 52311 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269086 52289 master.cpp:730] Asked to unregister framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269214 52291 hierarchical_allocator_process.hpp:357] Deactivated framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269321 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269405 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269484 52280 hierarchical_allocator_process.hpp:310] Removed framework 201211290953-2081170186-5432-52261-0000
> W1129 09:53:46.269382 52320 slave.cpp:671] Ignoring shutdown framework message from master@10.35.12.124:5432because it is not from the registered master (master@127.0.0.1:5432)
> I1129 09:53:46.270210 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.270334 52292 process.cpp:872] Socket closed while receiving
> /home/vinod/mesos/support/atexit.sh: line 11: 52261 Terminated ${MESOS_BUILD_DIR}/src/mesos-master --port=5432
> /home/vinod/mesos/support/atexit.sh: line 11: 52293 Terminated ${MESOS_BUILD_DIR}/src/mesos-slave --master=localhost:5432 --isolation=cgroups --cgroups_hierarchy_root=${HIERARCHY} --cgroups_subsystems="cpuset,memory,freezer" --resources="cpus:2;mem:1024"
> [ OK ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework (6215 ms)
> [----------] 1 test from CgroupsCpusetIsolationTest (6215 ms total)
>
> [----------] 5 tests from CgroupsCpusetTest
> [ RUN ] CgroupsCpusetTest.OneCPUOneCpuset
> [ OK ] CgroupsCpusetTest.OneCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.OneCPUManyCpusets
> [ OK ] CgroupsCpusetTest.OneCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUOneCpuset
> [ OK ] CgroupsCpusetTest.ManyCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUManyCpusets
> [ OK ] CgroupsCpusetTest.ManyCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.IntegerAllocations
> [ OK ] CgroupsCpusetTest.IntegerAllocations (0 ms)
> [----------] 5 tests from CgroupsCpusetTest (0 ms total)
>
> [----------] Global test environment tear-down
> [==========] 6 tests from 2 test cases ran. (6216 ms total)
> [ PASSED ] 6 tests.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/#review13878
-----------------------------------------------------------
Just to confirm, this isn't testing the actual isolation, correct?
It's more work, but we could use the getcpu call to verify exclusivity of cores (if doing integer allocations).
src/slave/cgroups_isolation_module.cpp
<https://reviews.apache.org/r/8255/#comment29664>
This might come out a little nicer: s/ \t /\t\t
src/slave/cgroups_isolation_module.cpp
<https://reviews.apache.org/r/8255/#comment29665>
ditto here
src/slave/cgroups_isolation_module.cpp
<https://reviews.apache.org/r/8255/#comment29666>
What are the semantics of destroyWaited()? And is the window safe to have?
- Ben Mahler
On Nov. 29, 2012, 7:09 p.m., Vinod Kone wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8255/
> -----------------------------------------------------------
>
> (Updated Nov. 29, 2012, 7:09 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Ben Mahler.
>
>
> Description
> -------
>
> Tested by modifying balloon framework test to start the slave cpusets enabled.
>
> Will be writing more tests soon.
>
>
> Diffs
> -----
>
> src/Makefile.am 8dbb799c6203b2860e29860dcadc152c7d29755b
> src/examples/cpuset_executor.cpp PRE-CREATION
> src/examples/cpuset_framework.cpp PRE-CREATION
> src/master/master.hpp 9b9b550adbc501f90ab280da74f2ba9852c19ff7
> src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
> src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
> src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
> src/tests/cgroups_isolation_tests.cpp 2685e22eeaf0b764b84e2a6e33e73f75db0515b6
> src/tests/cpuset_framework_test.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/8255/diff/
>
>
> Testing
> -------
>
> [vinod@smfd-aki-27-sr1:~/mesos/build] $ sudo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cpuset*
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.089644 52201 process.cpp:1404] libprocess is initialized on 10.35.12.124:36552 for 16 cpus
> I1129 09:53:40.090000 52201 logging.cpp:179] Logging to STDERR
> Source directory: /home/vinod/mesos
> Build directory: /home/vinod/mesos/build
> Note: Google Test filter = *Cpuset*
> [==========] Running 6 tests from 2 test cases.
> [----------] Global test environment set-up.
> [----------] 1 test from CgroupsCpusetIsolationTest
> [ RUN ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework
> Using temporary directory '/tmp/CgroupsCpusetIsolationTest_ROOT_CGROUPS_CpusetFramework_vgnCzO'
> Launched master at 52261
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:40.140386 52261 process.cpp:1404] libprocess is initialized on 10.35.12.124:5432 for 16 cpus
> I1129 09:53:40.140688 52261 logging.cpp:179] Logging to STDERR
> I1129 09:53:40.140991 52261 main.cpp:116] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:40.141032 52261 main.cpp:117] Starting Mesos master
> I1129 09:53:40.141351 52283 master.cpp:306] Master started on 10.35.12.124:5432
> I1129 09:53:40.141427 52283 master.cpp:321] Master ID: 201211290953-2081170186-5432-52261
> W1129 09:53:40.141677 52282 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:40.141690 52288 hierarchical_allocator_process.hpp:234] Initializing hierarchical allocator process with master : master@10.35.12.124:5432
> I1129 09:53:40.142582 52283 master.cpp:559] Elected as master!
> I1129 09:53:41.143687 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:41.143728 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 50.20us
> Launched slave at 52293
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:42.142441 52293 process.cpp:1404] libprocess is initialized on 10.35.12.124:58922 for 16 cpus
> I1129 09:53:42.142765 52293 logging.cpp:179] Logging to STDERR
> I1129 09:53:42.143033 52293 main.cpp:122] Creating "cgroups" isolation module
> I1129 09:53:42.143414 52293 main.cpp:130] Build: 2012-11-02 23:22:34 by vinod
> I1129 09:53:42.143451 52293 main.cpp:131] Starting Mesos slave
> I1129 09:53:42.143728 52309 slave.cpp:166] Slave started on 1)@10.35.12.124:58922
> I1129 09:53:42.143776 52309 slave.cpp:167] Slave resources: cpus=2; mem=1024
> I1129 09:53:42.144351 52321 cgroups_isolation_module.cpp:219] Using /cgroup as cgroups hierarchy root
> I1129 09:53:42.144350 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.144395 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 52.87us
> I1129 09:53:42.145091 52309 slave.cpp:368] New master detected at master@127.0.0.1:5432
> I1129 09:53:42.145892 52278 master.cpp:924] Attempting to register slave on smfd-aki-27-sr1.devel.twitter.com at slave(1)@10.35.12.124:58922
> I1129 09:53:42.145926 52278 master.cpp:1159] Master now considering a slave at smfd-aki-27-sr1.devel.twitter.com:58922 as active
> I1129 09:53:42.145949 52278 master.cpp:1729] Adding slave 201211290953-2081170186-5432-52261-0 at smfd-aki-27-sr1.devel.twitter.com with cpus=2; mem=1024
> I1129 09:53:42.146421 52278 hierarchical_allocator_process.hpp:393] Added slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) with cpus=2; mem=1024 (and cpus=2; mem=1024 available)
> I1129 09:53:42.146469 52278 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:42.146488 52278 hierarchical_allocator_process.hpp:616] Performed allocation for slave 201211290953-2081170186-5432-52261-0 in 19.15us
> I1129 09:53:42.146875 52318 slave.cpp:388] Registered with master; given slave ID 201211290953-2081170186-5432-52261-0
> I1129 09:53:42.147235 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:42.161064 52321 cgroups_isolation_module.cpp:315] Removing orphaned cgroup 'mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b'
> I1129 09:53:42.163172 52310 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.163269 52310 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b after 1 attempts
> I1129 09:53:42.163727 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:0, core:0, socket:0)
> I1129 09:53:42.163769 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:1, core:1, socket:0)
> I1129 09:53:42.164930 52314 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.165045 52314 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:42.179460 52312 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
> I1129 09:53:43.146617 52289 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:43.146682 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 73.24us
> I1129 09:53:44.147394 52291 hierarchical_allocator_process.hpp:629] No users to allocate resources!
> I1129 09:53:44.147459 52291 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 80.86us
> Initialized scheduler with 4 tasks
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1129 09:53:44.266501 52325 process.cpp:1404] libprocess is initialized on 10.35.12.124:39959 for 16 cpus
> I1129 09:53:44.266835 52325 logging.cpp:179] Logging to STDERR
> I1129 09:53:44.268498 52348 sched.cpp:181] New master at master@127.0.0.1:5432
> I1129 09:53:44.269433 52289 master.cpp:602] Registering framework 201211290953-2081170186-5432-52261-0000 at scheduler(1)@10.35.12.124:39959
> I1129 09:53:44.269762 52289 hierarchical_allocator_process.hpp:266] Added framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269809 52289 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.269863 52289 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.269948 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 151.08us
> I1129 09:53:44.269989 52349 sched.cpp:216] Framework registered with 201211290953-2081170186-5432-52261-0000
> Registered
> I1129 09:53:44.270150 52288 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.270256 52288 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.270597 52350 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 1
> Starting task 2
> I1129 09:53:44.271181 52283 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-0 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.271360 52283 master.hpp:285] Adding task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271396 52283 master.cpp:1580] Launching task 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271618 52283 master.hpp:285] Adding task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271663 52283 master.cpp:1580] Launching task 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:44.271772 52283 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.271889 52309 slave.cpp:477] Got assigned task 1 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.273299 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.273581 52309 slave.cpp:477] Got assigned task 2 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.275192 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.275202 52321 cgroups_isolation_module.cpp:441] Launching 1 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c
> I1129 09:53:44.275431 52309 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
> I1129 09:53:44.275568 52310 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
> I1129 09:53:44.277354 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.277768 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.277865 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.278280 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.278712 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.279321 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52361
> I1129 09:53:44.280704 52321 cgroups_isolation_module.cpp:441] Launching 2 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b
> I1129 09:53:44.293046 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.293432 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.293527 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.293939 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294385 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.294860 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52364
> I1129 09:53:44.532757 52323 slave.cpp:775] Got registration for executor '1' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533000 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.533148 52323 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533500 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.533584 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.534265 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.536386 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.536618 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.536713 52318 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.536906 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.537102 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.537147 52318 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537211 52318 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.537238 52289 master.hpp:297] Removing task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537382 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.537376 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.537499 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.537716 52351 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.537915 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.537931 52322 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538000 52318 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.538142 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.538341 52319 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.539729 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.594710 52313 slave.cpp:775] Got registration for executor '2' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.594856 52312 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:44.595021 52313 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595425 52312 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.595496 52312 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.596107 52312 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.598393 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598541 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.598624 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:44.598870 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:44.598999 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599064 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:44.599083 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:44.599196 52280 master.hpp:297] Removing task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599321 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599367 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:44.599483 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:44.599690 52347 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:44.599812 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.599819 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.599839 52323 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.600172 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:44.600360 52321 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:44.623421 52324 process.cpp:872] Socket closed while receiving
> W1129 09:53:45.143110 52280 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I1129 09:53:45.147429 52308 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147513 52308 cgroups_isolation_module.cpp:543] Killing executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.147531 52314 slave.cpp:1024] Executor '1' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:45.148080 52321 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c for removal
> I1129 09:53:45.148236 52321 gc.cpp:125] Scheduling GC removal event to fire after 7.00days
> I1129 09:53:45.148641 52308 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.148715 52308 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 1 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:45.148777 52308 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.148810 52308 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149700 52309 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.149817 52309 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc after 1 attempts
> I1129 09:53:45.151449 52317 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.151527 52317 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:45.147889 52276 hierarchical_allocator_process.hpp:665] No resources available to allocate!
> I1129 09:53:45.167976 52276 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 20.11ms
> I1129 09:53:45.147966 52279 master.cpp:1123] Executor 1 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:45.168719 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:45.174438 52310 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
> I1129 09:53:46.149343 52309 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149416 52309 cgroups_isolation_module.cpp:543] Killing executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.149461 52311 slave.cpp:1024] Executor '2' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
> I1129 09:53:46.149678 52311 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b for removal
> I1129 09:53:46.149778 52279 master.cpp:1123] Executor 2 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
> I1129 09:53:46.149931 52283 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150512 52309 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.150578 52309 cgroups_isolation_module.cpp:959] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 2 201211290953-2081170186-5432-52261-0000 { }
> ******************************************************************
> I1129 09:53:46.150712 52309 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.150753 52309 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151849 52319 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.151981 52319 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789 after 1 attempts
> I1129 09:53:46.153518 52309 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.153612 52309 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.168972 52284 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169070 52284 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169142 52284 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 192.05us
> I1129 09:53:46.169292 52284 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.169360 52284 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.169692 52353 sched.cpp:281] Received 1 offers
> Resource offers received: 1
> Starting task 3
> Starting task 4
> I1129 09:53:46.170192 52288 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-1 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170331 52288 master.hpp:285] Adding task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170371 52288 master.cpp:1580] Launching task 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170524 52288 master.hpp:285] Adding task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.170580 52288 master.cpp:1580] Launching task 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
> I1129 09:53:46.170658 52314 slave.cpp:477] Got assigned task 3 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.170697 52288 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.171835 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.172061 52314 slave.cpp:477] Got assigned task 4 for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.173218 52323 cgroups_isolation_module.cpp:441] Launching 3 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_3_tag_9d1040aa-d38a-4702-971d-41d88052555c
> I1129 09:53:46.173284 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098
> I1129 09:53:46.173493 52314 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
> I1129 09:53:46.173651 52313 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
> I1129 09:53:46.175964 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.176360 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.176440 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.176902 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177340 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.177829 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52439
> I1129 09:53:46.179039 52323 cgroups_isolation_module.cpp:441] Launching 4 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_4_tag_aee80a43-b55a-4d91-9885-0be61dd1da88
> Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36
> I1129 09:53:46.193050 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.193454 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.193539 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.194010 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194481 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.194953 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52442
> I1129 09:53:46.195163 52323 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
> I1129 09:53:46.242967 52310 slave.cpp:775] Got registration for executor '3' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243160 52322 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.243348 52310 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243744 52322 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.243871 52322 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.244596 52322 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247010 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247195 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247309 52320 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.247477 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.247652 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.247760 52320 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247808 52291 master.hpp:297] Removing task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.247818 52320 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.247889 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.247928 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.247969 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.248142 52354 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.248278 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248327 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.248522 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.248623 52320 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264420 52319 slave.cpp:775] Got registration for executor '4' of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.264575 52316 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
> I1129 09:53:46.264660 52319 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265015 52316 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.265108 52316 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.265822 52316 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.266053 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.266679 52314 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267032 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> I1129 09:53:46.267441 52355 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
> Task in state TASK_RUNNING
> I1129 09:53:46.267706 52310 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.267735 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.267887 52311 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
> I1129 09:53:46.267972 52310 slave.cpp:752] Got acknowledgement of status update for task 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268178 52281 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> I1129 09:53:46.268337 52281 master.hpp:297] Removing task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268324 52311 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268412 52311 cgroups_isolation_module.cpp:695] Cpusets allocation
> ******************************************************************
> Executor Framework Allocation
> ******************************************************************
> 4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> 3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
> ******************************************************************
> I1129 09:53:46.268451 52281 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.268563 52352 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
> Task in state TASK_FINISHED
> I1129 09:53:46.268631 52352 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
> I1129 09:53:46.268764 52352 sched.cpp:421] Stopping framework '201211290953-2081170186-5432-52261-0000'
> I1129 09:53:46.268933 52324 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.269044 52311 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269086 52289 master.cpp:730] Asked to unregister framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269214 52291 hierarchical_allocator_process.hpp:357] Deactivated framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269321 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269405 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
> I1129 09:53:46.269484 52280 hierarchical_allocator_process.hpp:310] Removed framework 201211290953-2081170186-5432-52261-0000
> W1129 09:53:46.269382 52320 slave.cpp:671] Ignoring shutdown framework message from master@10.35.12.124:5432because it is not from the registered master (master@127.0.0.1:5432)
> I1129 09:53:46.270210 52292 process.cpp:872] Socket closed while receiving
> I1129 09:53:46.270334 52292 process.cpp:872] Socket closed while receiving
> /home/vinod/mesos/support/atexit.sh: line 11: 52261 Terminated ${MESOS_BUILD_DIR}/src/mesos-master --port=5432
> /home/vinod/mesos/support/atexit.sh: line 11: 52293 Terminated ${MESOS_BUILD_DIR}/src/mesos-slave --master=localhost:5432 --isolation=cgroups --cgroups_hierarchy_root=${HIERARCHY} --cgroups_subsystems="cpuset,memory,freezer" --resources="cpus:2;mem:1024"
> [ OK ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework (6215 ms)
> [----------] 1 test from CgroupsCpusetIsolationTest (6215 ms total)
>
> [----------] 5 tests from CgroupsCpusetTest
> [ RUN ] CgroupsCpusetTest.OneCPUOneCpuset
> [ OK ] CgroupsCpusetTest.OneCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.OneCPUManyCpusets
> [ OK ] CgroupsCpusetTest.OneCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUOneCpuset
> [ OK ] CgroupsCpusetTest.ManyCPUOneCpuset (0 ms)
> [ RUN ] CgroupsCpusetTest.ManyCPUManyCpusets
> [ OK ] CgroupsCpusetTest.ManyCPUManyCpusets (0 ms)
> [ RUN ] CgroupsCpusetTest.IntegerAllocations
> [ OK ] CgroupsCpusetTest.IntegerAllocations (0 ms)
> [----------] 5 tests from CgroupsCpusetTest (0 ms total)
>
> [----------] Global test environment tear-down
> [==========] 6 tests from 2 test cases ran. (6216 ms total)
> [ PASSED ] 6 tests.
>
>
> Thanks,
>
> Vinod Kone
>
>
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/
-----------------------------------------------------------
(Updated Nov. 29, 2012, 7:09 p.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
minor fixes
Description
-------
Tested by modifying balloon framework test to start the slave cpusets enabled.
Will be writing more tests soon.
Diffs (updated)
-----
src/Makefile.am 8dbb799c6203b2860e29860dcadc152c7d29755b
src/examples/cpuset_executor.cpp PRE-CREATION
src/examples/cpuset_framework.cpp PRE-CREATION
src/master/master.hpp 9b9b550adbc501f90ab280da74f2ba9852c19ff7
src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
src/tests/cgroups_isolation_tests.cpp 2685e22eeaf0b764b84e2a6e33e73f75db0515b6
src/tests/cpuset_framework_test.sh PRE-CREATION
Diff: https://reviews.apache.org/r/8255/diff/
Testing
-------
[vinod@smfd-aki-27-sr1:~/mesos/build] $ sudo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cpuset*
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:40.089644 52201 process.cpp:1404] libprocess is initialized on 10.35.12.124:36552 for 16 cpus
I1129 09:53:40.090000 52201 logging.cpp:179] Logging to STDERR
Source directory: /home/vinod/mesos
Build directory: /home/vinod/mesos/build
Note: Google Test filter = *Cpuset*
[==========] Running 6 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from CgroupsCpusetIsolationTest
[ RUN ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework
Using temporary directory '/tmp/CgroupsCpusetIsolationTest_ROOT_CGROUPS_CpusetFramework_vgnCzO'
Launched master at 52261
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:40.140386 52261 process.cpp:1404] libprocess is initialized on 10.35.12.124:5432 for 16 cpus
I1129 09:53:40.140688 52261 logging.cpp:179] Logging to STDERR
I1129 09:53:40.140991 52261 main.cpp:116] Build: 2012-11-02 23:22:34 by vinod
I1129 09:53:40.141032 52261 main.cpp:117] Starting Mesos master
I1129 09:53:40.141351 52283 master.cpp:306] Master started on 10.35.12.124:5432
I1129 09:53:40.141427 52283 master.cpp:321] Master ID: 201211290953-2081170186-5432-52261
W1129 09:53:40.141677 52282 master.cpp:79] No whitelist given. Advertising offers for all slaves
I1129 09:53:40.141690 52288 hierarchical_allocator_process.hpp:234] Initializing hierarchical allocator process with master : master@10.35.12.124:5432
I1129 09:53:40.142582 52283 master.cpp:559] Elected as master!
I1129 09:53:41.143687 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:41.143728 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 50.20us
Launched slave at 52293
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:42.142441 52293 process.cpp:1404] libprocess is initialized on 10.35.12.124:58922 for 16 cpus
I1129 09:53:42.142765 52293 logging.cpp:179] Logging to STDERR
I1129 09:53:42.143033 52293 main.cpp:122] Creating "cgroups" isolation module
I1129 09:53:42.143414 52293 main.cpp:130] Build: 2012-11-02 23:22:34 by vinod
I1129 09:53:42.143451 52293 main.cpp:131] Starting Mesos slave
I1129 09:53:42.143728 52309 slave.cpp:166] Slave started on 1)@10.35.12.124:58922
I1129 09:53:42.143776 52309 slave.cpp:167] Slave resources: cpus=2; mem=1024
I1129 09:53:42.144351 52321 cgroups_isolation_module.cpp:219] Using /cgroup as cgroups hierarchy root
I1129 09:53:42.144350 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:42.144395 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 52.87us
I1129 09:53:42.145091 52309 slave.cpp:368] New master detected at master@127.0.0.1:5432
I1129 09:53:42.145892 52278 master.cpp:924] Attempting to register slave on smfd-aki-27-sr1.devel.twitter.com at slave(1)@10.35.12.124:58922
I1129 09:53:42.145926 52278 master.cpp:1159] Master now considering a slave at smfd-aki-27-sr1.devel.twitter.com:58922 as active
I1129 09:53:42.145949 52278 master.cpp:1729] Adding slave 201211290953-2081170186-5432-52261-0 at smfd-aki-27-sr1.devel.twitter.com with cpus=2; mem=1024
I1129 09:53:42.146421 52278 hierarchical_allocator_process.hpp:393] Added slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) with cpus=2; mem=1024 (and cpus=2; mem=1024 available)
I1129 09:53:42.146469 52278 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:42.146488 52278 hierarchical_allocator_process.hpp:616] Performed allocation for slave 201211290953-2081170186-5432-52261-0 in 19.15us
I1129 09:53:42.146875 52318 slave.cpp:388] Registered with master; given slave ID 201211290953-2081170186-5432-52261-0
I1129 09:53:42.147235 52292 process.cpp:872] Socket closed while receiving
I1129 09:53:42.161064 52321 cgroups_isolation_module.cpp:315] Removing orphaned cgroup 'mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b'
I1129 09:53:42.163172 52310 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:42.163269 52310 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b after 1 attempts
I1129 09:53:42.163727 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:0, core:0, socket:0)
I1129 09:53:42.163769 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:1, core:1, socket:0)
I1129 09:53:42.164930 52314 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:42.165045 52314 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:42.179460 52312 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:43.146617 52289 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:43.146682 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 73.24us
I1129 09:53:44.147394 52291 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:44.147459 52291 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 80.86us
Initialized scheduler with 4 tasks
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:44.266501 52325 process.cpp:1404] libprocess is initialized on 10.35.12.124:39959 for 16 cpus
I1129 09:53:44.266835 52325 logging.cpp:179] Logging to STDERR
I1129 09:53:44.268498 52348 sched.cpp:181] New master at master@127.0.0.1:5432
I1129 09:53:44.269433 52289 master.cpp:602] Registering framework 201211290953-2081170186-5432-52261-0000 at scheduler(1)@10.35.12.124:39959
I1129 09:53:44.269762 52289 hierarchical_allocator_process.hpp:266] Added framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.269809 52289 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.269863 52289 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.269948 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 151.08us
I1129 09:53:44.269989 52349 sched.cpp:216] Framework registered with 201211290953-2081170186-5432-52261-0000
Registered
I1129 09:53:44.270150 52288 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.270256 52288 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.270597 52350 sched.cpp:281] Received 1 offers
Resource offers received: 1
Starting task 1
Starting task 2
I1129 09:53:44.271181 52283 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-0 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.271360 52283 master.hpp:285] Adding task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.271396 52283 master.cpp:1580] Launching task 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:44.271618 52283 master.hpp:285] Adding task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.271663 52283 master.cpp:1580] Launching task 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:44.271772 52283 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.271889 52309 slave.cpp:477] Got assigned task 1 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.273299 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
I1129 09:53:44.273581 52309 slave.cpp:477] Got assigned task 2 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.275192 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
I1129 09:53:44.275202 52321 cgroups_isolation_module.cpp:441] Launching 1 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c
I1129 09:53:44.275431 52309 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
I1129 09:53:44.275568 52310 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
I1129 09:53:44.277354 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.277768 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.277865 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.278280 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.278712 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.279321 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52361
I1129 09:53:44.280704 52321 cgroups_isolation_module.cpp:441] Launching 2 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b
I1129 09:53:44.293046 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.293432 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.293527 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.293939 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.294385 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.294860 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52364
I1129 09:53:44.532757 52323 slave.cpp:775] Got registration for executor '1' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.533000 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:44.533148 52323 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.533500 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.533584 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.534265 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.536386 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.536618 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.536713 52318 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.536906 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.537102 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.537147 52318 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.537211 52318 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.537238 52289 master.hpp:297] Removing task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.537382 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.537376 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:44.537499 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:44.537716 52351 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.537915 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.537931 52322 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.538000 52318 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.538142 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.538341 52319 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.539729 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.594710 52313 slave.cpp:775] Got registration for executor '2' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.594856 52312 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:44.595021 52313 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.595425 52312 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.595496 52312 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.596107 52312 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.598393 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.598541 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.598624 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.598870 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.598999 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.599064 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.599083 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.599196 52280 master.hpp:297] Removing task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.599321 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.599367 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:44.599483 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:44.599690 52347 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.599812 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.599819 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.599839 52323 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.600172 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.600360 52321 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.623421 52324 process.cpp:872] Socket closed while receiving
W1129 09:53:45.143110 52280 master.cpp:79] No whitelist given. Advertising offers for all slaves
I1129 09:53:45.147429 52308 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.147513 52308 cgroups_isolation_module.cpp:543] Killing executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.147531 52314 slave.cpp:1024] Executor '1' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
I1129 09:53:45.148080 52321 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c for removal
I1129 09:53:45.148236 52321 gc.cpp:125] Scheduling GC removal event to fire after 7.00days
I1129 09:53:45.148641 52308 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.148715 52308 cgroups_isolation_module.cpp:959] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { }
******************************************************************
I1129 09:53:45.148777 52308 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.148810 52308 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.149700 52309 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.149817 52309 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc after 1 attempts
I1129 09:53:45.151449 52317 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.151527 52317 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.147889 52276 hierarchical_allocator_process.hpp:665] No resources available to allocate!
I1129 09:53:45.167976 52276 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 20.11ms
I1129 09:53:45.147966 52279 master.cpp:1123] Executor 1 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
I1129 09:53:45.168719 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.174438 52310 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:46.149343 52309 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.149416 52309 cgroups_isolation_module.cpp:543] Killing executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.149461 52311 slave.cpp:1024] Executor '2' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
I1129 09:53:46.149678 52311 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b for removal
I1129 09:53:46.149778 52279 master.cpp:1123] Executor 2 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
I1129 09:53:46.149931 52283 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.150512 52309 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.150578 52309 cgroups_isolation_module.cpp:959] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { }
******************************************************************
I1129 09:53:46.150712 52309 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.150753 52309 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.151849 52319 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.151981 52319 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789 after 1 attempts
I1129 09:53:46.153518 52309 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.153612 52309 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.168972 52284 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.169070 52284 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.169142 52284 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 192.05us
I1129 09:53:46.169292 52284 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.169360 52284 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.169692 52353 sched.cpp:281] Received 1 offers
Resource offers received: 1
Starting task 3
Starting task 4
I1129 09:53:46.170192 52288 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-1 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.170331 52288 master.hpp:285] Adding task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.170371 52288 master.cpp:1580] Launching task 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:46.170524 52288 master.hpp:285] Adding task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.170580 52288 master.cpp:1580] Launching task 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:46.170658 52314 slave.cpp:477] Got assigned task 3 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.170697 52288 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.171835 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
I1129 09:53:46.172061 52314 slave.cpp:477] Got assigned task 4 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.173218 52323 cgroups_isolation_module.cpp:441] Launching 3 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_3_tag_9d1040aa-d38a-4702-971d-41d88052555c
I1129 09:53:46.173284 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098
I1129 09:53:46.173493 52314 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
I1129 09:53:46.173651 52313 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
I1129 09:53:46.175964 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.176360 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.176440 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.176902 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.177340 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.177829 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52439
I1129 09:53:46.179039 52323 cgroups_isolation_module.cpp:441] Launching 4 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_4_tag_aee80a43-b55a-4d91-9885-0be61dd1da88
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36
I1129 09:53:46.193050 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.193454 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.193539 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.194010 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.194481 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.194953 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52442
I1129 09:53:46.195163 52323 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.242967 52310 slave.cpp:775] Got registration for executor '3' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.243160 52322 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:46.243348 52310 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.243744 52322 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.243871 52322 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.244596 52322 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.247010 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.247195 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.247309 52320 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.247477 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.247652 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.247760 52320 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.247808 52291 master.hpp:297] Removing task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.247818 52320 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.247889 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:46.247928 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.247969 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:46.248142 52354 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.248278 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.248327 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.248522 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.248623 52320 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.264420 52319 slave.cpp:775] Got registration for executor '4' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.264575 52316 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:46.264660 52319 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.265015 52316 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.265108 52316 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.265822 52316 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.266053 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.266679 52314 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.267032 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.267441 52355 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:46.267706 52310 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.267735 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.267887 52311 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.267972 52310 slave.cpp:752] Got acknowledgement of status update for task 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.268178 52281 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.268337 52281 master.hpp:297] Removing task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.268324 52311 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.268412 52311 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.268451 52281 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.268563 52352 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:46.268631 52352 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.268764 52352 sched.cpp:421] Stopping framework '201211290953-2081170186-5432-52261-0000'
I1129 09:53:46.268933 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.269044 52311 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269086 52289 master.cpp:730] Asked to unregister framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269214 52291 hierarchical_allocator_process.hpp:357] Deactivated framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269321 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269405 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269484 52280 hierarchical_allocator_process.hpp:310] Removed framework 201211290953-2081170186-5432-52261-0000
W1129 09:53:46.269382 52320 slave.cpp:671] Ignoring shutdown framework message from master@10.35.12.124:5432because it is not from the registered master (master@127.0.0.1:5432)
I1129 09:53:46.270210 52292 process.cpp:872] Socket closed while receiving
I1129 09:53:46.270334 52292 process.cpp:872] Socket closed while receiving
/home/vinod/mesos/support/atexit.sh: line 11: 52261 Terminated ${MESOS_BUILD_DIR}/src/mesos-master --port=5432
/home/vinod/mesos/support/atexit.sh: line 11: 52293 Terminated ${MESOS_BUILD_DIR}/src/mesos-slave --master=localhost:5432 --isolation=cgroups --cgroups_hierarchy_root=${HIERARCHY} --cgroups_subsystems="cpuset,memory,freezer" --resources="cpus:2;mem:1024"
[ OK ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework (6215 ms)
[----------] 1 test from CgroupsCpusetIsolationTest (6215 ms total)
[----------] 5 tests from CgroupsCpusetTest
[ RUN ] CgroupsCpusetTest.OneCPUOneCpuset
[ OK ] CgroupsCpusetTest.OneCPUOneCpuset (0 ms)
[ RUN ] CgroupsCpusetTest.OneCPUManyCpusets
[ OK ] CgroupsCpusetTest.OneCPUManyCpusets (0 ms)
[ RUN ] CgroupsCpusetTest.ManyCPUOneCpuset
[ OK ] CgroupsCpusetTest.ManyCPUOneCpuset (0 ms)
[ RUN ] CgroupsCpusetTest.ManyCPUManyCpusets
[ OK ] CgroupsCpusetTest.ManyCPUManyCpusets (0 ms)
[ RUN ] CgroupsCpusetTest.IntegerAllocations
[ OK ] CgroupsCpusetTest.IntegerAllocations (0 ms)
[----------] 5 tests from CgroupsCpusetTest (0 ms total)
[----------] Global test environment tear-down
[==========] 6 tests from 2 test cases ran. (6216 ms total)
[ PASSED ] 6 tests.
Thanks,
Vinod Kone
Re: Review Request: Properly deallocates exited executor's resources"
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8255/
-----------------------------------------------------------
(Updated Nov. 29, 2012, 9:54 a.m.)
Review request for mesos, Benjamin Hindman and Ben Mahler.
Changes
-------
Added an integration test.
BenM's comments.
Description
-------
Tested by modifying balloon framework test to start the slave cpusets enabled.
Will be writing more tests soon.
Diffs (updated)
-----
src/Makefile.am 8dbb799c6203b2860e29860dcadc152c7d29755b
src/examples/cpuset_executor.cpp PRE-CREATION
src/examples/cpuset_framework.cpp PRE-CREATION
src/master/master.hpp 9b9b550adbc501f90ab280da74f2ba9852c19ff7
src/slave/cgroups_isolation_module.hpp 9f80fc5a969b959b34eaea4cac40700662d7f8b2
src/slave/cgroups_isolation_module.cpp 8211618d7729350654e2d17946c5b912ed9dda6a
src/tests/balloon_framework_test.sh 93a733f64cfde08349b7781eb3d5e13594c74498
src/tests/cgroups_isolation_tests.cpp 2685e22eeaf0b764b84e2a6e33e73f75db0515b6
src/tests/cpuset_framework_test.sh PRE-CREATION
Diff: https://reviews.apache.org/r/8255/diff/
Testing (updated)
-------
[vinod@smfd-aki-27-sr1:~/mesos/build] $ sudo GLOG_v=1 ./bin/mesos-tests.sh --verbose --gtest_filter=*Cpuset*
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:40.089644 52201 process.cpp:1404] libprocess is initialized on 10.35.12.124:36552 for 16 cpus
I1129 09:53:40.090000 52201 logging.cpp:179] Logging to STDERR
Source directory: /home/vinod/mesos
Build directory: /home/vinod/mesos/build
Note: Google Test filter = *Cpuset*
[==========] Running 6 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from CgroupsCpusetIsolationTest
[ RUN ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework
Using temporary directory '/tmp/CgroupsCpusetIsolationTest_ROOT_CGROUPS_CpusetFramework_vgnCzO'
Launched master at 52261
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:40.140386 52261 process.cpp:1404] libprocess is initialized on 10.35.12.124:5432 for 16 cpus
I1129 09:53:40.140688 52261 logging.cpp:179] Logging to STDERR
I1129 09:53:40.140991 52261 main.cpp:116] Build: 2012-11-02 23:22:34 by vinod
I1129 09:53:40.141032 52261 main.cpp:117] Starting Mesos master
I1129 09:53:40.141351 52283 master.cpp:306] Master started on 10.35.12.124:5432
I1129 09:53:40.141427 52283 master.cpp:321] Master ID: 201211290953-2081170186-5432-52261
W1129 09:53:40.141677 52282 master.cpp:79] No whitelist given. Advertising offers for all slaves
I1129 09:53:40.141690 52288 hierarchical_allocator_process.hpp:234] Initializing hierarchical allocator process with master : master@10.35.12.124:5432
I1129 09:53:40.142582 52283 master.cpp:559] Elected as master!
I1129 09:53:41.143687 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:41.143728 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 50.20us
Launched slave at 52293
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:42.142441 52293 process.cpp:1404] libprocess is initialized on 10.35.12.124:58922 for 16 cpus
I1129 09:53:42.142765 52293 logging.cpp:179] Logging to STDERR
I1129 09:53:42.143033 52293 main.cpp:122] Creating "cgroups" isolation module
I1129 09:53:42.143414 52293 main.cpp:130] Build: 2012-11-02 23:22:34 by vinod
I1129 09:53:42.143451 52293 main.cpp:131] Starting Mesos slave
I1129 09:53:42.143728 52309 slave.cpp:166] Slave started on 1)@10.35.12.124:58922
I1129 09:53:42.143776 52309 slave.cpp:167] Slave resources: cpus=2; mem=1024
I1129 09:53:42.144351 52321 cgroups_isolation_module.cpp:219] Using /cgroup as cgroups hierarchy root
I1129 09:53:42.144350 52290 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:42.144395 52290 hierarchical_allocator_process.hpp:596] Performed allocation for 0 slaves in 52.87us
I1129 09:53:42.145091 52309 slave.cpp:368] New master detected at master@127.0.0.1:5432
I1129 09:53:42.145892 52278 master.cpp:924] Attempting to register slave on smfd-aki-27-sr1.devel.twitter.com at slave(1)@10.35.12.124:58922
I1129 09:53:42.145926 52278 master.cpp:1159] Master now considering a slave at smfd-aki-27-sr1.devel.twitter.com:58922 as active
I1129 09:53:42.145949 52278 master.cpp:1729] Adding slave 201211290953-2081170186-5432-52261-0 at smfd-aki-27-sr1.devel.twitter.com with cpus=2; mem=1024
I1129 09:53:42.146421 52278 hierarchical_allocator_process.hpp:393] Added slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) with cpus=2; mem=1024 (and cpus=2; mem=1024 available)
I1129 09:53:42.146469 52278 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:42.146488 52278 hierarchical_allocator_process.hpp:616] Performed allocation for slave 201211290953-2081170186-5432-52261-0 in 19.15us
I1129 09:53:42.146875 52318 slave.cpp:388] Registered with master; given slave ID 201211290953-2081170186-5432-52261-0
I1129 09:53:42.147235 52292 process.cpp:872] Socket closed while receiving
I1129 09:53:42.161064 52321 cgroups_isolation_module.cpp:315] Removing orphaned cgroup 'mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b'
I1129 09:53:42.163172 52310 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:42.163269 52310 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b after 1 attempts
I1129 09:53:42.163727 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:0, core:0, socket:0)
I1129 09:53:42.163769 52321 cgroups_isolation_module.cpp:411] Initializing cpu allocation for CPU (id:1, core:1, socket:0)
I1129 09:53:42.164930 52314 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:42.165045 52314 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:42.179460 52312 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290944-2081170186-5432-50075-0000_executor_4_tag_da85086e-c7a1-480d-bdde-7f0b675f217b
I1129 09:53:43.146617 52289 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:43.146682 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 73.24us
I1129 09:53:44.147394 52291 hierarchical_allocator_process.hpp:629] No users to allocate resources!
I1129 09:53:44.147459 52291 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 80.86us
Initialized scheduler with 4 tasks
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1129 09:53:44.266501 52325 process.cpp:1404] libprocess is initialized on 10.35.12.124:39959 for 16 cpus
I1129 09:53:44.266835 52325 logging.cpp:179] Logging to STDERR
I1129 09:53:44.268498 52348 sched.cpp:181] New master at master@127.0.0.1:5432
I1129 09:53:44.269433 52289 master.cpp:602] Registering framework 201211290953-2081170186-5432-52261-0000 at scheduler(1)@10.35.12.124:39959
I1129 09:53:44.269762 52289 hierarchical_allocator_process.hpp:266] Added framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.269809 52289 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.269863 52289 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.269948 52289 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 151.08us
I1129 09:53:44.269989 52349 sched.cpp:216] Framework registered with 201211290953-2081170186-5432-52261-0000
Registered
I1129 09:53:44.270150 52288 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.270256 52288 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.270597 52350 sched.cpp:281] Received 1 offers
Resource offers received: 1
Starting task 1
Starting task 2
I1129 09:53:44.271181 52283 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-0 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.271360 52283 master.hpp:285] Adding task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.271396 52283 master.cpp:1580] Launching task 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:44.271618 52283 master.hpp:285] Adding task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.271663 52283 master.cpp:1580] Launching task 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:44.271772 52283 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-0 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.271889 52309 slave.cpp:477] Got assigned task 1 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.273299 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
I1129 09:53:44.273581 52309 slave.cpp:477] Got assigned task 2 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.275192 52309 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
I1129 09:53:44.275202 52321 cgroups_isolation_module.cpp:441] Launching 1 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c
I1129 09:53:44.275431 52309 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c'
I1129 09:53:44.275568 52310 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b'
I1129 09:53:44.277354 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.277768 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.277865 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.278280 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.278712 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.279321 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52361
I1129 09:53:44.280704 52321 cgroups_isolation_module.cpp:441] Launching 2 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b
I1129 09:53:44.293046 52321 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.293432 52321 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.293527 52321 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.293939 52321 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.294385 52321 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.294860 52321 cgroups_isolation_module.cpp:504] Forked executor at = 52364
I1129 09:53:44.532757 52323 slave.cpp:775] Got registration for executor '1' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.533000 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:44.533148 52323 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.533500 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.533584 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.534265 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.536386 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.536618 52311 slave.cpp:855] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.536713 52318 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.536906 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.537102 52289 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.537147 52318 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.537211 52318 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.537238 52289 master.hpp:297] Removing task 1 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.537382 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.537376 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:44.537499 52351 sched.cpp:326] Status update: task 1 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:44.537716 52351 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.537915 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.537931 52322 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.538000 52318 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.538142 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.538341 52319 slave.cpp:752] Got acknowledgement of status update for task 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.539729 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.594710 52313 slave.cpp:775] Got registration for executor '2' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.594856 52312 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:44.595021 52313 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.595425 52312 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.595496 52312 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.596107 52312 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.598393 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.598541 52315 slave.cpp:855] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.598624 52309 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:44.598870 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:44.598999 52309 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.599064 52309 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:44.599083 52280 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:44.599196 52280 master.hpp:297] Removing task 2 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.599321 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.599367 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:44.599483 52347 sched.cpp:326] Status update: task 2 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:44.599690 52347 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:44.599812 52309 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.599819 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.599839 52323 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.600172 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:44.600360 52321 slave.cpp:752] Got acknowledgement of status update for task 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:44.623421 52324 process.cpp:872] Socket closed while receiving
W1129 09:53:45.143110 52280 master.cpp:79] No whitelist given. Advertising offers for all slaves
I1129 09:53:45.147429 52308 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.147513 52308 cgroups_isolation_module.cpp:543] Killing executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.147531 52314 slave.cpp:1024] Executor '1' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
I1129 09:53:45.148080 52321 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/1/runs/e444812d-0f8e-4c6a-beaa-c2bcdc53c24c for removal
I1129 09:53:45.148236 52321 gc.cpp:125] Scheduling GC removal event to fire after 7.00days
I1129 09:53:45.148641 52308 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 1 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.148715 52308 cgroups_isolation_module.cpp:959] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
1 201211290953-2081170186-5432-52261-0000 { }
******************************************************************
I1129 09:53:45.148777 52308 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.148810 52308 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 1 of framework 201211290953-2081170186-5432-52261-0000 with tag 4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.149700 52309 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.149817 52309 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc after 1 attempts
I1129 09:53:45.151449 52317 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.151527 52317 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:45.147889 52276 hierarchical_allocator_process.hpp:665] No resources available to allocate!
I1129 09:53:45.167976 52276 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 20.11ms
I1129 09:53:45.147966 52279 master.cpp:1123] Executor 1 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
I1129 09:53:45.168719 52290 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:45.174438 52310 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_1_tag_4bc88390-7ad0-46b1-b8b5-229e544fc0bc
I1129 09:53:46.149343 52309 cgroups_isolation_module.cpp:605] Telling slave of terminated executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.149416 52309 cgroups_isolation_module.cpp:543] Killing executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.149461 52311 slave.cpp:1024] Executor '2' of framework 201211290953-2081170186-5432-52261-0000 has exited with status 0
I1129 09:53:46.149678 52311 gc.cpp:97] Scheduling /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/2/runs/a763863c-d45c-41da-b97c-75b98957439b for removal
I1129 09:53:46.149778 52279 master.cpp:1123] Executor 2 of framework 201211290953-2081170186-5432-52261-0000 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) exited with status 0
I1129 09:53:46.149931 52283 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.150512 52309 cgroups_isolation_module.cpp:949] Deallocating cpusets 0 for executor 2 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.150578 52309 cgroups_isolation_module.cpp:959] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
2 201211290953-2081170186-5432-52261-0000 { }
******************************************************************
I1129 09:53:46.150712 52309 cgroups_isolation_module.cpp:818] OOM notifier is triggered for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.150753 52309 cgroups_isolation_module.cpp:823] Discarded OOM notifier for executor 2 of framework 201211290953-2081170186-5432-52261-0000 with tag 53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.151849 52319 cgroups.cpp:1156] Trying to freeze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.151981 52319 cgroups.cpp:1195] Successfully froze cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789 after 1 attempts
I1129 09:53:46.153518 52309 cgroups.cpp:1171] Trying to thaw cgroup /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.153612 52309 cgroups.cpp:1278] Successfully thawed /cgroup/mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.168972 52284 hierarchical_allocator_process.hpp:657] Found available resources: cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.169070 52284 hierarchical_allocator_process.hpp:683] Offering cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0 to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.169142 52284 hierarchical_allocator_process.hpp:596] Performed allocation for 1 slaves in 192.05us
I1129 09:53:46.169292 52284 master.hpp:307] Adding offer201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.169360 52284 master.cpp:1262] Sending 1 offers to framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.169692 52353 sched.cpp:281] Received 1 offers
Resource offers received: 1
Starting task 3
Starting task 4
I1129 09:53:46.170192 52288 master.cpp:1463] Processing reply for offer 201211290953-2081170186-5432-52261-1 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com) for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.170331 52288 master.hpp:285] Adding task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.170371 52288 master.cpp:1580] Launching task 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:46.170524 52288 master.hpp:285] Adding task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.170580 52288 master.cpp:1580] Launching task 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 (smfd-aki-27-sr1.devel.twitter.com)
I1129 09:53:46.170658 52314 slave.cpp:477] Got assigned task 3 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.170697 52288 master.hpp:317] Removing offer 201211290953-2081170186-5432-52261-1 with resources cpus=2; mem=1024 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.171835 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
I1129 09:53:46.172061 52314 slave.cpp:477] Got assigned task 4 for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.173218 52323 cgroups_isolation_module.cpp:441] Launching 3 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_3_tag_9d1040aa-d38a-4702-971d-41d88052555c
I1129 09:53:46.173284 52314 paths.hpp:234] Created executor directory '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098
I1129 09:53:46.173493 52314 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/3/runs/b08c6e73-68ad-4a0d-88fa-d0a143828098'
I1129 09:53:46.173651 52313 slave.cpp:358] Successfully attached file '/tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36'
I1129 09:53:46.175964 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.176360 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.176440 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.176902 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.177340 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.177829 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52439
I1129 09:53:46.179039 52323 cgroups_isolation_module.cpp:441] Launching 4 (/home/vinod/mesos/build/src/cpuset-executor) in /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36 with resources cpus=0.5; mem=512 for framework 201211290953-2081170186-5432-52261-0000 in cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_4_tag_aee80a43-b55a-4d91-9885-0be61dd1da88
Fetching resources into /tmp/mesos/slaves/201211290953-2081170186-5432-52261-0/frameworks/201211290953-2081170186-5432-52261-0000/executors/4/runs/c6be1669-c466-49f3-8abc-dbda86c11b36
I1129 09:53:46.193050 52323 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.193454 52323 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.193539 52323 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.194010 52323 cgroups_isolation_module.cpp:773] Updated 'memory.limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.194481 52323 cgroups_isolation_module.cpp:799] Started listening for OOM events for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.194953 52323 cgroups_isolation_module.cpp:504] Forked executor at = 52442
I1129 09:53:46.195163 52323 cgroups_isolation_module.cpp:902] Successfully destroyed the cgroup mesos/framework_201211290953-2081170186-5432-52261-0000_executor_2_tag_53de5dcf-8141-4ab0-9f34-7a4e2d531789
I1129 09:53:46.242967 52310 slave.cpp:775] Got registration for executor '3' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.243160 52322 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:46.243348 52310 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.243744 52322 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.243871 52322 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.244596 52322 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.247010 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.247195 52319 slave.cpp:855] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.247309 52320 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 3 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.247477 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.247652 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.247760 52320 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.247808 52291 master.hpp:297] Removing task 3 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.247818 52320 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.247889 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:46.247928 52289 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.247969 52354 sched.cpp:326] Status update: task 3 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:46.248142 52354 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.248278 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.248327 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.248522 52312 slave.cpp:752] Got acknowledgement of status update for task 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.248623 52320 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 3 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.264420 52319 slave.cpp:775] Got registration for executor '4' of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.264575 52316 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=1; mem=512
I1129 09:53:46.264660 52319 slave.cpp:833] Flushing queued tasks for framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.265015 52316 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0,1 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.265108 52316 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5, CPU (id:1, core:1, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.265822 52316 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.266053 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.266679 52314 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.267032 52291 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
I1129 09:53:46.267441 52355 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_RUNNING
Task in state TASK_RUNNING
I1129 09:53:46.267706 52310 slave.cpp:855] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.267735 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.267887 52311 cgroups_isolation_module.cpp:582] Changing cgroup controls for executor 4 of framework 201211290953-2081170186-5432-52261-0000 with resources cpus=0.5; mem=512
I1129 09:53:46.267972 52310 slave.cpp:752] Got acknowledgement of status update for task 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.268178 52281 master.cpp:1022] Status update from slave(1)@10.35.12.124:58922: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
I1129 09:53:46.268337 52281 master.hpp:297] Removing task 4 with resources cpus=0.5 on slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.268324 52311 cgroups_isolation_module.cpp:691] Updated 'cpuset.cpus' to 0 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.268412 52311 cgroups_isolation_module.cpp:695] Cpusets allocation
******************************************************************
Executor Framework Allocation
******************************************************************
4 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
3 201211290953-2081170186-5432-52261-0000 { CPU (id:0, core:0, socket:0): 0.5 }
******************************************************************
I1129 09:53:46.268451 52281 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.268563 52352 sched.cpp:326] Status update: task 4 of framework 201211290953-2081170186-5432-52261-0000 is now in state TASK_FINISHED
Task in state TASK_FINISHED
I1129 09:53:46.268631 52352 sched.cpp:586] Asked to send framework message to slave 201211290953-2081170186-5432-52261-0
I1129 09:53:46.268764 52352 sched.cpp:421] Stopping framework '201211290953-2081170186-5432-52261-0000'
I1129 09:53:46.268933 52324 process.cpp:872] Socket closed while receiving
I1129 09:53:46.269044 52311 cgroups_isolation_module.cpp:773] Updated 'memory.soft_limit_in_bytes' to 536870912 for executor 4 of framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269086 52289 master.cpp:730] Asked to unregister framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269214 52291 hierarchical_allocator_process.hpp:357] Deactivated framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269321 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269405 52280 hierarchical_allocator_process.hpp:542] Recovered cpus=0.5; mem=512 on slave 201211290953-2081170186-5432-52261-0 from framework 201211290953-2081170186-5432-52261-0000
I1129 09:53:46.269484 52280 hierarchical_allocator_process.hpp:310] Removed framework 201211290953-2081170186-5432-52261-0000
W1129 09:53:46.269382 52320 slave.cpp:671] Ignoring shutdown framework message from master@10.35.12.124:5432because it is not from the registered master (master@127.0.0.1:5432)
I1129 09:53:46.270210 52292 process.cpp:872] Socket closed while receiving
I1129 09:53:46.270334 52292 process.cpp:872] Socket closed while receiving
/home/vinod/mesos/support/atexit.sh: line 11: 52261 Terminated ${MESOS_BUILD_DIR}/src/mesos-master --port=5432
/home/vinod/mesos/support/atexit.sh: line 11: 52293 Terminated ${MESOS_BUILD_DIR}/src/mesos-slave --master=localhost:5432 --isolation=cgroups --cgroups_hierarchy_root=${HIERARCHY} --cgroups_subsystems="cpuset,memory,freezer" --resources="cpus:2;mem:1024"
[ OK ] CgroupsCpusetIsolationTest.ROOT_CGROUPS_CpusetFramework (6215 ms)
[----------] 1 test from CgroupsCpusetIsolationTest (6215 ms total)
[----------] 5 tests from CgroupsCpusetTest
[ RUN ] CgroupsCpusetTest.OneCPUOneCpuset
[ OK ] CgroupsCpusetTest.OneCPUOneCpuset (0 ms)
[ RUN ] CgroupsCpusetTest.OneCPUManyCpusets
[ OK ] CgroupsCpusetTest.OneCPUManyCpusets (0 ms)
[ RUN ] CgroupsCpusetTest.ManyCPUOneCpuset
[ OK ] CgroupsCpusetTest.ManyCPUOneCpuset (0 ms)
[ RUN ] CgroupsCpusetTest.ManyCPUManyCpusets
[ OK ] CgroupsCpusetTest.ManyCPUManyCpusets (0 ms)
[ RUN ] CgroupsCpusetTest.IntegerAllocations
[ OK ] CgroupsCpusetTest.IntegerAllocations (0 ms)
[----------] 5 tests from CgroupsCpusetTest (0 ms total)
[----------] Global test environment tear-down
[==========] 6 tests from 2 test cases ran. (6216 ms total)
[ PASSED ] 6 tests.
Thanks,
Vinod Kone