You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Muhuan Huang <mh...@cs.ucla.edu> on 2014/03/16 06:14:06 UTC
slave keeps getting asked to shut down frameworks
Hello there!
I'm trying to test mesos on two machines within a LAN. I launch a master on
one machine and a slave on the other. When I try to launch the example
framework (test-framework) on the machine where the slave resides in, it
hangs. The slave keeps getting asked to shut down unknown frameworks (with
increasing id).
Could anyone help to explain what's going on?
I have mesos-0.17.0 and CentOS installed.
Muhuan
Here's the output from framework:
-------------------------------------------------
./src/examples/python/test-framework 10.0.0.8:5050
I0315 13:43:12.697850 2504 sched.cpp:218] No credentials provided.
Attempting to register without authentication
I0315 13:43:12.698140 2504 sched.cpp:230] Detecting new master
output from slave
-------------------------------------------------
[root@computer4 build]# ./bin/mesos-slave.sh --master=10.0.0.8:5050
I0315 13:41:24.226071 2440 main.cpp:118] Build: 2014-03-07 07:39:55 by root
I0315 13:41:24.226307 2440 main.cpp:120] Creating "process" isolator
I0315 13:41:24.227330 2440 main.cpp:132] Starting Mesos slave
I0315 13:41:24.228863 2460 slave.cpp:111] Slave started on 1)@10.0.0.6:5051
I0315 13:41:24.229421 2460 slave.cpp:211] Slave resources: cpus(*):2;
mem(*):2789; disk(*):45276; ports(*):[31000-32000]
I0315 13:41:24.235052 2456 state.cpp:33] Recovering state from
'/tmp/mesos/meta'
I0315 13:41:24.236737 2456 status_update_manager.cpp:188] Recovering
status update manager
I0315 13:41:24.237052 2463 process_isolator.cpp:317] Recovering isolator
I0315 13:41:24.237324 2463 slave.cpp:2752] Finished recovery
I0315 13:41:24.238035 2463 slave.cpp:2784] Garbage collecting old slave
201403152002-134217738-5050-28509-0
I0315 13:41:24.238256 2461 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/201403152002-134217738-5050-28509-0' for gc
6.99999724372741days in the future
I0315 13:41:24.238311 2463 slave.cpp:2784] Garbage collecting old slave
201403050752-2950738819-5050-1262-0
I0315 13:41:24.238489 2461 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/201403152002-134217738-5050-28509-0' for gc
6.99999724218667days in the future
I0315 13:41:24.238534 2463 slave.cpp:2784] Garbage collecting old slave
201403041402-16777343-5050-8512-0
I0315 13:41:24.238579 2461 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/201403050752-2950738819-5050-1262-0' for gc
6.99999724048593days in the future
I0315 13:41:24.238670 2461 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/201403050752-2950738819-5050-1262-0' for gc
6.99999723967704days in the future
I0315 13:41:24.238698 2463 slave.cpp:2784] Garbage collecting old slave
201403071310-16777343-5050-28263-0
I0315 13:41:24.238764 2461 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/201403041402-16777343-5050-8512-0' for gc
6.99999723857778days in the future
I0315 13:41:24.238837 2461 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/201403041402-16777343-5050-8512-0' for gc
6.99999723786963days in the future
I0315 13:41:24.238873 2463 slave.cpp:2784] Garbage collecting old slave
201403110754-2950738819-5050-1396-0
I0315 13:41:24.238911 2461 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/201403071310-16777343-5050-28263-0' for gc
6.99999723667852days in the future
I0315 13:41:24.239322 2461 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/201403071310-16777343-5050-28263-0' for gc
6.99999723581037days in the future
I0315 13:41:24.239413 2461 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/201403110754-2950738819-5050-1396-0' for gc
6.99999723348444days in the future
I0315 13:41:24.239478 2461 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/201403110754-2950738819-5050-1396-0' for gc
6.99999723250074days in the future
I0315 13:41:24.239799 2462 slave.cpp:500] New master detected at
master@10.0.0.8:5050
I0315 13:41:24.240231 2461 status_update_manager.cpp:162] New master
detected at master@10.0.0.8:5050
I0315 13:41:24.240411 2462 slave.cpp:525] Detecting new master
I0315 13:41:24.242296 2461 slave.cpp:593] Re-registered with master
master@10.0.0.8:5050
I0315 13:42:24.231416 2461 slave.cpp:2602] Current usage 9.32%. Max
allowed age: 5.647927204010996days
I0315 13:43:12.699659 2459 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0000 by master@10.0.0.8:5050
W0315 13:43:12.699748 2459 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0000
I0315 13:43:13.699810 2459 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0001 by master@10.0.0.8:5050
W0315 13:43:13.699880 2459 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0001
I0315 13:43:14.700316 2457 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0002 by master@10.0.0.8:5050
W0315 13:43:14.700397 2457 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0002
I0315 13:43:15.700831 2461 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0003 by master@10.0.0.8:5050
W0315 13:43:15.700930 2461 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0003
I0315 13:43:16.701084 2463 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0004 by master@10.0.0.8:5050
W0315 13:43:16.701154 2463 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0004
I0315 13:43:17.701400 2456 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0005 by master@10.0.0.8:5050
W0315 13:43:17.701467 2456 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0005
I0315 13:43:18.701666 2456 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0006 by master@10.0.0.8:5050
W0315 13:43:18.701738 2456 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0006
I0315 13:43:19.701906 2462 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0007 by master@10.0.0.8:5050
W0315 13:43:19.701979 2462 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0007
I0315 13:43:20.702301 2458 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0008 by master@10.0.0.8:5050
W0315 13:43:20.702368 2458 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0008
I0315 13:43:21.702569 2458 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0009 by master@10.0.0.8:5050
W0315 13:43:21.702638 2458 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0009
I0315 13:43:22.702926 2460 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0010 by master@10.0.0.8:5050
W0315 13:43:22.702997 2460 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0010
I0315 13:43:23.703258 2459 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0011 by master@10.0.0.8:5050
W0315 13:43:23.703331 2459 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0011
I0315 13:43:24.231906 2457 slave.cpp:2602] Current usage 9.32%. Max
allowed age: 5.647927204010996days
I0315 13:43:24.703522 2456 slave.cpp:1134] Asked to shut down framework
201403152138-134217738-5050-30706-0012 by master@10.0.0.8:5050
W0315 13:43:24.703595 2456 slave.cpp:1149] Cannot shut down unknown
framework 201403152138-134217738-5050-30706-0012
...
Re: slave keeps getting asked to shut down frameworks
Posted by Vinod Kone <vi...@twitter.com>.
the iptable settings, if needed, should be on the receiving end points in
the mesos cluster. in other words you should accept all incoming
connections to port 5050 (for master) and 5051 (for slave).
@vinodkone
On Mon, Mar 17, 2014 at 3:22 PM, Muhuan Huang <mh...@cs.ucla.edu> wrote:
> The problem is solved after I disable the iptables on the slave node. My
> previous iptables setting has been rejecting packages from port other than
> 5050 and 5051.
>
> What's the standard iptables setting for mesos? Are there specific ranges
> of the ports that I should leave open ("ACCEPT"), or shall I just leave all
> the ports open?
>
> Muhuan
>
>
>
>
> On Mon, Mar 17, 2014 at 2:34 PM, Muhuan Huang <mh...@cs.ucla.edu> wrote:
>
>> Good afternoon Vinod,
>>
>> I just ran it again and here's the output.
>>
>> Thanks in advance!
>>
>> Muhuan
>>
>> -------------------------------------
>> framework logs
>>
>> ./src/test-framework --master=10.0.0.1:5050
>> I0317 06:29:48.240273 2204 sched.cpp:218] No credentials provided.
>> Attempting to register without authentication
>> I0317 06:29:48.240890 2204 sched.cpp:230] Detecting new master
>> ^C
>>
>> -------------------------------------
>> master logs
>>
>> ./bin/mesos-master.sh --ip=10.0.0.1
>> I0317 14:26:02.114706 26276 main.cpp:126] Build: 2014-03-15 10:01:48 by
>> mhhuang
>> I0317 14:26:02.115300 26276 main.cpp:167] Starting Mesos master
>> I0317 14:26:02.115432 26276 master.cpp:284] Master started on
>> 10.0.0.1:5050
>> I0317 14:26:02.115481 26276 master.cpp:299] Master ID:
>> 201403171426-16777226-5050-26276
>> I0317 14:26:02.115490 26276 master.cpp:309] Master allowing
>> unauthenticated frameworks to register!!
>> I0317 14:26:02.118011 26294 master.cpp:734] The newly elected leader is
>> master@10.0.0.1:5050
>> I0317 14:26:02.118055 26294 master.cpp:742] Elected as the leading master!
>> I0317 14:26:17.713840 26292 master.cpp:1812] Attempting to register slave
>> on computer4 at slave(1)@10.0.0.6:5051
>> I0317 14:26:17.713894 26292 master.cpp:2782] Adding slave
>> 201403171426-16777226-5050-26276-0 at computer4 with cpus(*):2;
>> mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> I0317 14:26:17.714452 26295 hierarchical_allocator_process.hpp:445] Added
>> slave 201403171426-16777226-5050-26276-0 (computer4) with cpus(*):2;
>> mem(*):2789; disk(*):25118; ports(*):[31000-32000] (and cpus(*):2;
>> mem(*):2789; disk(*):25118; ports(*):[31000-32000] available)
>> I0317 14:27:51.226738 26291 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:27:51.226831 26291 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0000 at scheduler(1)@10.0.0.6:52223
>> I0317 14:27:51.227170 26290 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0000
>> I0317 14:27:51.227221 26293 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0000 disconnected
>> I0317 14:27:51.227246 26293 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0000
>> I0317 14:27:51.227285 26293 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0000 0ns to failover
>> I0317 14:27:51.227421 26290 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0000
>> I0317 14:27:51.227427 26291 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0000
>> I0317 14:27:51.227480 26291 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0000
>> W0317 14:27:51.227589 26291 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0000 because the
>> framework has terminated or is inactive
>> I0317 14:27:51.227661 26297 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0000
>> I0317 14:27:51.227730 26297 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0000
>> I0317 14:27:52.227407 26290 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:27:52.227494 26290 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0001 at scheduler(1)@10.0.0.6:52223
>> I0317 14:27:52.227632 26290 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0001
>> I0317 14:27:52.227790 26291 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0001 disconnected
>> I0317 14:27:52.227815 26291 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0001
>> I0317 14:27:52.227866 26291 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0001 0ns to failover
>> I0317 14:27:52.227888 26294 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0001
>> W0317 14:27:52.227896 26291 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0001 because the
>> framework has terminated or is inactive
>> I0317 14:27:52.227998 26291 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0001
>> I0317 14:27:52.228010 26291 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0001
>> I0317 14:27:52.228111 26290 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0001
>> I0317 14:27:52.228157 26290 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0001
>> I0317 14:27:53.227839 26297 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:27:53.227886 26297 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0002 at scheduler(1)@10.0.0.6:52223
>> I0317 14:27:53.228054 26297 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0002
>> I0317 14:27:53.228199 26292 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0002 disconnected
>> I0317 14:27:53.228245 26292 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0002
>> I0317 14:27:53.228288 26292 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0002 0ns to failover
>> I0317 14:27:53.228313 26297 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0002
>> W0317 14:27:53.228323 26292 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0002 because the
>> framework has terminated or is inactive
>> I0317 14:27:53.228410 26292 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0002
>> I0317 14:27:53.228436 26292 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0002
>> I0317 14:27:53.228615 26296 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0002
>> I0317 14:27:53.228698 26296 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0002
>> I0317 14:27:54.228085 26295 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:27:54.228142 26295 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0003 at scheduler(1)@10.0.0.6:52223
>> I0317 14:27:54.228312 26295 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0003
>> I0317 14:27:54.228427 26290 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0003 disconnected
>> I0317 14:27:54.228456 26290 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0003
>> I0317 14:27:54.228479 26290 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0003 0ns to failover
>> I0317 14:27:54.228509 26295 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0003
>> W0317 14:27:54.228515 26290 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0003 because the
>> framework has terminated or is inactive
>> I0317 14:27:54.228618 26290 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0003
>> I0317 14:27:54.228627 26290 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0003
>> I0317 14:27:54.228761 26293 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0003
>> I0317 14:27:54.228834 26293 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0003
>> I0317 14:27:55.228446 26291 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:27:55.228588 26291 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0004 at scheduler(1)@10.0.0.6:52223
>> I0317 14:27:55.228703 26291 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0004
>> I0317 14:27:55.228940 26297 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0004 disconnected
>> I0317 14:27:55.228965 26297 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0004
>> I0317 14:27:55.228994 26297 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0004 0ns to failover
>> I0317 14:27:55.229010 26291 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0004
>> W0317 14:27:55.229023 26297 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0004 because the
>> framework has terminated or is inactive
>> I0317 14:27:55.229115 26297 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0004
>> I0317 14:27:55.229126 26297 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0004
>> I0317 14:27:55.229244 26295 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0004
>> I0317 14:27:55.229326 26295 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0004
>> I0317 14:27:56.228770 26294 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:27:56.228827 26294 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0005 at scheduler(1)@10.0.0.6:52223
>> I0317 14:27:56.229012 26294 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0005
>> I0317 14:27:56.229126 26293 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0005 disconnected
>> I0317 14:27:56.229146 26293 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0005
>> I0317 14:27:56.229163 26293 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0005 0ns to failover
>> W0317 14:27:56.229202 26293 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0005 because the
>> framework has terminated or is inactive
>> I0317 14:27:56.229204 26294 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0005
>> I0317 14:27:56.229360 26293 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0005
>> I0317 14:27:56.229449 26290 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0005
>> I0317 14:27:56.229465 26290 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0005
>> I0317 14:27:56.229558 26290 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0005
>> I0317 14:27:57.229128 26291 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:27:57.229202 26291 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0006 at scheduler(1)@10.0.0.6:52223
>> I0317 14:27:57.229369 26296 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0006
>> I0317 14:27:57.229470 26291 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0006 disconnected
>> I0317 14:27:57.229481 26291 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0006
>> I0317 14:27:57.229509 26291 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0006 0ns to failover
>> W0317 14:27:57.229567 26291 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0006 because the
>> framework has terminated or is inactive
>> I0317 14:27:57.229578 26296 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0006
>> I0317 14:27:57.229683 26291 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0006
>> I0317 14:27:57.229693 26291 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0006
>> I0317 14:27:57.229773 26294 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0006
>> I0317 14:27:57.229827 26294 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0006
>> I0317 14:27:58.229428 26293 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:27:58.229504 26293 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0007 at scheduler(1)@10.0.0.6:52223
>> I0317 14:27:58.229611 26293 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0007
>> I0317 14:27:58.229749 26297 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0007 disconnected
>> I0317 14:27:58.229766 26297 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0007
>> I0317 14:27:58.229797 26297 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0007 0ns to failover
>> I0317 14:27:58.229809 26293 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0007
>> W0317 14:27:58.229827 26297 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0007 because the
>> framework has terminated or is inactive
>> I0317 14:27:58.229934 26297 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0007
>> I0317 14:27:58.229946 26297 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0007
>> I0317 14:27:58.230026 26296 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0007
>> I0317 14:27:58.230103 26296 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0007
>> I0317 14:27:59.229773 26292 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:27:59.229818 26292 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0008 at scheduler(1)@10.0.0.6:52223
>> I0317 14:27:59.229957 26292 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0008
>> I0317 14:27:59.230077 26297 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0008 disconnected
>> I0317 14:27:59.230093 26297 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0008
>> I0317 14:27:59.230129 26297 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0008 0ns to failover
>> I0317 14:27:59.230204 26292 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0008
>> W0317 14:27:59.230203 26290 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0008 because the
>> framework has terminated or is inactive
>> I0317 14:27:59.230330 26290 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0008
>> I0317 14:27:59.230340 26290 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0008
>> I0317 14:27:59.230438 26296 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0008
>> I0317 14:27:59.230492 26296 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0008
>> I0317 14:28:00.230130 26290 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:28:00.230202 26290 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0009 at scheduler(1)@10.0.0.6:52223
>> I0317 14:28:00.230335 26290 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0009
>> I0317 14:28:00.230501 26296 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0009 disconnected
>> I0317 14:28:00.230520 26296 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0009
>> I0317 14:28:00.230540 26296 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0009 0ns to failover
>> W0317 14:28:00.230590 26296 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0009 because the
>> framework has terminated or is inactive
>> I0317 14:28:00.230599 26290 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0009
>> I0317 14:28:00.230679 26296 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0009
>> I0317 14:28:00.230690 26296 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0009
>> I0317 14:28:00.230751 26295 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0009
>> I0317 14:28:00.230808 26295 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0009
>> I0317 14:28:01.230473 26295 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:28:01.230556 26295 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0010 at scheduler(1)@10.0.0.6:52223
>> I0317 14:28:01.230681 26291 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0010
>> I0317 14:28:01.230865 26295 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0010 disconnected
>> I0317 14:28:01.230890 26295 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0010
>> I0317 14:28:01.230913 26295 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0010 0ns to failover
>> I0317 14:28:01.230937 26291 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0010
>> W0317 14:28:01.230980 26295 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0010 because the
>> framework has terminated or is inactive
>> I0317 14:28:01.231137 26295 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0010
>> I0317 14:28:01.231160 26295 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0010
>> I0317 14:28:01.231288 26297 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0010
>> I0317 14:28:01.231353 26297 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0010
>> I0317 14:28:02.230877 26294 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:28:02.230938 26294 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0011 at scheduler(1)@10.0.0.6:52223
>> I0317 14:28:02.231088 26294 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0011
>> I0317 14:28:02.231173 26291 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0011 disconnected
>> I0317 14:28:02.231201 26291 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0011
>> I0317 14:28:02.231220 26291 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0011 0ns to failover
>> I0317 14:28:02.231305 26294 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0011
>> I0317 14:28:02.231310 26290 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0011
>> I0317 14:28:02.231328 26290 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0011
>> W0317 14:28:02.231406 26290 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0011 because the
>> framework has terminated or is inactive
>> I0317 14:28:02.231529 26296 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0011
>> I0317 14:28:02.231581 26296 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0011
>> I0317 14:28:03.231256 26290 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:28:03.231305 26290 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0012 at scheduler(1)@10.0.0.6:52223
>> I0317 14:28:03.231448 26290 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0012
>> I0317 14:28:03.231561 26296 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0012 disconnected
>> I0317 14:28:03.231578 26296 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0012
>> I0317 14:28:03.231601 26296 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0012 0ns to failover
>> I0317 14:28:03.231703 26293 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0012
>> I0317 14:28:03.231703 26290 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0012
>> I0317 14:28:03.231745 26293 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0012
>> W0317 14:28:03.231829 26293 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0012 because the
>> framework has terminated or is inactive
>> I0317 14:28:03.231923 26291 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0012
>> I0317 14:28:03.231979 26291 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0012
>> I0317 14:28:04.231626 26296 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:28:04.231684 26296 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0013 at scheduler(1)@10.0.0.6:52223
>> I0317 14:28:04.231876 26296 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0013
>> I0317 14:28:04.231951 26293 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0013 disconnected
>> I0317 14:28:04.231966 26293 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0013
>> I0317 14:28:04.231982 26293 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0013 0ns to failover
>> I0317 14:28:04.232044 26291 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0013
>> I0317 14:28:04.232067 26291 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0013
>> I0317 14:28:04.232121 26296 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0013
>> W0317 14:28:04.232131 26291 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0013 because the
>> framework has terminated or is inactive
>> I0317 14:28:04.232197 26296 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0013
>> I0317 14:28:04.232234 26296 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0013
>> I0317 14:28:05.231998 26295 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:28:05.232053 26295 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0014 at scheduler(1)@10.0.0.6:52223
>> I0317 14:28:05.232189 26295 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0014
>> I0317 14:28:05.232316 26293 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0014 disconnected
>> I0317 14:28:05.232342 26293 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0014
>> I0317 14:28:05.232370 26293 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0014 0ns to failover
>> W0317 14:28:05.232440 26294 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0014 because the
>> framework has terminated or is inactive
>> I0317 14:28:05.232442 26295 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0014
>> I0317 14:28:05.232537 26294 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0014
>> I0317 14:28:05.232549 26294 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0014
>> I0317 14:28:05.232631 26292 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0014
>> I0317 14:28:05.232679 26292 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0014
>> I0317 14:28:06.232396 26297 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:28:06.232435 26297 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0015 at scheduler(1)@10.0.0.6:52223
>> I0317 14:28:06.232583 26297 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0015
>> I0317 14:28:06.232739 26293 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0015 disconnected
>> I0317 14:28:06.232764 26293 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0015
>> I0317 14:28:06.232803 26293 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0015 0ns to failover
>> I0317 14:28:06.232846 26296 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0015
>> W0317 14:28:06.232864 26293 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0015 because the
>> framework has terminated or is inactive
>> I0317 14:28:06.232976 26293 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0015
>> I0317 14:28:06.232996 26293 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0015
>> I0317 14:28:06.233083 26295 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0015
>> I0317 14:28:06.233134 26295 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0015
>> I0317 14:28:07.232770 26294 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:28:07.232810 26294 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0016 at scheduler(1)@10.0.0.6:52223
>> I0317 14:28:07.232951 26294 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0016
>> I0317 14:28:07.233062 26297 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0016 disconnected
>> I0317 14:28:07.233078 26297 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0016
>> I0317 14:28:07.233098 26297 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0016 0ns to failover
>> W0317 14:28:07.233137 26297 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0016 because the
>> framework has terminated or is inactive
>> I0317 14:28:07.233146 26294 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0016
>> I0317 14:28:07.233223 26297 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0016
>> I0317 14:28:07.233237 26297 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0016
>> I0317 14:28:07.233338 26291 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0016
>> I0317 14:28:07.233405 26291 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0016
>> I0317 14:28:08.233158 26297 master.cpp:790] Received registration request
>> from scheduler(1)@10.0.0.6:52223
>> I0317 14:28:08.233260 26297 master.cpp:808] Registering framework
>> 201403171426-16777226-5050-26276-0017 at scheduler(1)@10.0.0.6:52223
>> I0317 14:28:08.233382 26297 hierarchical_allocator_process.hpp:332] Added
>> framework 201403171426-16777226-5050-26276-0017
>> I0317 14:28:08.233528 26291 master.cpp:569] Framework
>> 201403171426-16777226-5050-26276-0017 disconnected
>> I0317 14:28:08.233544 26291 master.cpp:1041] Deactivating framework
>> 201403171426-16777226-5050-26276-0017
>> I0317 14:28:08.233574 26291 master.cpp:591] Giving framework
>> 201403171426-16777226-5050-26276-0017 0ns to failover
>> W0317 14:28:08.233605 26291 master.cpp:2178] Master returning resources
>> offered to framework 201403171426-16777226-5050-26276-0017 because the
>> framework has terminated or is inactive
>> I0317 14:28:08.233629 26297 hierarchical_allocator_process.hpp:408]
>> Deactivated framework 201403171426-16777226-5050-26276-0017
>> I0317 14:28:08.233739 26291 master.cpp:2166] Framework failover timeout,
>> removing framework 201403171426-16777226-5050-26276-0017
>> I0317 14:28:08.233752 26291 master.cpp:2654] Removing framework
>> 201403171426-16777226-5050-26276-0017
>> I0317 14:28:08.233839 26292 hierarchical_allocator_process.hpp:637]
>> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
>> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
>> framework 201403171426-16777226-5050-26276-0017
>> I0317 14:28:08.233880 26292 hierarchical_allocator_process.hpp:363]
>> Removed framework 201403171426-16777226-5050-26276-0017
>>
>>
>> ---------------------------------
>> slave logs
>> ./bin/mesos-slave.sh --master=10.0.0.1:5050
>> I0317 06:28:14.692621 2102 main.cpp:118] Build: 2014-03-15 10:01:48 by
>> mhhuang
>> I0317 06:28:14.692896 2102 main.cpp:120] Creating "process" isolator
>> I0317 06:28:14.693625 2102 main.cpp:132] Starting Mesos slave
>> I0317 06:28:14.694778 2122 slave.cpp:111] Slave started on 1)@
>> 10.0.0.6:5051
>> I0317 06:28:14.698225 2122 slave.cpp:211] Slave resources: cpus(*):2;
>> mem(*):2789; disk(*):25118; ports(*):[31000-32000]
>> I0317 06:28:14.703965 2124 state.cpp:33] Recovering state from
>> '/curr/mhhuang/tmp/mesos_work_dir/meta'
>> I0317 06:28:14.708154 2124 state.cpp:50] Slave host rebooted
>> I0317 06:28:14.708582 2118 status_update_manager.cpp:188] Recovering
>> status update manager
>> I0317 06:28:14.708757 2118 process_isolator.cpp:317] Recovering isolator
>> I0317 06:28:14.708938 2118 slave.cpp:2752] Finished recovery
>> I0317 06:28:14.718215 2118 slave.cpp:2784] Garbage collecting old slave
>> 201403152128-134217738-5050-30225-0
>> I0317 06:28:14.720899 2119 gc.cpp:56] Scheduling
>> '/curr/mhhuang/tmp/mesos_work_dir/slaves/201403152128-134217738-5050-30
>> 225-0' for gc 1.04742440377312weeks in the future
>> I0317 06:28:14.724704 2118 slave.cpp:2784] Garbage collecting old slave
>> 201403161523-1004581763-5050-28848-0
>> I0317 06:28:14.724864 2124 gc.cpp:56] Scheduling
>> '/curr/mhhuang/tmp/mesos_work_dir/meta/slaves/201403152128-134217738-50
>> 50-30225-0' for gc 1.04742439720085weeks in the future
>> I0317 06:28:14.726724 2124 gc.cpp:56] Scheduling
>> '/curr/mhhuang/tmp/mesos_work_dir/slaves/201403161523-1004581763-5050-2
>> 8848-0' for gc 1.0474243938146weeks in the future
>> I0317 06:28:14.729722 2120 gc.cpp:56] Scheduling
>> '/curr/mhhuang/tmp/mesos_work_dir/meta/slaves/201403161523-1004581763-5
>> 050-28848-0' for gc 1.04742438890413weeks in the future
>> I0317 06:28:14.730007 2121 slave.cpp:500] New master detected at
>> master@10.0.0.1:5050
>> I0317 06:28:14.730569 2121 slave.cpp:525] Detecting new master
>> I0317 06:28:14.730666 2121 status_update_manager.cpp:162] New master
>> detected at master@10.0.0.1:5050
>> I0317 06:28:14.732645 2121 slave.cpp:543] Registered with master
>> master@10.0.0.1:5050; given slave ID 201403171426-16777
>> 226-5050-26276-0
>> I0317 06:28:14.815829 2121 slave.cpp:556] Checkpointing SlaveInfo to
>> '/curr/mhhuang/tmp/mesos_work_dir/meta/slaves/20140
>> 3171426-16777226-5050-26276-0/slave.info'
>> I0317 06:29:14.701341 2118 slave.cpp:2602] Current usage 81.49%. Max
>> allowed age: 14.289382564984445hrs
>> I0317 06:29:48.242293 2120 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0000 by master@ 10.0.0.1:5050
>> W0317 06:29:48.242368 2120 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0000
>> I0317 06:29:49.242784 2120 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0001 by master@ 10.0.0.1:5050
>> W0317 06:29:49.242876 2120 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0001
>> I0317 06:29:50.243255 2119 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0002 by master@ 10.0.0.1:5050
>> W0317 06:29:50.243334 2119 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0002
>> I0317 06:29:51.243321 2125 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0003 by master@ 10.0.0.1:5050
>> W0317 06:29:51.243391 2125 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0003
>> I0317 06:29:52.243748 2118 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0004 by master@ 10.0.0.1:5050
>> W0317 06:29:52.243816 2118 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0004
>> I0317 06:29:53.244084 2118 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0005 by master@ 10.0.0.1:5050
>> W0317 06:29:53.244160 2118 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0005
>> I0317 06:29:54.244231 2123 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0006 by master@ 10.0.0.1:5050
>> W0317 06:29:54.244303 2123 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0006
>> I0317 06:29:55.244420 2121 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0007 by master@ 10.0.0.1:5050
>> W0317 06:29:55.244485 2121 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0007
>> I0317 06:29:56.244861 2121 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0008 by master@ 10.0.0.1:5050
>> W0317 06:29:56.244936 2121 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0008
>> I0317 06:29:57.245123 2124 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0009 by master@ 10.0.0.1:5050
>> W0317 06:29:57.245193 2124 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0009
>> I0317 06:29:58.245638 2122 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0010 by master@ 10.0.0.1:5050
>> W0317 06:29:58.245734 2122 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0010
>> I0317 06:29:59.245710 2118 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0011 by master@ 10.0.0.1:5050
>> W0317 06:29:59.245775 2118 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0011
>> I0317 06:30:00.246095 2120 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0012 by master@ 10.0.0.1:5050
>> W0317 06:30:00.246172 2120 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0012
>> I0317 06:30:01.246351 2125 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0013 by master@ 10.0.0.1:5050
>> W0317 06:30:01.246423 2125 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0013
>> I0317 06:30:02.246826 2124 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0014 by master@ 10.0.0.1:5050
>> W0317 06:30:02.246893 2124 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0014
>> I0317 06:30:03.247225 2119 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0015 by master@ 10.0.0.1:5050
>> W0317 06:30:03.247293 2119 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0015
>> I0317 06:30:04.247455 2123 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0016 by master@ 10.0.0.1:5050
>> W0317 06:30:04.247527 2123 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0016
>> I0317 06:30:05.247911 2124 slave.cpp:1134] Asked to shut down framework
>> 201403171426-16777226-5050-26276-0017 by master@ 10.0.0.1:5050
>> W0317 06:30:05.247989 2124 slave.cpp:1149] Cannot shut down unknown
>> framework 201403171426-16777226-5050-26276-0017
>> I0317 06:30:14.703604 2124 slave.cpp:2602] Current usage 81.49%. Max
>> allowed age: 14.289382564984445hrs
>>
>>
>>
>>
>> On Mon, Mar 17, 2014 at 12:19 PM, Vinod Kone <vi...@gmail.com> wrote:
>>
>>> What do the master logs say?
>>>
>>>
>>> On Sat, Mar 15, 2014 at 10:14 PM, Muhuan Huang <mh...@cs.ucla.edu>wrote:
>>>
>>>> Hello there!
>>>>
>>>> I'm trying to test mesos on two machines within a LAN. I launch a
>>>> master on one machine and a slave on the other. When I try to launch the
>>>> example framework (test-framework) on the machine where the slave resides
>>>> in, it hangs. The slave keeps getting asked to shut down unknown frameworks
>>>> (with increasing id).
>>>>
>>>> Could anyone help to explain what's going on?
>>>>
>>>> I have mesos-0.17.0 and CentOS installed.
>>>>
>>>> Muhuan
>>>>
>>>>
>>>>
>>>>
>>>> Here's the output from framework:
>>>> -------------------------------------------------
>>>> ./src/examples/python/test-framework 10.0.0.8:5050
>>>> I0315 13:43:12.697850 2504 sched.cpp:218] No credentials provided.
>>>> Attempting to register without authentication
>>>> I0315 13:43:12.698140 2504 sched.cpp:230] Detecting new master
>>>>
>>>>
>>>>
>>>> output from slave
>>>> -------------------------------------------------
>>>> [root@computer4 build]# ./bin/mesos-slave.sh --master=10.0.0.8:5050
>>>> I0315 13:41:24.226071 2440 main.cpp:118] Build: 2014-03-07 07:39:55 by
>>>> root
>>>> I0315 13:41:24.226307 2440 main.cpp:120] Creating "process" isolator
>>>> I0315 13:41:24.227330 2440 main.cpp:132] Starting Mesos slave
>>>> I0315 13:41:24.228863 2460 slave.cpp:111] Slave started on 1)@
>>>> 10.0.0.6:5051
>>>> I0315 13:41:24.229421 2460 slave.cpp:211] Slave resources: cpus(*):2;
>>>> mem(*):2789; disk(*):45276; ports(*):[31000-32000]
>>>> I0315 13:41:24.235052 2456 state.cpp:33] Recovering state from
>>>> '/tmp/mesos/meta'
>>>> I0315 13:41:24.236737 2456 status_update_manager.cpp:188] Recovering
>>>> status update manager
>>>> I0315 13:41:24.237052 2463 process_isolator.cpp:317] Recovering
>>>> isolator
>>>> I0315 13:41:24.237324 2463 slave.cpp:2752] Finished recovery
>>>> I0315 13:41:24.238035 2463 slave.cpp:2784] Garbage collecting old
>>>> slave 201403152002-134217738-5050-28509-0
>>>> I0315 13:41:24.238256 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/slaves/201403152002-134217738-5050-28509-0' for gc
>>>> 6.99999724372741days in the future
>>>> I0315 13:41:24.238311 2463 slave.cpp:2784] Garbage collecting old
>>>> slave 201403050752-2950738819-5050-1262-0
>>>> I0315 13:41:24.238489 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/meta/slaves/201403152002-134217738-5050-28509-0' for gc
>>>> 6.99999724218667days in the future
>>>> I0315 13:41:24.238534 2463 slave.cpp:2784] Garbage collecting old
>>>> slave 201403041402-16777343-5050-8512-0
>>>> I0315 13:41:24.238579 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/slaves/201403050752-2950738819-5050-1262-0' for gc
>>>> 6.99999724048593days in the future
>>>> I0315 13:41:24.238670 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/meta/slaves/201403050752-2950738819-5050-1262-0' for gc
>>>> 6.99999723967704days in the future
>>>> I0315 13:41:24.238698 2463 slave.cpp:2784] Garbage collecting old
>>>> slave 201403071310-16777343-5050-28263-0
>>>> I0315 13:41:24.238764 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/slaves/201403041402-16777343-5050-8512-0' for gc
>>>> 6.99999723857778days in the future
>>>> I0315 13:41:24.238837 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/meta/slaves/201403041402-16777343-5050-8512-0' for gc
>>>> 6.99999723786963days in the future
>>>> I0315 13:41:24.238873 2463 slave.cpp:2784] Garbage collecting old
>>>> slave 201403110754-2950738819-5050-1396-0
>>>> I0315 13:41:24.238911 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/slaves/201403071310-16777343-5050-28263-0' for gc
>>>> 6.99999723667852days in the future
>>>> I0315 13:41:24.239322 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/meta/slaves/201403071310-16777343-5050-28263-0' for gc
>>>> 6.99999723581037days in the future
>>>> I0315 13:41:24.239413 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/slaves/201403110754-2950738819-5050-1396-0' for gc
>>>> 6.99999723348444days in the future
>>>> I0315 13:41:24.239478 2461 gc.cpp:56] Scheduling
>>>> '/tmp/mesos/meta/slaves/201403110754-2950738819-5050-1396-0' for gc
>>>> 6.99999723250074days in the future
>>>> I0315 13:41:24.239799 2462 slave.cpp:500] New master detected at
>>>> master@10.0.0.8:5050
>>>> I0315 13:41:24.240231 2461 status_update_manager.cpp:162] New master
>>>> detected at master@10.0.0.8:5050
>>>> I0315 13:41:24.240411 2462 slave.cpp:525] Detecting new master
>>>> I0315 13:41:24.242296 2461 slave.cpp:593] Re-registered with master
>>>> master@10.0.0.8:5050
>>>> I0315 13:42:24.231416 2461 slave.cpp:2602] Current usage 9.32%. Max
>>>> allowed age: 5.647927204010996days
>>>> I0315 13:43:12.699659 2459 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0000 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:12.699748 2459 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0000
>>>> I0315 13:43:13.699810 2459 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0001 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:13.699880 2459 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0001
>>>> I0315 13:43:14.700316 2457 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0002 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:14.700397 2457 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0002
>>>> I0315 13:43:15.700831 2461 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0003 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:15.700930 2461 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0003
>>>> I0315 13:43:16.701084 2463 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0004 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:16.701154 2463 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0004
>>>> I0315 13:43:17.701400 2456 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0005 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:17.701467 2456 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0005
>>>> I0315 13:43:18.701666 2456 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0006 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:18.701738 2456 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0006
>>>> I0315 13:43:19.701906 2462 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0007 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:19.701979 2462 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0007
>>>> I0315 13:43:20.702301 2458 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0008 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:20.702368 2458 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0008
>>>> I0315 13:43:21.702569 2458 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0009 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:21.702638 2458 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0009
>>>> I0315 13:43:22.702926 2460 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0010 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:22.702997 2460 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0010
>>>> I0315 13:43:23.703258 2459 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0011 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:23.703331 2459 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0011
>>>> I0315 13:43:24.231906 2457 slave.cpp:2602] Current usage 9.32%. Max
>>>> allowed age: 5.647927204010996days
>>>> I0315 13:43:24.703522 2456 slave.cpp:1134] Asked to shut down
>>>> framework 201403152138-134217738-5050-30706-0012 by
>>>> master@10.0.0.8:5050
>>>> W0315 13:43:24.703595 2456 slave.cpp:1149] Cannot shut down unknown
>>>> framework 201403152138-134217738-5050-30706-0012
>>>> ...
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Muhuan Huang
>> Dept. of Computer Science, UCLA
>>
>
>
>
> --
> Muhuan Huang
> Dept. of Computer Science, UCLA
>
Re: slave keeps getting asked to shut down frameworks
Posted by Muhuan Huang <mh...@cs.ucla.edu>.
The problem is solved after I disable the iptables on the slave node. My
previous iptables setting has been rejecting packages from port other than
5050 and 5051.
What's the standard iptables setting for mesos? Are there specific ranges
of the ports that I should leave open ("ACCEPT"), or shall I just leave all
the ports open?
Muhuan
On Mon, Mar 17, 2014 at 2:34 PM, Muhuan Huang <mh...@cs.ucla.edu> wrote:
> Good afternoon Vinod,
>
> I just ran it again and here's the output.
>
> Thanks in advance!
>
> Muhuan
>
> -------------------------------------
> framework logs
>
> ./src/test-framework --master=10.0.0.1:5050
> I0317 06:29:48.240273 2204 sched.cpp:218] No credentials provided.
> Attempting to register without authentication
> I0317 06:29:48.240890 2204 sched.cpp:230] Detecting new master
> ^C
>
> -------------------------------------
> master logs
>
> ./bin/mesos-master.sh --ip=10.0.0.1
> I0317 14:26:02.114706 26276 main.cpp:126] Build: 2014-03-15 10:01:48 by
> mhhuang
> I0317 14:26:02.115300 26276 main.cpp:167] Starting Mesos master
> I0317 14:26:02.115432 26276 master.cpp:284] Master started on
> 10.0.0.1:5050
> I0317 14:26:02.115481 26276 master.cpp:299] Master ID:
> 201403171426-16777226-5050-26276
> I0317 14:26:02.115490 26276 master.cpp:309] Master allowing
> unauthenticated frameworks to register!!
> I0317 14:26:02.118011 26294 master.cpp:734] The newly elected leader is
> master@10.0.0.1:5050
> I0317 14:26:02.118055 26294 master.cpp:742] Elected as the leading master!
> I0317 14:26:17.713840 26292 master.cpp:1812] Attempting to register slave
> on computer4 at slave(1)@10.0.0.6:5051
> I0317 14:26:17.713894 26292 master.cpp:2782] Adding slave
> 201403171426-16777226-5050-26276-0 at computer4 with cpus(*):2;
> mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> I0317 14:26:17.714452 26295 hierarchical_allocator_process.hpp:445] Added
> slave 201403171426-16777226-5050-26276-0 (computer4) with cpus(*):2;
> mem(*):2789; disk(*):25118; ports(*):[31000-32000] (and cpus(*):2;
> mem(*):2789; disk(*):25118; ports(*):[31000-32000] available)
> I0317 14:27:51.226738 26291 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:27:51.226831 26291 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0000 at scheduler(1)@10.0.0.6:52223
> I0317 14:27:51.227170 26290 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0000
> I0317 14:27:51.227221 26293 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0000 disconnected
> I0317 14:27:51.227246 26293 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0000
> I0317 14:27:51.227285 26293 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0000 0ns to failover
> I0317 14:27:51.227421 26290 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0000
> I0317 14:27:51.227427 26291 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0000
> I0317 14:27:51.227480 26291 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0000
> W0317 14:27:51.227589 26291 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0000 because the
> framework has terminated or is inactive
> I0317 14:27:51.227661 26297 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0000
> I0317 14:27:51.227730 26297 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0000
> I0317 14:27:52.227407 26290 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:27:52.227494 26290 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0001 at scheduler(1)@10.0.0.6:52223
> I0317 14:27:52.227632 26290 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0001
> I0317 14:27:52.227790 26291 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0001 disconnected
> I0317 14:27:52.227815 26291 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0001
> I0317 14:27:52.227866 26291 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0001 0ns to failover
> I0317 14:27:52.227888 26294 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0001
> W0317 14:27:52.227896 26291 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0001 because the
> framework has terminated or is inactive
> I0317 14:27:52.227998 26291 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0001
> I0317 14:27:52.228010 26291 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0001
> I0317 14:27:52.228111 26290 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0001
> I0317 14:27:52.228157 26290 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0001
> I0317 14:27:53.227839 26297 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:27:53.227886 26297 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0002 at scheduler(1)@10.0.0.6:52223
> I0317 14:27:53.228054 26297 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0002
> I0317 14:27:53.228199 26292 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0002 disconnected
> I0317 14:27:53.228245 26292 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0002
> I0317 14:27:53.228288 26292 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0002 0ns to failover
> I0317 14:27:53.228313 26297 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0002
> W0317 14:27:53.228323 26292 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0002 because the
> framework has terminated or is inactive
> I0317 14:27:53.228410 26292 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0002
> I0317 14:27:53.228436 26292 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0002
> I0317 14:27:53.228615 26296 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0002
> I0317 14:27:53.228698 26296 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0002
> I0317 14:27:54.228085 26295 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:27:54.228142 26295 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0003 at scheduler(1)@10.0.0.6:52223
> I0317 14:27:54.228312 26295 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0003
> I0317 14:27:54.228427 26290 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0003 disconnected
> I0317 14:27:54.228456 26290 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0003
> I0317 14:27:54.228479 26290 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0003 0ns to failover
> I0317 14:27:54.228509 26295 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0003
> W0317 14:27:54.228515 26290 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0003 because the
> framework has terminated or is inactive
> I0317 14:27:54.228618 26290 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0003
> I0317 14:27:54.228627 26290 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0003
> I0317 14:27:54.228761 26293 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0003
> I0317 14:27:54.228834 26293 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0003
> I0317 14:27:55.228446 26291 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:27:55.228588 26291 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0004 at scheduler(1)@10.0.0.6:52223
> I0317 14:27:55.228703 26291 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0004
> I0317 14:27:55.228940 26297 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0004 disconnected
> I0317 14:27:55.228965 26297 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0004
> I0317 14:27:55.228994 26297 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0004 0ns to failover
> I0317 14:27:55.229010 26291 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0004
> W0317 14:27:55.229023 26297 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0004 because the
> framework has terminated or is inactive
> I0317 14:27:55.229115 26297 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0004
> I0317 14:27:55.229126 26297 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0004
> I0317 14:27:55.229244 26295 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0004
> I0317 14:27:55.229326 26295 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0004
> I0317 14:27:56.228770 26294 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:27:56.228827 26294 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0005 at scheduler(1)@10.0.0.6:52223
> I0317 14:27:56.229012 26294 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0005
> I0317 14:27:56.229126 26293 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0005 disconnected
> I0317 14:27:56.229146 26293 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0005
> I0317 14:27:56.229163 26293 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0005 0ns to failover
> W0317 14:27:56.229202 26293 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0005 because the
> framework has terminated or is inactive
> I0317 14:27:56.229204 26294 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0005
> I0317 14:27:56.229360 26293 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0005
> I0317 14:27:56.229449 26290 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0005
> I0317 14:27:56.229465 26290 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0005
> I0317 14:27:56.229558 26290 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0005
> I0317 14:27:57.229128 26291 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:27:57.229202 26291 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0006 at scheduler(1)@10.0.0.6:52223
> I0317 14:27:57.229369 26296 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0006
> I0317 14:27:57.229470 26291 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0006 disconnected
> I0317 14:27:57.229481 26291 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0006
> I0317 14:27:57.229509 26291 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0006 0ns to failover
> W0317 14:27:57.229567 26291 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0006 because the
> framework has terminated or is inactive
> I0317 14:27:57.229578 26296 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0006
> I0317 14:27:57.229683 26291 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0006
> I0317 14:27:57.229693 26291 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0006
> I0317 14:27:57.229773 26294 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0006
> I0317 14:27:57.229827 26294 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0006
> I0317 14:27:58.229428 26293 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:27:58.229504 26293 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0007 at scheduler(1)@10.0.0.6:52223
> I0317 14:27:58.229611 26293 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0007
> I0317 14:27:58.229749 26297 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0007 disconnected
> I0317 14:27:58.229766 26297 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0007
> I0317 14:27:58.229797 26297 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0007 0ns to failover
> I0317 14:27:58.229809 26293 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0007
> W0317 14:27:58.229827 26297 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0007 because the
> framework has terminated or is inactive
> I0317 14:27:58.229934 26297 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0007
> I0317 14:27:58.229946 26297 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0007
> I0317 14:27:58.230026 26296 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0007
> I0317 14:27:58.230103 26296 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0007
> I0317 14:27:59.229773 26292 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:27:59.229818 26292 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0008 at scheduler(1)@10.0.0.6:52223
> I0317 14:27:59.229957 26292 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0008
> I0317 14:27:59.230077 26297 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0008 disconnected
> I0317 14:27:59.230093 26297 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0008
> I0317 14:27:59.230129 26297 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0008 0ns to failover
> I0317 14:27:59.230204 26292 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0008
> W0317 14:27:59.230203 26290 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0008 because the
> framework has terminated or is inactive
> I0317 14:27:59.230330 26290 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0008
> I0317 14:27:59.230340 26290 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0008
> I0317 14:27:59.230438 26296 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0008
> I0317 14:27:59.230492 26296 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0008
> I0317 14:28:00.230130 26290 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:28:00.230202 26290 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0009 at scheduler(1)@10.0.0.6:52223
> I0317 14:28:00.230335 26290 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0009
> I0317 14:28:00.230501 26296 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0009 disconnected
> I0317 14:28:00.230520 26296 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0009
> I0317 14:28:00.230540 26296 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0009 0ns to failover
> W0317 14:28:00.230590 26296 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0009 because the
> framework has terminated or is inactive
> I0317 14:28:00.230599 26290 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0009
> I0317 14:28:00.230679 26296 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0009
> I0317 14:28:00.230690 26296 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0009
> I0317 14:28:00.230751 26295 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0009
> I0317 14:28:00.230808 26295 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0009
> I0317 14:28:01.230473 26295 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:28:01.230556 26295 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0010 at scheduler(1)@10.0.0.6:52223
> I0317 14:28:01.230681 26291 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0010
> I0317 14:28:01.230865 26295 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0010 disconnected
> I0317 14:28:01.230890 26295 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0010
> I0317 14:28:01.230913 26295 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0010 0ns to failover
> I0317 14:28:01.230937 26291 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0010
> W0317 14:28:01.230980 26295 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0010 because the
> framework has terminated or is inactive
> I0317 14:28:01.231137 26295 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0010
> I0317 14:28:01.231160 26295 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0010
> I0317 14:28:01.231288 26297 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0010
> I0317 14:28:01.231353 26297 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0010
> I0317 14:28:02.230877 26294 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:28:02.230938 26294 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0011 at scheduler(1)@10.0.0.6:52223
> I0317 14:28:02.231088 26294 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0011
> I0317 14:28:02.231173 26291 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0011 disconnected
> I0317 14:28:02.231201 26291 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0011
> I0317 14:28:02.231220 26291 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0011 0ns to failover
> I0317 14:28:02.231305 26294 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0011
> I0317 14:28:02.231310 26290 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0011
> I0317 14:28:02.231328 26290 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0011
> W0317 14:28:02.231406 26290 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0011 because the
> framework has terminated or is inactive
> I0317 14:28:02.231529 26296 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0011
> I0317 14:28:02.231581 26296 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0011
> I0317 14:28:03.231256 26290 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:28:03.231305 26290 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0012 at scheduler(1)@10.0.0.6:52223
> I0317 14:28:03.231448 26290 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0012
> I0317 14:28:03.231561 26296 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0012 disconnected
> I0317 14:28:03.231578 26296 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0012
> I0317 14:28:03.231601 26296 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0012 0ns to failover
> I0317 14:28:03.231703 26293 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0012
> I0317 14:28:03.231703 26290 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0012
> I0317 14:28:03.231745 26293 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0012
> W0317 14:28:03.231829 26293 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0012 because the
> framework has terminated or is inactive
> I0317 14:28:03.231923 26291 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0012
> I0317 14:28:03.231979 26291 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0012
> I0317 14:28:04.231626 26296 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:28:04.231684 26296 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0013 at scheduler(1)@10.0.0.6:52223
> I0317 14:28:04.231876 26296 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0013
> I0317 14:28:04.231951 26293 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0013 disconnected
> I0317 14:28:04.231966 26293 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0013
> I0317 14:28:04.231982 26293 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0013 0ns to failover
> I0317 14:28:04.232044 26291 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0013
> I0317 14:28:04.232067 26291 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0013
> I0317 14:28:04.232121 26296 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0013
> W0317 14:28:04.232131 26291 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0013 because the
> framework has terminated or is inactive
> I0317 14:28:04.232197 26296 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0013
> I0317 14:28:04.232234 26296 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0013
> I0317 14:28:05.231998 26295 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:28:05.232053 26295 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0014 at scheduler(1)@10.0.0.6:52223
> I0317 14:28:05.232189 26295 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0014
> I0317 14:28:05.232316 26293 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0014 disconnected
> I0317 14:28:05.232342 26293 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0014
> I0317 14:28:05.232370 26293 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0014 0ns to failover
> W0317 14:28:05.232440 26294 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0014 because the
> framework has terminated or is inactive
> I0317 14:28:05.232442 26295 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0014
> I0317 14:28:05.232537 26294 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0014
> I0317 14:28:05.232549 26294 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0014
> I0317 14:28:05.232631 26292 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0014
> I0317 14:28:05.232679 26292 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0014
> I0317 14:28:06.232396 26297 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:28:06.232435 26297 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0015 at scheduler(1)@10.0.0.6:52223
> I0317 14:28:06.232583 26297 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0015
> I0317 14:28:06.232739 26293 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0015 disconnected
> I0317 14:28:06.232764 26293 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0015
> I0317 14:28:06.232803 26293 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0015 0ns to failover
> I0317 14:28:06.232846 26296 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0015
> W0317 14:28:06.232864 26293 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0015 because the
> framework has terminated or is inactive
> I0317 14:28:06.232976 26293 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0015
> I0317 14:28:06.232996 26293 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0015
> I0317 14:28:06.233083 26295 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0015
> I0317 14:28:06.233134 26295 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0015
> I0317 14:28:07.232770 26294 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:28:07.232810 26294 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0016 at scheduler(1)@10.0.0.6:52223
> I0317 14:28:07.232951 26294 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0016
> I0317 14:28:07.233062 26297 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0016 disconnected
> I0317 14:28:07.233078 26297 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0016
> I0317 14:28:07.233098 26297 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0016 0ns to failover
> W0317 14:28:07.233137 26297 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0016 because the
> framework has terminated or is inactive
> I0317 14:28:07.233146 26294 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0016
> I0317 14:28:07.233223 26297 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0016
> I0317 14:28:07.233237 26297 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0016
> I0317 14:28:07.233338 26291 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0016
> I0317 14:28:07.233405 26291 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0016
> I0317 14:28:08.233158 26297 master.cpp:790] Received registration request
> from scheduler(1)@10.0.0.6:52223
> I0317 14:28:08.233260 26297 master.cpp:808] Registering framework
> 201403171426-16777226-5050-26276-0017 at scheduler(1)@10.0.0.6:52223
> I0317 14:28:08.233382 26297 hierarchical_allocator_process.hpp:332] Added
> framework 201403171426-16777226-5050-26276-0017
> I0317 14:28:08.233528 26291 master.cpp:569] Framework
> 201403171426-16777226-5050-26276-0017 disconnected
> I0317 14:28:08.233544 26291 master.cpp:1041] Deactivating framework
> 201403171426-16777226-5050-26276-0017
> I0317 14:28:08.233574 26291 master.cpp:591] Giving framework
> 201403171426-16777226-5050-26276-0017 0ns to failover
> W0317 14:28:08.233605 26291 master.cpp:2178] Master returning resources
> offered to framework 201403171426-16777226-5050-26276-0017 because the
> framework has terminated or is inactive
> I0317 14:28:08.233629 26297 hierarchical_allocator_process.hpp:408]
> Deactivated framework 201403171426-16777226-5050-26276-0017
> I0317 14:28:08.233739 26291 master.cpp:2166] Framework failover timeout,
> removing framework 201403171426-16777226-5050-26276-0017
> I0317 14:28:08.233752 26291 master.cpp:2654] Removing framework
> 201403171426-16777226-5050-26276-0017
> I0317 14:28:08.233839 26292 hierarchical_allocator_process.hpp:637]
> Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> (total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
> ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
> framework 201403171426-16777226-5050-26276-0017
> I0317 14:28:08.233880 26292 hierarchical_allocator_process.hpp:363]
> Removed framework 201403171426-16777226-5050-26276-0017
>
>
> ---------------------------------
> slave logs
> ./bin/mesos-slave.sh --master=10.0.0.1:5050
> I0317 06:28:14.692621 2102 main.cpp:118] Build: 2014-03-15 10:01:48 by
> mhhuang
> I0317 06:28:14.692896 2102 main.cpp:120] Creating "process" isolator
> I0317 06:28:14.693625 2102 main.cpp:132] Starting Mesos slave
> I0317 06:28:14.694778 2122 slave.cpp:111] Slave started on 1)@
> 10.0.0.6:5051
> I0317 06:28:14.698225 2122 slave.cpp:211] Slave resources: cpus(*):2;
> mem(*):2789; disk(*):25118; ports(*):[31000-32000]
> I0317 06:28:14.703965 2124 state.cpp:33] Recovering state from
> '/curr/mhhuang/tmp/mesos_work_dir/meta'
> I0317 06:28:14.708154 2124 state.cpp:50] Slave host rebooted
> I0317 06:28:14.708582 2118 status_update_manager.cpp:188] Recovering
> status update manager
> I0317 06:28:14.708757 2118 process_isolator.cpp:317] Recovering isolator
> I0317 06:28:14.708938 2118 slave.cpp:2752] Finished recovery
> I0317 06:28:14.718215 2118 slave.cpp:2784] Garbage collecting old slave
> 201403152128-134217738-5050-30225-0
> I0317 06:28:14.720899 2119 gc.cpp:56] Scheduling
> '/curr/mhhuang/tmp/mesos_work_dir/slaves/201403152128-134217738-5050-30
> 225-0' for gc 1.04742440377312weeks in the future
> I0317 06:28:14.724704 2118 slave.cpp:2784] Garbage collecting old slave
> 201403161523-1004581763-5050-28848-0
> I0317 06:28:14.724864 2124 gc.cpp:56] Scheduling
> '/curr/mhhuang/tmp/mesos_work_dir/meta/slaves/201403152128-134217738-50
> 50-30225-0' for gc 1.04742439720085weeks in the future
> I0317 06:28:14.726724 2124 gc.cpp:56] Scheduling
> '/curr/mhhuang/tmp/mesos_work_dir/slaves/201403161523-1004581763-5050-2
> 8848-0' for gc 1.0474243938146weeks in the future
> I0317 06:28:14.729722 2120 gc.cpp:56] Scheduling
> '/curr/mhhuang/tmp/mesos_work_dir/meta/slaves/201403161523-1004581763-5
> 050-28848-0' for gc 1.04742438890413weeks in the future
> I0317 06:28:14.730007 2121 slave.cpp:500] New master detected at
> master@10.0.0.1:5050
> I0317 06:28:14.730569 2121 slave.cpp:525] Detecting new master
> I0317 06:28:14.730666 2121 status_update_manager.cpp:162] New master
> detected at master@10.0.0.1:5050
> I0317 06:28:14.732645 2121 slave.cpp:543] Registered with master
> master@10.0.0.1:5050; given slave ID 201403171426-16777
> 226-5050-26276-0
> I0317 06:28:14.815829 2121 slave.cpp:556] Checkpointing SlaveInfo to
> '/curr/mhhuang/tmp/mesos_work_dir/meta/slaves/20140
> 3171426-16777226-5050-26276-0/slave.info'
> I0317 06:29:14.701341 2118 slave.cpp:2602] Current usage 81.49%. Max
> allowed age: 14.289382564984445hrs
> I0317 06:29:48.242293 2120 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0000 by master@ 10.0.0.1:5050
> W0317 06:29:48.242368 2120 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0000
> I0317 06:29:49.242784 2120 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0001 by master@ 10.0.0.1:5050
> W0317 06:29:49.242876 2120 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0001
> I0317 06:29:50.243255 2119 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0002 by master@ 10.0.0.1:5050
> W0317 06:29:50.243334 2119 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0002
> I0317 06:29:51.243321 2125 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0003 by master@ 10.0.0.1:5050
> W0317 06:29:51.243391 2125 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0003
> I0317 06:29:52.243748 2118 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0004 by master@ 10.0.0.1:5050
> W0317 06:29:52.243816 2118 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0004
> I0317 06:29:53.244084 2118 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0005 by master@ 10.0.0.1:5050
> W0317 06:29:53.244160 2118 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0005
> I0317 06:29:54.244231 2123 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0006 by master@ 10.0.0.1:5050
> W0317 06:29:54.244303 2123 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0006
> I0317 06:29:55.244420 2121 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0007 by master@ 10.0.0.1:5050
> W0317 06:29:55.244485 2121 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0007
> I0317 06:29:56.244861 2121 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0008 by master@ 10.0.0.1:5050
> W0317 06:29:56.244936 2121 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0008
> I0317 06:29:57.245123 2124 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0009 by master@ 10.0.0.1:5050
> W0317 06:29:57.245193 2124 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0009
> I0317 06:29:58.245638 2122 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0010 by master@ 10.0.0.1:5050
> W0317 06:29:58.245734 2122 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0010
> I0317 06:29:59.245710 2118 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0011 by master@ 10.0.0.1:5050
> W0317 06:29:59.245775 2118 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0011
> I0317 06:30:00.246095 2120 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0012 by master@ 10.0.0.1:5050
> W0317 06:30:00.246172 2120 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0012
> I0317 06:30:01.246351 2125 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0013 by master@ 10.0.0.1:5050
> W0317 06:30:01.246423 2125 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0013
> I0317 06:30:02.246826 2124 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0014 by master@ 10.0.0.1:5050
> W0317 06:30:02.246893 2124 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0014
> I0317 06:30:03.247225 2119 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0015 by master@ 10.0.0.1:5050
> W0317 06:30:03.247293 2119 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0015
> I0317 06:30:04.247455 2123 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0016 by master@ 10.0.0.1:5050
> W0317 06:30:04.247527 2123 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0016
> I0317 06:30:05.247911 2124 slave.cpp:1134] Asked to shut down framework
> 201403171426-16777226-5050-26276-0017 by master@ 10.0.0.1:5050
> W0317 06:30:05.247989 2124 slave.cpp:1149] Cannot shut down unknown
> framework 201403171426-16777226-5050-26276-0017
> I0317 06:30:14.703604 2124 slave.cpp:2602] Current usage 81.49%. Max
> allowed age: 14.289382564984445hrs
>
>
>
>
> On Mon, Mar 17, 2014 at 12:19 PM, Vinod Kone <vi...@gmail.com> wrote:
>
>> What do the master logs say?
>>
>>
>> On Sat, Mar 15, 2014 at 10:14 PM, Muhuan Huang <mh...@cs.ucla.edu>wrote:
>>
>>> Hello there!
>>>
>>> I'm trying to test mesos on two machines within a LAN. I launch a master
>>> on one machine and a slave on the other. When I try to launch the example
>>> framework (test-framework) on the machine where the slave resides in, it
>>> hangs. The slave keeps getting asked to shut down unknown frameworks (with
>>> increasing id).
>>>
>>> Could anyone help to explain what's going on?
>>>
>>> I have mesos-0.17.0 and CentOS installed.
>>>
>>> Muhuan
>>>
>>>
>>>
>>>
>>> Here's the output from framework:
>>> -------------------------------------------------
>>> ./src/examples/python/test-framework 10.0.0.8:5050
>>> I0315 13:43:12.697850 2504 sched.cpp:218] No credentials provided.
>>> Attempting to register without authentication
>>> I0315 13:43:12.698140 2504 sched.cpp:230] Detecting new master
>>>
>>>
>>>
>>> output from slave
>>> -------------------------------------------------
>>> [root@computer4 build]# ./bin/mesos-slave.sh --master=10.0.0.8:5050
>>> I0315 13:41:24.226071 2440 main.cpp:118] Build: 2014-03-07 07:39:55 by
>>> root
>>> I0315 13:41:24.226307 2440 main.cpp:120] Creating "process" isolator
>>> I0315 13:41:24.227330 2440 main.cpp:132] Starting Mesos slave
>>> I0315 13:41:24.228863 2460 slave.cpp:111] Slave started on 1)@
>>> 10.0.0.6:5051
>>> I0315 13:41:24.229421 2460 slave.cpp:211] Slave resources: cpus(*):2;
>>> mem(*):2789; disk(*):45276; ports(*):[31000-32000]
>>> I0315 13:41:24.235052 2456 state.cpp:33] Recovering state from
>>> '/tmp/mesos/meta'
>>> I0315 13:41:24.236737 2456 status_update_manager.cpp:188] Recovering
>>> status update manager
>>> I0315 13:41:24.237052 2463 process_isolator.cpp:317] Recovering isolator
>>> I0315 13:41:24.237324 2463 slave.cpp:2752] Finished recovery
>>> I0315 13:41:24.238035 2463 slave.cpp:2784] Garbage collecting old slave
>>> 201403152002-134217738-5050-28509-0
>>> I0315 13:41:24.238256 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/slaves/201403152002-134217738-5050-28509-0' for gc
>>> 6.99999724372741days in the future
>>> I0315 13:41:24.238311 2463 slave.cpp:2784] Garbage collecting old slave
>>> 201403050752-2950738819-5050-1262-0
>>> I0315 13:41:24.238489 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/meta/slaves/201403152002-134217738-5050-28509-0' for gc
>>> 6.99999724218667days in the future
>>> I0315 13:41:24.238534 2463 slave.cpp:2784] Garbage collecting old slave
>>> 201403041402-16777343-5050-8512-0
>>> I0315 13:41:24.238579 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/slaves/201403050752-2950738819-5050-1262-0' for gc
>>> 6.99999724048593days in the future
>>> I0315 13:41:24.238670 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/meta/slaves/201403050752-2950738819-5050-1262-0' for gc
>>> 6.99999723967704days in the future
>>> I0315 13:41:24.238698 2463 slave.cpp:2784] Garbage collecting old slave
>>> 201403071310-16777343-5050-28263-0
>>> I0315 13:41:24.238764 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/slaves/201403041402-16777343-5050-8512-0' for gc
>>> 6.99999723857778days in the future
>>> I0315 13:41:24.238837 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/meta/slaves/201403041402-16777343-5050-8512-0' for gc
>>> 6.99999723786963days in the future
>>> I0315 13:41:24.238873 2463 slave.cpp:2784] Garbage collecting old slave
>>> 201403110754-2950738819-5050-1396-0
>>> I0315 13:41:24.238911 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/slaves/201403071310-16777343-5050-28263-0' for gc
>>> 6.99999723667852days in the future
>>> I0315 13:41:24.239322 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/meta/slaves/201403071310-16777343-5050-28263-0' for gc
>>> 6.99999723581037days in the future
>>> I0315 13:41:24.239413 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/slaves/201403110754-2950738819-5050-1396-0' for gc
>>> 6.99999723348444days in the future
>>> I0315 13:41:24.239478 2461 gc.cpp:56] Scheduling
>>> '/tmp/mesos/meta/slaves/201403110754-2950738819-5050-1396-0' for gc
>>> 6.99999723250074days in the future
>>> I0315 13:41:24.239799 2462 slave.cpp:500] New master detected at
>>> master@10.0.0.8:5050
>>> I0315 13:41:24.240231 2461 status_update_manager.cpp:162] New master
>>> detected at master@10.0.0.8:5050
>>> I0315 13:41:24.240411 2462 slave.cpp:525] Detecting new master
>>> I0315 13:41:24.242296 2461 slave.cpp:593] Re-registered with master
>>> master@10.0.0.8:5050
>>> I0315 13:42:24.231416 2461 slave.cpp:2602] Current usage 9.32%. Max
>>> allowed age: 5.647927204010996days
>>> I0315 13:43:12.699659 2459 slave.cpp:1134] Asked to shut down framework
>>> 201403152138-134217738-5050-30706-0000 by master@10.0.0.8:5050
>>> W0315 13:43:12.699748 2459 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0000
>>> I0315 13:43:13.699810 2459 slave.cpp:1134] Asked to shut down framework
>>> 201403152138-134217738-5050-30706-0001 by master@10.0.0.8:5050
>>> W0315 13:43:13.699880 2459 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0001
>>> I0315 13:43:14.700316 2457 slave.cpp:1134] Asked to shut down framework
>>> 201403152138-134217738-5050-30706-0002 by master@10.0.0.8:5050
>>> W0315 13:43:14.700397 2457 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0002
>>> I0315 13:43:15.700831 2461 slave.cpp:1134] Asked to shut down framework
>>> 201403152138-134217738-5050-30706-0003 by master@10.0.0.8:5050
>>> W0315 13:43:15.700930 2461 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0003
>>> I0315 13:43:16.701084 2463 slave.cpp:1134] Asked to shut down framework
>>> 201403152138-134217738-5050-30706-0004 by master@10.0.0.8:5050
>>> W0315 13:43:16.701154 2463 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0004
>>> I0315 13:43:17.701400 2456 slave.cpp:1134] Asked to shut down
>>> framework 201403152138-134217738-5050-30706-0005 by master@10.0.0.8:5050
>>> W0315 13:43:17.701467 2456 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0005
>>> I0315 13:43:18.701666 2456 slave.cpp:1134] Asked to shut down
>>> framework 201403152138-134217738-5050-30706-0006 by master@10.0.0.8:5050
>>> W0315 13:43:18.701738 2456 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0006
>>> I0315 13:43:19.701906 2462 slave.cpp:1134] Asked to shut down
>>> framework 201403152138-134217738-5050-30706-0007 by master@10.0.0.8:5050
>>> W0315 13:43:19.701979 2462 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0007
>>> I0315 13:43:20.702301 2458 slave.cpp:1134] Asked to shut down
>>> framework 201403152138-134217738-5050-30706-0008 by master@10.0.0.8:5050
>>> W0315 13:43:20.702368 2458 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0008
>>> I0315 13:43:21.702569 2458 slave.cpp:1134] Asked to shut down
>>> framework 201403152138-134217738-5050-30706-0009 by master@10.0.0.8:5050
>>> W0315 13:43:21.702638 2458 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0009
>>> I0315 13:43:22.702926 2460 slave.cpp:1134] Asked to shut down
>>> framework 201403152138-134217738-5050-30706-0010 by master@10.0.0.8:5050
>>> W0315 13:43:22.702997 2460 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0010
>>> I0315 13:43:23.703258 2459 slave.cpp:1134] Asked to shut down
>>> framework 201403152138-134217738-5050-30706-0011 by master@10.0.0.8:5050
>>> W0315 13:43:23.703331 2459 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0011
>>> I0315 13:43:24.231906 2457 slave.cpp:2602] Current usage 9.32%. Max
>>> allowed age: 5.647927204010996days
>>> I0315 13:43:24.703522 2456 slave.cpp:1134] Asked to shut down
>>> framework 201403152138-134217738-5050-30706-0012 by master@10.0.0.8:5050
>>> W0315 13:43:24.703595 2456 slave.cpp:1149] Cannot shut down unknown
>>> framework 201403152138-134217738-5050-30706-0012
>>> ...
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
>
> --
> Muhuan Huang
> Dept. of Computer Science, UCLA
>
--
Muhuan Huang
Dept. of Computer Science, UCLA
Re: slave keeps getting asked to shut down frameworks
Posted by Muhuan Huang <mh...@cs.ucla.edu>.
Good afternoon Vinod,
I just ran it again and here's the output.
Thanks in advance!
Muhuan
-------------------------------------
framework logs
./src/test-framework --master=10.0.0.1:5050
I0317 06:29:48.240273 2204 sched.cpp:218] No credentials provided.
Attempting to register without authentication
I0317 06:29:48.240890 2204 sched.cpp:230] Detecting new master
^C
-------------------------------------
master logs
./bin/mesos-master.sh --ip=10.0.0.1
I0317 14:26:02.114706 26276 main.cpp:126] Build: 2014-03-15 10:01:48 by
mhhuang
I0317 14:26:02.115300 26276 main.cpp:167] Starting Mesos master
I0317 14:26:02.115432 26276 master.cpp:284] Master started on 10.0.0.1:5050
I0317 14:26:02.115481 26276 master.cpp:299] Master ID:
201403171426-16777226-5050-26276
I0317 14:26:02.115490 26276 master.cpp:309] Master allowing unauthenticated
frameworks to register!!
I0317 14:26:02.118011 26294 master.cpp:734] The newly elected leader is
master@10.0.0.1:5050
I0317 14:26:02.118055 26294 master.cpp:742] Elected as the leading master!
I0317 14:26:17.713840 26292 master.cpp:1812] Attempting to register slave
on computer4 at slave(1)@10.0.0.6:5051
I0317 14:26:17.713894 26292 master.cpp:2782] Adding slave
201403171426-16777226-5050-26276-0 at computer4 with cpus(*):2;
mem(*):2789; disk(*):25118; ports(*):[31000-32000]
I0317 14:26:17.714452 26295 hierarchical_allocator_process.hpp:445] Added
slave 201403171426-16777226-5050-26276-0 (computer4) with cpus(*):2;
mem(*):2789; disk(*):25118; ports(*):[31000-32000] (and cpus(*):2;
mem(*):2789; disk(*):25118; ports(*):[31000-32000] available)
I0317 14:27:51.226738 26291 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:27:51.226831 26291 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0000 at scheduler(1)@10.0.0.6:52223
I0317 14:27:51.227170 26290 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0000
I0317 14:27:51.227221 26293 master.cpp:569] Framework
201403171426-16777226-5050-26276-0000 disconnected
I0317 14:27:51.227246 26293 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0000
I0317 14:27:51.227285 26293 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0000 0ns to failover
I0317 14:27:51.227421 26290 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0000
I0317 14:27:51.227427 26291 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0000
I0317 14:27:51.227480 26291 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0000
W0317 14:27:51.227589 26291 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0000 because the
framework has terminated or is inactive
I0317 14:27:51.227661 26297 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0000
I0317 14:27:51.227730 26297 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0000
I0317 14:27:52.227407 26290 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:27:52.227494 26290 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0001 at scheduler(1)@10.0.0.6:52223
I0317 14:27:52.227632 26290 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0001
I0317 14:27:52.227790 26291 master.cpp:569] Framework
201403171426-16777226-5050-26276-0001 disconnected
I0317 14:27:52.227815 26291 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0001
I0317 14:27:52.227866 26291 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0001 0ns to failover
I0317 14:27:52.227888 26294 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0001
W0317 14:27:52.227896 26291 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0001 because the
framework has terminated or is inactive
I0317 14:27:52.227998 26291 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0001
I0317 14:27:52.228010 26291 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0001
I0317 14:27:52.228111 26290 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0001
I0317 14:27:52.228157 26290 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0001
I0317 14:27:53.227839 26297 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:27:53.227886 26297 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0002 at scheduler(1)@10.0.0.6:52223
I0317 14:27:53.228054 26297 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0002
I0317 14:27:53.228199 26292 master.cpp:569] Framework
201403171426-16777226-5050-26276-0002 disconnected
I0317 14:27:53.228245 26292 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0002
I0317 14:27:53.228288 26292 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0002 0ns to failover
I0317 14:27:53.228313 26297 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0002
W0317 14:27:53.228323 26292 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0002 because the
framework has terminated or is inactive
I0317 14:27:53.228410 26292 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0002
I0317 14:27:53.228436 26292 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0002
I0317 14:27:53.228615 26296 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0002
I0317 14:27:53.228698 26296 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0002
I0317 14:27:54.228085 26295 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:27:54.228142 26295 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0003 at scheduler(1)@10.0.0.6:52223
I0317 14:27:54.228312 26295 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0003
I0317 14:27:54.228427 26290 master.cpp:569] Framework
201403171426-16777226-5050-26276-0003 disconnected
I0317 14:27:54.228456 26290 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0003
I0317 14:27:54.228479 26290 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0003 0ns to failover
I0317 14:27:54.228509 26295 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0003
W0317 14:27:54.228515 26290 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0003 because the
framework has terminated or is inactive
I0317 14:27:54.228618 26290 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0003
I0317 14:27:54.228627 26290 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0003
I0317 14:27:54.228761 26293 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0003
I0317 14:27:54.228834 26293 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0003
I0317 14:27:55.228446 26291 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:27:55.228588 26291 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0004 at scheduler(1)@10.0.0.6:52223
I0317 14:27:55.228703 26291 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0004
I0317 14:27:55.228940 26297 master.cpp:569] Framework
201403171426-16777226-5050-26276-0004 disconnected
I0317 14:27:55.228965 26297 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0004
I0317 14:27:55.228994 26297 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0004 0ns to failover
I0317 14:27:55.229010 26291 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0004
W0317 14:27:55.229023 26297 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0004 because the
framework has terminated or is inactive
I0317 14:27:55.229115 26297 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0004
I0317 14:27:55.229126 26297 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0004
I0317 14:27:55.229244 26295 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0004
I0317 14:27:55.229326 26295 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0004
I0317 14:27:56.228770 26294 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:27:56.228827 26294 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0005 at scheduler(1)@10.0.0.6:52223
I0317 14:27:56.229012 26294 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0005
I0317 14:27:56.229126 26293 master.cpp:569] Framework
201403171426-16777226-5050-26276-0005 disconnected
I0317 14:27:56.229146 26293 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0005
I0317 14:27:56.229163 26293 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0005 0ns to failover
W0317 14:27:56.229202 26293 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0005 because the
framework has terminated or is inactive
I0317 14:27:56.229204 26294 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0005
I0317 14:27:56.229360 26293 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0005
I0317 14:27:56.229449 26290 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0005
I0317 14:27:56.229465 26290 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0005
I0317 14:27:56.229558 26290 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0005
I0317 14:27:57.229128 26291 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:27:57.229202 26291 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0006 at scheduler(1)@10.0.0.6:52223
I0317 14:27:57.229369 26296 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0006
I0317 14:27:57.229470 26291 master.cpp:569] Framework
201403171426-16777226-5050-26276-0006 disconnected
I0317 14:27:57.229481 26291 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0006
I0317 14:27:57.229509 26291 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0006 0ns to failover
W0317 14:27:57.229567 26291 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0006 because the
framework has terminated or is inactive
I0317 14:27:57.229578 26296 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0006
I0317 14:27:57.229683 26291 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0006
I0317 14:27:57.229693 26291 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0006
I0317 14:27:57.229773 26294 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0006
I0317 14:27:57.229827 26294 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0006
I0317 14:27:58.229428 26293 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:27:58.229504 26293 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0007 at scheduler(1)@10.0.0.6:52223
I0317 14:27:58.229611 26293 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0007
I0317 14:27:58.229749 26297 master.cpp:569] Framework
201403171426-16777226-5050-26276-0007 disconnected
I0317 14:27:58.229766 26297 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0007
I0317 14:27:58.229797 26297 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0007 0ns to failover
I0317 14:27:58.229809 26293 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0007
W0317 14:27:58.229827 26297 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0007 because the
framework has terminated or is inactive
I0317 14:27:58.229934 26297 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0007
I0317 14:27:58.229946 26297 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0007
I0317 14:27:58.230026 26296 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0007
I0317 14:27:58.230103 26296 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0007
I0317 14:27:59.229773 26292 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:27:59.229818 26292 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0008 at scheduler(1)@10.0.0.6:52223
I0317 14:27:59.229957 26292 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0008
I0317 14:27:59.230077 26297 master.cpp:569] Framework
201403171426-16777226-5050-26276-0008 disconnected
I0317 14:27:59.230093 26297 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0008
I0317 14:27:59.230129 26297 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0008 0ns to failover
I0317 14:27:59.230204 26292 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0008
W0317 14:27:59.230203 26290 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0008 because the
framework has terminated or is inactive
I0317 14:27:59.230330 26290 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0008
I0317 14:27:59.230340 26290 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0008
I0317 14:27:59.230438 26296 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0008
I0317 14:27:59.230492 26296 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0008
I0317 14:28:00.230130 26290 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:28:00.230202 26290 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0009 at scheduler(1)@10.0.0.6:52223
I0317 14:28:00.230335 26290 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0009
I0317 14:28:00.230501 26296 master.cpp:569] Framework
201403171426-16777226-5050-26276-0009 disconnected
I0317 14:28:00.230520 26296 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0009
I0317 14:28:00.230540 26296 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0009 0ns to failover
W0317 14:28:00.230590 26296 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0009 because the
framework has terminated or is inactive
I0317 14:28:00.230599 26290 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0009
I0317 14:28:00.230679 26296 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0009
I0317 14:28:00.230690 26296 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0009
I0317 14:28:00.230751 26295 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0009
I0317 14:28:00.230808 26295 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0009
I0317 14:28:01.230473 26295 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:28:01.230556 26295 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0010 at scheduler(1)@10.0.0.6:52223
I0317 14:28:01.230681 26291 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0010
I0317 14:28:01.230865 26295 master.cpp:569] Framework
201403171426-16777226-5050-26276-0010 disconnected
I0317 14:28:01.230890 26295 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0010
I0317 14:28:01.230913 26295 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0010 0ns to failover
I0317 14:28:01.230937 26291 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0010
W0317 14:28:01.230980 26295 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0010 because the
framework has terminated or is inactive
I0317 14:28:01.231137 26295 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0010
I0317 14:28:01.231160 26295 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0010
I0317 14:28:01.231288 26297 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0010
I0317 14:28:01.231353 26297 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0010
I0317 14:28:02.230877 26294 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:28:02.230938 26294 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0011 at scheduler(1)@10.0.0.6:52223
I0317 14:28:02.231088 26294 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0011
I0317 14:28:02.231173 26291 master.cpp:569] Framework
201403171426-16777226-5050-26276-0011 disconnected
I0317 14:28:02.231201 26291 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0011
I0317 14:28:02.231220 26291 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0011 0ns to failover
I0317 14:28:02.231305 26294 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0011
I0317 14:28:02.231310 26290 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0011
I0317 14:28:02.231328 26290 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0011
W0317 14:28:02.231406 26290 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0011 because the
framework has terminated or is inactive
I0317 14:28:02.231529 26296 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0011
I0317 14:28:02.231581 26296 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0011
I0317 14:28:03.231256 26290 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:28:03.231305 26290 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0012 at scheduler(1)@10.0.0.6:52223
I0317 14:28:03.231448 26290 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0012
I0317 14:28:03.231561 26296 master.cpp:569] Framework
201403171426-16777226-5050-26276-0012 disconnected
I0317 14:28:03.231578 26296 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0012
I0317 14:28:03.231601 26296 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0012 0ns to failover
I0317 14:28:03.231703 26293 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0012
I0317 14:28:03.231703 26290 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0012
I0317 14:28:03.231745 26293 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0012
W0317 14:28:03.231829 26293 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0012 because the
framework has terminated or is inactive
I0317 14:28:03.231923 26291 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0012
I0317 14:28:03.231979 26291 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0012
I0317 14:28:04.231626 26296 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:28:04.231684 26296 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0013 at scheduler(1)@10.0.0.6:52223
I0317 14:28:04.231876 26296 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0013
I0317 14:28:04.231951 26293 master.cpp:569] Framework
201403171426-16777226-5050-26276-0013 disconnected
I0317 14:28:04.231966 26293 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0013
I0317 14:28:04.231982 26293 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0013 0ns to failover
I0317 14:28:04.232044 26291 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0013
I0317 14:28:04.232067 26291 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0013
I0317 14:28:04.232121 26296 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0013
W0317 14:28:04.232131 26291 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0013 because the
framework has terminated or is inactive
I0317 14:28:04.232197 26296 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0013
I0317 14:28:04.232234 26296 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0013
I0317 14:28:05.231998 26295 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:28:05.232053 26295 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0014 at scheduler(1)@10.0.0.6:52223
I0317 14:28:05.232189 26295 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0014
I0317 14:28:05.232316 26293 master.cpp:569] Framework
201403171426-16777226-5050-26276-0014 disconnected
I0317 14:28:05.232342 26293 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0014
I0317 14:28:05.232370 26293 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0014 0ns to failover
W0317 14:28:05.232440 26294 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0014 because the
framework has terminated or is inactive
I0317 14:28:05.232442 26295 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0014
I0317 14:28:05.232537 26294 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0014
I0317 14:28:05.232549 26294 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0014
I0317 14:28:05.232631 26292 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0014
I0317 14:28:05.232679 26292 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0014
I0317 14:28:06.232396 26297 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:28:06.232435 26297 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0015 at scheduler(1)@10.0.0.6:52223
I0317 14:28:06.232583 26297 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0015
I0317 14:28:06.232739 26293 master.cpp:569] Framework
201403171426-16777226-5050-26276-0015 disconnected
I0317 14:28:06.232764 26293 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0015
I0317 14:28:06.232803 26293 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0015 0ns to failover
I0317 14:28:06.232846 26296 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0015
W0317 14:28:06.232864 26293 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0015 because the
framework has terminated or is inactive
I0317 14:28:06.232976 26293 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0015
I0317 14:28:06.232996 26293 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0015
I0317 14:28:06.233083 26295 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0015
I0317 14:28:06.233134 26295 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0015
I0317 14:28:07.232770 26294 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:28:07.232810 26294 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0016 at scheduler(1)@10.0.0.6:52223
I0317 14:28:07.232951 26294 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0016
I0317 14:28:07.233062 26297 master.cpp:569] Framework
201403171426-16777226-5050-26276-0016 disconnected
I0317 14:28:07.233078 26297 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0016
I0317 14:28:07.233098 26297 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0016 0ns to failover
W0317 14:28:07.233137 26297 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0016 because the
framework has terminated or is inactive
I0317 14:28:07.233146 26294 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0016
I0317 14:28:07.233223 26297 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0016
I0317 14:28:07.233237 26297 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0016
I0317 14:28:07.233338 26291 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0016
I0317 14:28:07.233405 26291 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0016
I0317 14:28:08.233158 26297 master.cpp:790] Received registration request
from scheduler(1)@10.0.0.6:52223
I0317 14:28:08.233260 26297 master.cpp:808] Registering framework
201403171426-16777226-5050-26276-0017 at scheduler(1)@10.0.0.6:52223
I0317 14:28:08.233382 26297 hierarchical_allocator_process.hpp:332] Added
framework 201403171426-16777226-5050-26276-0017
I0317 14:28:08.233528 26291 master.cpp:569] Framework
201403171426-16777226-5050-26276-0017 disconnected
I0317 14:28:08.233544 26291 master.cpp:1041] Deactivating framework
201403171426-16777226-5050-26276-0017
I0317 14:28:08.233574 26291 master.cpp:591] Giving framework
201403171426-16777226-5050-26276-0017 0ns to failover
W0317 14:28:08.233605 26291 master.cpp:2178] Master returning resources
offered to framework 201403171426-16777226-5050-26276-0017 because the
framework has terminated or is inactive
I0317 14:28:08.233629 26297 hierarchical_allocator_process.hpp:408]
Deactivated framework 201403171426-16777226-5050-26276-0017
I0317 14:28:08.233739 26291 master.cpp:2166] Framework failover timeout,
removing framework 201403171426-16777226-5050-26276-0017
I0317 14:28:08.233752 26291 master.cpp:2654] Removing framework
201403171426-16777226-5050-26276-0017
I0317 14:28:08.233839 26292 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):2789; disk(*):25118; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2789; disk(*):25118;
ports(*):[31000-32000]) on slave 201403171426-16777226-5050-26276-0 from
framework 201403171426-16777226-5050-26276-0017
I0317 14:28:08.233880 26292 hierarchical_allocator_process.hpp:363] Removed
framework 201403171426-16777226-5050-26276-0017
---------------------------------
slave logs
./bin/mesos-slave.sh --master=10.0.0.1:5050
I0317 06:28:14.692621 2102 main.cpp:118] Build: 2014-03-15 10:01:48 by
mhhuang
I0317 06:28:14.692896 2102 main.cpp:120] Creating "process" isolator
I0317 06:28:14.693625 2102 main.cpp:132] Starting Mesos slave
I0317 06:28:14.694778 2122 slave.cpp:111] Slave started on 1)@10.0.0.6:5051
I0317 06:28:14.698225 2122 slave.cpp:211] Slave resources: cpus(*):2;
mem(*):2789; disk(*):25118; ports(*):[31000-32000]
I0317 06:28:14.703965 2124 state.cpp:33] Recovering state from
'/curr/mhhuang/tmp/mesos_work_dir/meta'
I0317 06:28:14.708154 2124 state.cpp:50] Slave host rebooted
I0317 06:28:14.708582 2118 status_update_manager.cpp:188] Recovering
status update manager
I0317 06:28:14.708757 2118 process_isolator.cpp:317] Recovering isolator
I0317 06:28:14.708938 2118 slave.cpp:2752] Finished recovery
I0317 06:28:14.718215 2118 slave.cpp:2784] Garbage collecting old slave
201403152128-134217738-5050-30225-0
I0317 06:28:14.720899 2119 gc.cpp:56] Scheduling
'/curr/mhhuang/tmp/mesos_work_dir/slaves/201403152128-134217738-5050-30
225-0' for gc 1.04742440377312weeks in the future
I0317 06:28:14.724704 2118 slave.cpp:2784] Garbage collecting old slave
201403161523-1004581763-5050-28848-0
I0317 06:28:14.724864 2124 gc.cpp:56] Scheduling
'/curr/mhhuang/tmp/mesos_work_dir/meta/slaves/201403152128-134217738-50
50-30225-0' for gc 1.04742439720085weeks in the future
I0317 06:28:14.726724 2124 gc.cpp:56] Scheduling
'/curr/mhhuang/tmp/mesos_work_dir/slaves/201403161523-1004581763-5050-2
8848-0' for gc 1.0474243938146weeks in the future
I0317 06:28:14.729722 2120 gc.cpp:56] Scheduling
'/curr/mhhuang/tmp/mesos_work_dir/meta/slaves/201403161523-1004581763-5
050-28848-0' for gc 1.04742438890413weeks in the future
I0317 06:28:14.730007 2121 slave.cpp:500] New master detected at
master@10.0.0.1:5050
I0317 06:28:14.730569 2121 slave.cpp:525] Detecting new master
I0317 06:28:14.730666 2121 status_update_manager.cpp:162] New master
detected at master@10.0.0.1:5050
I0317 06:28:14.732645 2121 slave.cpp:543] Registered with master
master@10.0.0.1:5050; given slave ID 201403171426-16777 226-5050-26276-0
I0317 06:28:14.815829 2121 slave.cpp:556] Checkpointing SlaveInfo to
'/curr/mhhuang/tmp/mesos_work_dir/meta/slaves/20140
3171426-16777226-5050-26276-0/slave.info'
I0317 06:29:14.701341 2118 slave.cpp:2602] Current usage 81.49%. Max
allowed age: 14.289382564984445hrs
I0317 06:29:48.242293 2120 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0000 by master@ 10.0.0.1:5050
W0317 06:29:48.242368 2120 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0000
I0317 06:29:49.242784 2120 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0001 by master@ 10.0.0.1:5050
W0317 06:29:49.242876 2120 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0001
I0317 06:29:50.243255 2119 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0002 by master@ 10.0.0.1:5050
W0317 06:29:50.243334 2119 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0002
I0317 06:29:51.243321 2125 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0003 by master@ 10.0.0.1:5050
W0317 06:29:51.243391 2125 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0003
I0317 06:29:52.243748 2118 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0004 by master@ 10.0.0.1:5050
W0317 06:29:52.243816 2118 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0004
I0317 06:29:53.244084 2118 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0005 by master@ 10.0.0.1:5050
W0317 06:29:53.244160 2118 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0005
I0317 06:29:54.244231 2123 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0006 by master@ 10.0.0.1:5050
W0317 06:29:54.244303 2123 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0006
I0317 06:29:55.244420 2121 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0007 by master@ 10.0.0.1:5050
W0317 06:29:55.244485 2121 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0007
I0317 06:29:56.244861 2121 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0008 by master@ 10.0.0.1:5050
W0317 06:29:56.244936 2121 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0008
I0317 06:29:57.245123 2124 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0009 by master@ 10.0.0.1:5050
W0317 06:29:57.245193 2124 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0009
I0317 06:29:58.245638 2122 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0010 by master@ 10.0.0.1:5050
W0317 06:29:58.245734 2122 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0010
I0317 06:29:59.245710 2118 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0011 by master@ 10.0.0.1:5050
W0317 06:29:59.245775 2118 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0011
I0317 06:30:00.246095 2120 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0012 by master@ 10.0.0.1:5050
W0317 06:30:00.246172 2120 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0012
I0317 06:30:01.246351 2125 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0013 by master@ 10.0.0.1:5050
W0317 06:30:01.246423 2125 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0013
I0317 06:30:02.246826 2124 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0014 by master@ 10.0.0.1:5050
W0317 06:30:02.246893 2124 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0014
I0317 06:30:03.247225 2119 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0015 by master@ 10.0.0.1:5050
W0317 06:30:03.247293 2119 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0015
I0317 06:30:04.247455 2123 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0016 by master@ 10.0.0.1:5050
W0317 06:30:04.247527 2123 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0016
I0317 06:30:05.247911 2124 slave.cpp:1134] Asked to shut down framework
201403171426-16777226-5050-26276-0017 by master@ 10.0.0.1:5050
W0317 06:30:05.247989 2124 slave.cpp:1149] Cannot shut down unknown
framework 201403171426-16777226-5050-26276-0017
I0317 06:30:14.703604 2124 slave.cpp:2602] Current usage 81.49%. Max
allowed age: 14.289382564984445hrs
On Mon, Mar 17, 2014 at 12:19 PM, Vinod Kone <vi...@gmail.com> wrote:
> What do the master logs say?
>
>
> On Sat, Mar 15, 2014 at 10:14 PM, Muhuan Huang <mh...@cs.ucla.edu>wrote:
>
>> Hello there!
>>
>> I'm trying to test mesos on two machines within a LAN. I launch a master
>> on one machine and a slave on the other. When I try to launch the example
>> framework (test-framework) on the machine where the slave resides in, it
>> hangs. The slave keeps getting asked to shut down unknown frameworks (with
>> increasing id).
>>
>> Could anyone help to explain what's going on?
>>
>> I have mesos-0.17.0 and CentOS installed.
>>
>> Muhuan
>>
>>
>>
>>
>> Here's the output from framework:
>> -------------------------------------------------
>> ./src/examples/python/test-framework 10.0.0.8:5050
>> I0315 13:43:12.697850 2504 sched.cpp:218] No credentials provided.
>> Attempting to register without authentication
>> I0315 13:43:12.698140 2504 sched.cpp:230] Detecting new master
>>
>>
>>
>> output from slave
>> -------------------------------------------------
>> [root@computer4 build]# ./bin/mesos-slave.sh --master=10.0.0.8:5050
>> I0315 13:41:24.226071 2440 main.cpp:118] Build: 2014-03-07 07:39:55 by
>> root
>> I0315 13:41:24.226307 2440 main.cpp:120] Creating "process" isolator
>> I0315 13:41:24.227330 2440 main.cpp:132] Starting Mesos slave
>> I0315 13:41:24.228863 2460 slave.cpp:111] Slave started on 1)@
>> 10.0.0.6:5051
>> I0315 13:41:24.229421 2460 slave.cpp:211] Slave resources: cpus(*):2;
>> mem(*):2789; disk(*):45276; ports(*):[31000-32000]
>> I0315 13:41:24.235052 2456 state.cpp:33] Recovering state from
>> '/tmp/mesos/meta'
>> I0315 13:41:24.236737 2456 status_update_manager.cpp:188] Recovering
>> status update manager
>> I0315 13:41:24.237052 2463 process_isolator.cpp:317] Recovering isolator
>> I0315 13:41:24.237324 2463 slave.cpp:2752] Finished recovery
>> I0315 13:41:24.238035 2463 slave.cpp:2784] Garbage collecting old slave
>> 201403152002-134217738-5050-28509-0
>> I0315 13:41:24.238256 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/slaves/201403152002-134217738-5050-28509-0' for gc
>> 6.99999724372741days in the future
>> I0315 13:41:24.238311 2463 slave.cpp:2784] Garbage collecting old slave
>> 201403050752-2950738819-5050-1262-0
>> I0315 13:41:24.238489 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/meta/slaves/201403152002-134217738-5050-28509-0' for gc
>> 6.99999724218667days in the future
>> I0315 13:41:24.238534 2463 slave.cpp:2784] Garbage collecting old slave
>> 201403041402-16777343-5050-8512-0
>> I0315 13:41:24.238579 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/slaves/201403050752-2950738819-5050-1262-0' for gc
>> 6.99999724048593days in the future
>> I0315 13:41:24.238670 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/meta/slaves/201403050752-2950738819-5050-1262-0' for gc
>> 6.99999723967704days in the future
>> I0315 13:41:24.238698 2463 slave.cpp:2784] Garbage collecting old slave
>> 201403071310-16777343-5050-28263-0
>> I0315 13:41:24.238764 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/slaves/201403041402-16777343-5050-8512-0' for gc
>> 6.99999723857778days in the future
>> I0315 13:41:24.238837 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/meta/slaves/201403041402-16777343-5050-8512-0' for gc
>> 6.99999723786963days in the future
>> I0315 13:41:24.238873 2463 slave.cpp:2784] Garbage collecting old slave
>> 201403110754-2950738819-5050-1396-0
>> I0315 13:41:24.238911 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/slaves/201403071310-16777343-5050-28263-0' for gc
>> 6.99999723667852days in the future
>> I0315 13:41:24.239322 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/meta/slaves/201403071310-16777343-5050-28263-0' for gc
>> 6.99999723581037days in the future
>> I0315 13:41:24.239413 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/slaves/201403110754-2950738819-5050-1396-0' for gc
>> 6.99999723348444days in the future
>> I0315 13:41:24.239478 2461 gc.cpp:56] Scheduling
>> '/tmp/mesos/meta/slaves/201403110754-2950738819-5050-1396-0' for gc
>> 6.99999723250074days in the future
>> I0315 13:41:24.239799 2462 slave.cpp:500] New master detected at
>> master@10.0.0.8:5050
>> I0315 13:41:24.240231 2461 status_update_manager.cpp:162] New master
>> detected at master@10.0.0.8:5050
>> I0315 13:41:24.240411 2462 slave.cpp:525] Detecting new master
>> I0315 13:41:24.242296 2461 slave.cpp:593] Re-registered with master
>> master@10.0.0.8:5050
>> I0315 13:42:24.231416 2461 slave.cpp:2602] Current usage 9.32%. Max
>> allowed age: 5.647927204010996days
>> I0315 13:43:12.699659 2459 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0000 by master@10.0.0.8:5050
>> W0315 13:43:12.699748 2459 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0000
>> I0315 13:43:13.699810 2459 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0001 by master@10.0.0.8:5050
>> W0315 13:43:13.699880 2459 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0001
>> I0315 13:43:14.700316 2457 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0002 by master@10.0.0.8:5050
>> W0315 13:43:14.700397 2457 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0002
>> I0315 13:43:15.700831 2461 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0003 by master@10.0.0.8:5050
>> W0315 13:43:15.700930 2461 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0003
>> I0315 13:43:16.701084 2463 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0004 by master@10.0.0.8:5050
>> W0315 13:43:16.701154 2463 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0004
>> I0315 13:43:17.701400 2456 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0005 by master@10.0.0.8:5050
>> W0315 13:43:17.701467 2456 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0005
>> I0315 13:43:18.701666 2456 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0006 by master@10.0.0.8:5050
>> W0315 13:43:18.701738 2456 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0006
>> I0315 13:43:19.701906 2462 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0007 by master@10.0.0.8:5050
>> W0315 13:43:19.701979 2462 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0007
>> I0315 13:43:20.702301 2458 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0008 by master@10.0.0.8:5050
>> W0315 13:43:20.702368 2458 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0008
>> I0315 13:43:21.702569 2458 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0009 by master@10.0.0.8:5050
>> W0315 13:43:21.702638 2458 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0009
>> I0315 13:43:22.702926 2460 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0010 by master@10.0.0.8:5050
>> W0315 13:43:22.702997 2460 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0010
>> I0315 13:43:23.703258 2459 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0011 by master@10.0.0.8:5050
>> W0315 13:43:23.703331 2459 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0011
>> I0315 13:43:24.231906 2457 slave.cpp:2602] Current usage 9.32%. Max
>> allowed age: 5.647927204010996days
>> I0315 13:43:24.703522 2456 slave.cpp:1134] Asked to shut down framework
>> 201403152138-134217738-5050-30706-0012 by master@10.0.0.8:5050
>> W0315 13:43:24.703595 2456 slave.cpp:1149] Cannot shut down unknown
>> framework 201403152138-134217738-5050-30706-0012
>> ...
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
--
Muhuan Huang
Dept. of Computer Science, UCLA
Re: slave keeps getting asked to shut down frameworks
Posted by Vinod Kone <vi...@gmail.com>.
What do the master logs say?
On Sat, Mar 15, 2014 at 10:14 PM, Muhuan Huang <mh...@cs.ucla.edu> wrote:
> Hello there!
>
> I'm trying to test mesos on two machines within a LAN. I launch a master
> on one machine and a slave on the other. When I try to launch the example
> framework (test-framework) on the machine where the slave resides in, it
> hangs. The slave keeps getting asked to shut down unknown frameworks (with
> increasing id).
>
> Could anyone help to explain what's going on?
>
> I have mesos-0.17.0 and CentOS installed.
>
> Muhuan
>
>
>
>
> Here's the output from framework:
> -------------------------------------------------
> ./src/examples/python/test-framework 10.0.0.8:5050
> I0315 13:43:12.697850 2504 sched.cpp:218] No credentials provided.
> Attempting to register without authentication
> I0315 13:43:12.698140 2504 sched.cpp:230] Detecting new master
>
>
>
> output from slave
> -------------------------------------------------
> [root@computer4 build]# ./bin/mesos-slave.sh --master=10.0.0.8:5050
> I0315 13:41:24.226071 2440 main.cpp:118] Build: 2014-03-07 07:39:55 by
> root
> I0315 13:41:24.226307 2440 main.cpp:120] Creating "process" isolator
> I0315 13:41:24.227330 2440 main.cpp:132] Starting Mesos slave
> I0315 13:41:24.228863 2460 slave.cpp:111] Slave started on 1)@
> 10.0.0.6:5051
> I0315 13:41:24.229421 2460 slave.cpp:211] Slave resources: cpus(*):2;
> mem(*):2789; disk(*):45276; ports(*):[31000-32000]
> I0315 13:41:24.235052 2456 state.cpp:33] Recovering state from
> '/tmp/mesos/meta'
> I0315 13:41:24.236737 2456 status_update_manager.cpp:188] Recovering
> status update manager
> I0315 13:41:24.237052 2463 process_isolator.cpp:317] Recovering isolator
> I0315 13:41:24.237324 2463 slave.cpp:2752] Finished recovery
> I0315 13:41:24.238035 2463 slave.cpp:2784] Garbage collecting old slave
> 201403152002-134217738-5050-28509-0
> I0315 13:41:24.238256 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/slaves/201403152002-134217738-5050-28509-0' for gc
> 6.99999724372741days in the future
> I0315 13:41:24.238311 2463 slave.cpp:2784] Garbage collecting old slave
> 201403050752-2950738819-5050-1262-0
> I0315 13:41:24.238489 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/meta/slaves/201403152002-134217738-5050-28509-0' for gc
> 6.99999724218667days in the future
> I0315 13:41:24.238534 2463 slave.cpp:2784] Garbage collecting old slave
> 201403041402-16777343-5050-8512-0
> I0315 13:41:24.238579 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/slaves/201403050752-2950738819-5050-1262-0' for gc
> 6.99999724048593days in the future
> I0315 13:41:24.238670 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/meta/slaves/201403050752-2950738819-5050-1262-0' for gc
> 6.99999723967704days in the future
> I0315 13:41:24.238698 2463 slave.cpp:2784] Garbage collecting old slave
> 201403071310-16777343-5050-28263-0
> I0315 13:41:24.238764 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/slaves/201403041402-16777343-5050-8512-0' for gc
> 6.99999723857778days in the future
> I0315 13:41:24.238837 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/meta/slaves/201403041402-16777343-5050-8512-0' for gc
> 6.99999723786963days in the future
> I0315 13:41:24.238873 2463 slave.cpp:2784] Garbage collecting old slave
> 201403110754-2950738819-5050-1396-0
> I0315 13:41:24.238911 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/slaves/201403071310-16777343-5050-28263-0' for gc
> 6.99999723667852days in the future
> I0315 13:41:24.239322 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/meta/slaves/201403071310-16777343-5050-28263-0' for gc
> 6.99999723581037days in the future
> I0315 13:41:24.239413 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/slaves/201403110754-2950738819-5050-1396-0' for gc
> 6.99999723348444days in the future
> I0315 13:41:24.239478 2461 gc.cpp:56] Scheduling
> '/tmp/mesos/meta/slaves/201403110754-2950738819-5050-1396-0' for gc
> 6.99999723250074days in the future
> I0315 13:41:24.239799 2462 slave.cpp:500] New master detected at
> master@10.0.0.8:5050
> I0315 13:41:24.240231 2461 status_update_manager.cpp:162] New master
> detected at master@10.0.0.8:5050
> I0315 13:41:24.240411 2462 slave.cpp:525] Detecting new master
> I0315 13:41:24.242296 2461 slave.cpp:593] Re-registered with master
> master@10.0.0.8:5050
> I0315 13:42:24.231416 2461 slave.cpp:2602] Current usage 9.32%. Max
> allowed age: 5.647927204010996days
> I0315 13:43:12.699659 2459 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0000 by master@10.0.0.8:5050
> W0315 13:43:12.699748 2459 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0000
> I0315 13:43:13.699810 2459 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0001 by master@10.0.0.8:5050
> W0315 13:43:13.699880 2459 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0001
> I0315 13:43:14.700316 2457 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0002 by master@10.0.0.8:5050
> W0315 13:43:14.700397 2457 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0002
> I0315 13:43:15.700831 2461 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0003 by master@10.0.0.8:5050
> W0315 13:43:15.700930 2461 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0003
> I0315 13:43:16.701084 2463 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0004 by master@10.0.0.8:5050
> W0315 13:43:16.701154 2463 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0004
> I0315 13:43:17.701400 2456 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0005 by master@10.0.0.8:5050
> W0315 13:43:17.701467 2456 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0005
> I0315 13:43:18.701666 2456 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0006 by master@10.0.0.8:5050
> W0315 13:43:18.701738 2456 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0006
> I0315 13:43:19.701906 2462 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0007 by master@10.0.0.8:5050
> W0315 13:43:19.701979 2462 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0007
> I0315 13:43:20.702301 2458 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0008 by master@10.0.0.8:5050
> W0315 13:43:20.702368 2458 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0008
> I0315 13:43:21.702569 2458 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0009 by master@10.0.0.8:5050
> W0315 13:43:21.702638 2458 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0009
> I0315 13:43:22.702926 2460 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0010 by master@10.0.0.8:5050
> W0315 13:43:22.702997 2460 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0010
> I0315 13:43:23.703258 2459 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0011 by master@10.0.0.8:5050
> W0315 13:43:23.703331 2459 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0011
> I0315 13:43:24.231906 2457 slave.cpp:2602] Current usage 9.32%. Max
> allowed age: 5.647927204010996days
> I0315 13:43:24.703522 2456 slave.cpp:1134] Asked to shut down framework
> 201403152138-134217738-5050-30706-0012 by master@10.0.0.8:5050
> W0315 13:43:24.703595 2456 slave.cpp:1149] Cannot shut down unknown
> framework 201403152138-134217738-5050-30706-0012
> ...
>
>
>
>
>
>
>
>
>
>