You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Benjamin Mahler (JIRA)" <ji...@apache.org> on 2015/04/17 02:03:37 UTC
[jira] [Comment Edited] (MESOS-2144) Segmentation Fault in
ExamplesTest.LowLevelSchedulerPthread
[ https://issues.apache.org/jira/browse/MESOS-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498789#comment-14498789 ]
Benjamin Mahler edited comment on MESOS-2144 at 4/17/15 12:00 AM:
------------------------------------------------------------------
Just got one of these with full backtrace:
{code}
F0416 12:21:01.664001 365723648 slave.cpp:3816] CHECK_SOME(state::checkpoint(path, bootId.get())): Failed to rename '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/0/meta/EntzLz' to '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/1/meta/boot_id': No such file or directory
*** Check failure stack trace: ***
@ 0x110e9284a google::LogMessage::Fail()
@ 0x110e917dd google::LogMessage::SendToLog()
@ 0x110e924ea google::LogMessage::Flush()
@ 0x110e99348 google::LogMessageFatal::~LogMessageFatal()
@ 0x110e92ca5 google::LogMessageFatal::~LogMessageFatal()
@ 0x10f3d33d3 _CheckFatal::~_CheckFatal()
@ 0x10f3d3025 _CheckFatal::~_CheckFatal()
@ 0x10fd94da6 mesos::internal::slave::Slave::__recover()
@ 0x10fe7f09d _ZZN7process8dispatchIN5mesos8internal5slave5SlaveERKNS_6FutureI7NothingEES7_EEvRKNS_3PIDIT_EEMSB_FvT0_ET1_ENKUlPNS_11ProcessBaseEE_clESK_
@ 0x10fe7ee7f _ZNSt3__110__function6__funcIZN7process8dispatchIN5mesos8internal5slave5SlaveERKNS2_6FutureI7NothingEESA_EEvRKNS2_3PIDIT_EEMSE_FvT0_ET1_EUlPNS2_11ProcessBaseEE_NS_9allocatorISO_EEFvSN_EEclEOSN_
@ 0x110d74e7b std::__1::function<>::operator()()
@ 0x110d5c5bf process::ProcessBase::visit()
@ 0x110de6c0e process::DispatchEvent::visit()
@ 0x10f3d0841 process::ProcessBase::serve()
@ 0x110d45abe process::ProcessManager::resume()
@ 0x110d451de process::schedule()
@ 0x7fff8f1eb268 _pthread_body
@ 0x7fff8f1eb1e5 _pthread_start
@ 0x7fff8f1e941d thread_start
{code}
The full log from the test (MESOS_VERBOSE, GLOG_v=2)
{code}
[ RUN ] ExamplesTest.LowLevelSchedulerPthread
Using temporary directory '/tmp/ExamplesTest_LowLevelSchedulerPthread_vVqryS'
I0416 12:21:01.637110 2105078528 logging.cpp:177] Logging to STDERR
Enabling authentication for the scheduler
I0416 12:21:01.639566 2105078528 process.cpp:2081] Spawned process __gc__@127.0.0.1:52945
I0416 12:21:01.639770 2105078528 process.cpp:2081] Spawned process help@127.0.0.1:52945
I0416 12:21:01.639583 365723648 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.639622912+00:00
I0416 12:21:01.639777 367869952 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.639796992+00:00
I0416 12:21:01.639875 366260224 process.cpp:2091] Resuming logging@127.0.0.1:52945 at 2015-04-16 19:21:01.639906816+00:00
I0416 12:21:01.639909 2105078528 process.cpp:2081] Spawned process logging@127.0.0.1:52945
I0416 12:21:01.639978 367869952 process.cpp:2091] Resuming profiler@127.0.0.1:52945 at 2015-04-16 19:21:01.640003840+00:00
I0416 12:21:01.640033 368943104 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.640058880+00:00
I0416 12:21:01.640051 2105078528 process.cpp:2081] Spawned process profiler@127.0.0.1:52945
I0416 12:21:01.640246 368406528 process.cpp:2091] Resuming system@127.0.0.1:52945 at 2015-04-16 19:21:01.640268032+00:00
I0416 12:21:01.640236 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.640258048+00:00
I0416 12:21:01.640318 2105078528 process.cpp:2081] Spawned process system@127.0.0.1:52945
I0416 12:21:01.640321 368943104 process.cpp:2091] Resuming __limiter__(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.640336128+00:00
I0416 12:21:01.640390 368406528 process.cpp:2081] Spawned process __limiter__(1)@127.0.0.1:52945
I0416 12:21:01.640425 365723648 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.640440064+00:00
I0416 12:21:01.640472 368406528 process.cpp:2081] Spawned process metrics@127.0.0.1:52945
I0416 12:21:01.640521 367869952 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.640538880+00:00
I0416 12:21:01.640733 366796800 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.640760064+00:00
I0416 12:21:01.640913 2105078528 process.cpp:2081] Spawned process __processes__@127.0.0.1:52945
I0416 12:21:01.640919 366796800 process.cpp:2091] Resuming __processes__@127.0.0.1:52945 at 2015-04-16 19:21:01.640937984+00:00
I0416 12:21:01.640949 2105078528 process.cpp:912] libprocess is initialized on 127.0.0.1:52945 for 8 cpus
I0416 12:21:01.640971 365723648 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.640985856+00:00
W0416 12:21:01.641326 2105078528 scheduler.cpp:134]
**************************************************
Scheduler driver bound to loopback interface! Cannot communicate with remote master(s). You might want to set 'LIBPROCESS_IP' environment variable to use a routable IP address.
**************************************************
I0416 12:21:01.641348 2105078528 scheduler.cpp:149] Version: 0.23.0
I0416 12:21:01.642009 2105078528 process.cpp:2081] Spawned process hierarchical-allocator(1)@127.0.0.1:52945
I0416 12:21:01.642019 368943104 process.cpp:2091] Resuming hierarchical-allocator(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.642038016+00:00
I0416 12:21:01.642069 365723648 process.cpp:2091] Resuming files@127.0.0.1:52945 at 2015-04-16 19:21:01.642086912+00:00
I0416 12:21:01.642115 2105078528 process.cpp:2081] Spawned process files@127.0.0.1:52945
I0416 12:21:01.642194 366260224 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.642208000+00:00
I0416 12:21:01.647266 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:33.647293184+00:00
I0416 12:21:01.648769 2105078528 leveldb.cpp:176] Opened db in 2269us
I0416 12:21:01.649345 2105078528 leveldb.cpp:183] Compacted db in 555us
I0416 12:21:01.649394 2105078528 leveldb.cpp:198] Created db iterator in 27us
I0416 12:21:01.649420 2105078528 leveldb.cpp:204] Seeked to beginning of db in 14us
I0416 12:21:01.649440 2105078528 leveldb.cpp:273] Iterated through 0 keys in the db in 15us
I0416 12:21:01.649510 2105078528 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
I0416 12:21:01.650113 2105078528 process.cpp:2081] Spawned process log-replica(1)@127.0.0.1:52945
I0416 12:21:01.650157 366796800 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650183936+00:00
I0416 12:21:01.650243 368943104 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650263040+00:00
I0416 12:21:01.650296 2105078528 process.cpp:2081] Spawned process (1)@127.0.0.1:52945
I0416 12:21:01.650413 2105078528 process.cpp:2081] Spawned process log(1)@127.0.0.1:52945
I0416 12:21:01.650424 365723648 process.cpp:2091] Resuming log(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650441984+00:00
I0416 12:21:01.650574 2105078528 process.cpp:2081] Spawned process log-reader(1)@127.0.0.1:52945
I0416 12:21:01.650580 366260224 process.cpp:2091] Resuming log-reader(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650599936+00:00
I0416 12:21:01.650606 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.650621952+00:00
I0416 12:21:01.650607 367333376 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650633984+00:00
I0416 12:21:01.650658 367333376 recover.cpp:449] Starting replica recovery
I0416 12:21:01.650636 369479680 process.cpp:2091] Resuming log-writer(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650664960+00:00
I0416 12:21:01.650686 2105078528 process.cpp:2081] Spawned process log-writer(1)@127.0.0.1:52945
I0416 12:21:01.650688 365723648 process.cpp:2081] Spawned process log-recover(1)@127.0.0.1:52945
I0416 12:21:01.650837 365723648 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.650851072+00:00
I0416 12:21:01.650836 369479680 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650855936+00:00
I0416 12:21:01.650857 366260224 process.cpp:2091] Resuming (2)@127.0.0.1:52945 at 2015-04-16 19:21:01.650881024+00:00
I0416 12:21:01.650900 2105078528 process.cpp:2081] Spawned process (2)@127.0.0.1:52945
I0416 12:21:01.650966 367333376 recover.cpp:475] Replica is in EMPTY status
I0416 12:21:01.651100 367869952 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.651115008+00:00
I0416 12:21:01.651093 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.651112192+00:00
I0416 12:21:01.651120 366260224 process.cpp:2091] Resuming log-recover-protocol(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651134976+00:00
I0416 12:21:01.651160 367333376 process.cpp:2081] Spawned process log-recover-protocol(1)@127.0.0.1:52945
I0416 12:21:01.651211 366260224 recover.cpp:131] Starting to wait for enough quorum of replicas before running recovery protocol, expected quroum size: 1
I0416 12:21:01.651298 2105078528 process.cpp:2081] Spawned process registrar(1)@127.0.0.1:52945
I0416 12:21:01.651301 365723648 process.cpp:2091] Resuming registrar(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651323136+00:00
I0416 12:21:01.651314 366796800 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651334144+00:00
I0416 12:21:01.651384 367333376 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651405056+00:00
I0416 12:21:01.651434 2105078528 process.cpp:2081] Spawned process standalone-master-detector(1)@127.0.0.1:52945
I0416 12:21:01.651576 2105078528 process.cpp:2081] Spawned process authorizer(1)@127.0.0.1:52945
I0416 12:21:01.651594 365723648 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.651608064+00:00
I0416 12:21:01.651600 368406528 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.651613952+00:00
I0416 12:21:01.651583 367869952 process.cpp:2091] Resuming authorizer(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651602944+00:00
I0416 12:21:01.651657 369479680 process.cpp:2091] Resuming __latch__(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651691008+00:00
I0416 12:21:01.651705 366260224 process.cpp:2081] Spawned process __latch__(1)@127.0.0.1:52945
I0416 12:21:01.651888 366260224 recover.cpp:145] Broadcasting recover request to all replicas
I0416 12:21:01.651996 368406528 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.652011008+00:00
I0416 12:21:01.652019 367333376 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.652039168+00:00
I0416 12:21:01.652148 366796800 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.652165888+00:00
I0416 12:21:01.652154 367869952 process.cpp:2091] Resuming (3)@127.0.0.1:52945 at 2015-04-16 19:21:01.652169984+00:00
I0416 12:21:01.652217 368406528 process.cpp:2081] Spawned process (3)@127.0.0.1:52945
I0416 12:21:01.652369 366260224 process.cpp:2091] Resuming (3)@127.0.0.1:52945 at 2015-04-16 19:21:01.652391936+00:00
I0416 12:21:01.652400 368943104 process.cpp:2091] Resuming log-recover-protocol(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.652436992+00:00
I0416 12:21:01.652465 368943104 recover.cpp:154] Broadcast request completed
I0416 12:21:01.652479 368406528 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.652500992+00:00
I0416 12:21:01.652529 368406528 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request
I0416 12:21:01.652642 366260224 process.cpp:2091] Resuming (3)@127.0.0.1:52945 at 2015-04-16 19:21:01.652664064+00:00
I0416 12:21:01.652740 366260224 process.cpp:2198] Cleaning up (3)@127.0.0.1:52945
I0416 12:21:01.652799 368943104 recover.cpp:195] Received a recover response from a replica in EMPTY status
I0416 12:21:01.652814 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.652847104+00:00
I0416 12:21:01.652892 368406528 process.cpp:2091] Resuming __latch__(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.652925952+00:00
I0416 12:21:01.652953 368406528 process.cpp:2198] Cleaning up __latch__(1)@127.0.0.1:52945
I0416 12:21:01.653031 366796800 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.653053952+00:00
I0416 12:21:01.653064 367869952 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.653084160+00:00
I0416 12:21:01.653147 367869952 recover.cpp:566] Updating replica status to STARTING
I0416 12:21:01.653162 368943104 process.cpp:2198] Cleaning up log-recover-protocol(1)@127.0.0.1:52945
I0416 12:21:01.653259 366796800 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.653281024+00:00
I0416 12:21:01.653301 368406528 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.653321984+00:00
I0416 12:21:01.653651 368406528 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 295us
I0416 12:21:01.653693 368406528 replica.cpp:323] Persisted replica status to STARTING
I0416 12:21:01.653681 365723648 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.653703936+00:00
I0416 12:21:01.653806 365723648 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.653827072+00:00
I0416 12:21:01.653833 366260224 process.cpp:2091] Resuming master@127.0.0.1:52945 at 2015-04-16 19:21:01.653851136+00:00
I0416 12:21:01.653863 2105078528 process.cpp:2081] Spawned process master@127.0.0.1:52945
I0416 12:21:01.653874 366260224 master.cpp:361] Master 20150416-122101-16777343-52945-98969 (localhost) started on 127.0.0.1:52945
I0416 12:21:01.653918 365723648 recover.cpp:475] Replica is in STARTING status
I0416 12:21:01.654013 365723648 process.cpp:2081] Spawned process log-recover-protocol(2)@127.0.0.1:52945
I0416 12:21:01.654022 367333376 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.654042880+00:00
I0416 12:21:01.654026 368406528 process.cpp:2091] Resuming log-recover-protocol(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.654049024+00:00
I0416 12:21:01.654101 368406528 recover.cpp:131] Starting to wait for enough quorum of replicas before running recovery protocol, expected quroum size: 1
I0416 12:21:01.654170 367869952 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.654191872+00:00
I0416 12:21:01.653894 366260224 master.cpp:363] Flags at startup: --acls="permissive: false
register_frameworks {
principals {
type: SOME
values: "test-principal"
}
roles {
type: SOME
values: "*"
}
}
run_tasks {
principals {
type: SOME
values: "test-principal"
}
users {
type: SOME
values: "osxbuildbot"
}
}
" --allocation_interval="1secs" --authenticate="true" --authenticate_slaves="false" --authenticators="crammd5" --credentials="/tmp/ExamplesTest_LowLevelSchedulerPthread_vVqryS/credentials" --framework_sorter="drf" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" --registry_strict="false" --root_submissions="true" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/Users/osxbuildbot/buildslave/slave/dev_test_osx/dev_distcheck_build_dist/mesos-0.23.0/src/webui" --work_dir="/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ" --zk_session_timeout="10secs"
W0416 12:21:01.654228 366260224 master.cpp:366]
**************************************************
Master bound to loopback interface! Cannot communicate with remote schedulers or slaves. You might want to set '--ip' flag to a routable IP address.
**************************************************
I0416 12:21:01.654275 368406528 process.cpp:2081] Spawned process __latch__(2)@127.0.0.1:52945
I0416 12:21:01.654292 366260224 master.cpp:408] Master only allowing authenticated frameworks to register
I0416 12:21:01.654278 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.654299136+00:00
I0416 12:21:01.654280 366796800 process.cpp:2091] Resuming __latch__(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.654301952+00:00
I0416 12:21:01.654309 366260224 master.cpp:415] Master allowing unauthenticated slaves to register
I0416 12:21:01.654340 366260224 credentials.hpp:37] Loading credentials for authentication from '/tmp/ExamplesTest_LowLevelSchedulerPthread_vVqryS/credentials'
I0416 12:21:01.654362 368406528 recover.cpp:145] Broadcasting recover request to all replicas
I0416 12:21:01.654424 365723648 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.654445056+00:00
I0416 12:21:01.654500 366260224 master.cpp:452] Using default 'crammd5' authenticator
I0416 12:21:01.654549 366260224 authenticator.cpp:511] Initializing server SASL
I0416 12:21:01.654570 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.654592000+00:00
I0416 12:21:01.654582 368943104 process.cpp:2091] Resuming (4)@127.0.0.1:52945 at 2015-04-16 19:21:01.654604032+00:00
I0416 12:21:01.654599 365723648 process.cpp:2081] Spawned process (4)@127.0.0.1:52945
I0416 12:21:01.654757 368943104 process.cpp:2091] Resuming (4)@127.0.0.1:52945 at 2015-04-16 19:21:01.654782976+00:00
I0416 12:21:01.654768 367869952 process.cpp:2091] Resuming log-recover-protocol(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.654791168+00:00
I0416 12:21:01.654835 367869952 recover.cpp:154] Broadcast request completed
I0416 12:21:01.654892 368406528 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.654915072+00:00
I0416 12:21:01.654942 368406528 replica.cpp:641] Replica in STARTING status received a broadcasted recover request
I0416 12:21:01.655048 367333376 process.cpp:2091] Resuming (4)@127.0.0.1:52945 at 2015-04-16 19:21:01.655097088+00:00
I0416 12:21:01.655110 366796800 process.cpp:2091] Resuming (5)@127.0.0.1:52945 at 2015-04-16 19:21:01.655124992+00:00
I0416 12:21:01.655139 366796800 process.cpp:2091] Resuming log-recover-protocol(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.655148032+00:00
I0416 12:21:01.655169 2105078528 process.cpp:2081] Spawned process (5)@127.0.0.1:52945
I0416 12:21:01.655215 366796800 recover.cpp:195] Received a recover response from a replica in STARTING status
I0416 12:21:01.655246 367333376 process.cpp:2198] Cleaning up (4)@127.0.0.1:52945
I0416 12:21:01.655344 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.655368192+00:00
I0416 12:21:01.655405 365723648 process.cpp:2091] Resuming __latch__(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.655430144+00:00
I0416 12:21:01.655441 2105078528 process.cpp:2081] Spawned process (6)@127.0.0.1:52945
I0416 12:21:01.655452 365723648 process.cpp:2198] Cleaning up __latch__(2)@127.0.0.1:52945
I0416 12:21:01.655447 367333376 process.cpp:2091] Resuming (6)@127.0.0.1:52945 at 2015-04-16 19:21:01.655467008+00:00
I0416 12:21:01.655568 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.655589120+00:00
I0416 12:21:01.655596 367333376 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.655617024+00:00
I0416 12:21:01.655603 367869952 process.cpp:2091] Resuming __fetcher__@127.0.0.1:52945 at 2015-04-16 19:21:01.655620096+00:00
I0416 12:21:01.655654 2105078528 process.cpp:2081] Spawned process __fetcher__@127.0.0.1:52945
I0416 12:21:01.655663 367333376 recover.cpp:566] Updating replica status to VOTING
I0416 12:21:01.655693 366796800 process.cpp:2198] Cleaning up log-recover-protocol(2)@127.0.0.1:52945
I0416 12:21:01.655778 365723648 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.655799040+00:00
I0416 12:21:01.655797 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.655819776+00:00
I0416 12:21:01.655854 2105078528 containerizer.cpp:113] Using isolation: posix/cpu,posix/mem
I0416 12:21:01.656002 2105078528 process.cpp:2081] Spawned process (7)@127.0.0.1:52945
I0416 12:21:01.655997 365723648 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 193us
I0416 12:21:01.656008 368406528 process.cpp:2091] Resuming (7)@127.0.0.1:52945 at 2015-04-16 19:21:01.656030976+00:00
I0416 12:21:01.656043 365723648 replica.cpp:323] Persisted replica status to VOTING
I0416 12:21:01.656143 2105078528 process.cpp:2081] Spawned process (8)@127.0.0.1:52945
I0416 12:21:01.656160 368943104 process.cpp:2091] Resuming (8)@127.0.0.1:52945 at 2015-04-16 19:21:01.656179968+00:00
I0416 12:21:01.656152 368406528 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.656173056+00:00
I0416 12:21:01.656216 368406528 recover.cpp:580] Successfully joined the Paxos group
I0416 12:21:01.656354 368406528 recover.cpp:464] Recover process terminated
I0416 12:21:01.656347 365723648 process.cpp:2091] Resuming log(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.656369920+00:00
I0416 12:21:01.656373 368406528 process.cpp:2198] Cleaning up log-recover(1)@127.0.0.1:52945
I0416 12:21:01.656404 365723648 log.cpp:358] Log recovery completed
I0416 12:21:01.656476 367869952 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.656493824+00:00
I0416 12:21:01.656512 369479680 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.656531200+00:00
I0416 12:21:01.656528 367869952 process.cpp:2091] Resuming log-reader(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.656544000+00:00
I0416 12:21:01.656625 366796800 process.cpp:2091] Resuming log-writer(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.656647936+00:00
I0416 12:21:01.656646 2105078528 process.cpp:2081] Spawned process (9)@127.0.0.1:52945
I0416 12:21:01.656658 367333376 process.cpp:2091] Resuming (9)@127.0.0.1:52945 at 2015-04-16 19:21:01.656680192+00:00
I0416 12:21:01.657052 368943104 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.657074944+00:00
I0416 12:21:01.657438 2105078528 process.cpp:2081] Spawned process __limiter__(2)@127.0.0.1:52945
I0416 12:21:01.657449 367869952 process.cpp:2091] Resuming __limiter__(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.657474048+00:00
I0416 12:21:01.657488 366796800 process.cpp:2091] Resuming monitor@127.0.0.1:52945 at 2015-04-16 19:21:01.657510144+00:00
I0416 12:21:01.657542 2105078528 process.cpp:2081] Spawned process monitor@127.0.0.1:52945
I0416 12:21:01.657567 365723648 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.657587968+00:00
I0416 12:21:01.657670 2105078528 process.cpp:2081] Spawned process slave(1)@127.0.0.1:52945
I0416 12:21:01.657703 367333376 process.cpp:2091] Resuming slave(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.657718784+00:00
I0416 12:21:01.657738 367333376 slave.cpp:176] Slave started on 1)@127.0.0.1:52945
I0416 12:21:01.657760 367333376 slave.cpp:177] Flags at startup: --authenticatee="crammd5" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_remove_delay="6hrs" --docker_sandbox_directory="/mnt/mesos/sandbox" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/Users/osxbuildbot/buildslave/slave/dev_test_osx/dev_distcheck_build_dist/mesos-0.23.0/_build/src" --logbufsecs="0" --logging_level="INFO" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --strict="true" --switch_user="true" --version="false" --work_dir="/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/0"
W0416 12:21:01.657997 367333376 slave.cpp:180]
**************************************************
Slave bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
I0416 12:21:01.658305 367333376 slave.cpp:316] Slave resources: cpus(*):2; mem(*):10240; disk(*):233432; ports(*):[31000-32000]
I0416 12:21:01.658526 2105078528 process.cpp:2081] Spawned process (10)@127.0.0.1:52945
I0416 12:21:01.658538 366796800 process.cpp:2091] Resuming (10)@127.0.0.1:52945 at 2015-04-16 19:21:01.658558976+00:00
I0416 12:21:01.658727 2105078528 process.cpp:2081] Spawned process (11)@127.0.0.1:52945
I0416 12:21:01.658721 367333376 slave.cpp:346] Slave hostname: localhost
I0416 12:21:01.658736 368943104 process.cpp:2091] Resuming (11)@127.0.0.1:52945 at 2015-04-16 19:21:01.658759168+00:00
I0416 12:21:01.658768 367333376 slave.cpp:351] Slave checkpoint: true
I0416 12:21:01.658927 2105078528 containerizer.cpp:113] Using isolation: posix/cpu,posix/mem
I0416 12:21:01.658936 367869952 process.cpp:2091] Resuming (6)@127.0.0.1:52945 at 2015-04-16 19:21:01.658972928+00:00
I0416 12:21:01.659078 2105078528 process.cpp:2081] Spawned process (12)@127.0.0.1:52945
I0416 12:21:01.659090 365723648 process.cpp:2091] Resuming (12)@127.0.0.1:52945 at 2015-04-16 19:21:01.659112960+00:00
I0416 12:21:01.659210 2105078528 process.cpp:2081] Spawned process (13)@127.0.0.1:52945
I0416 12:21:01.659231 366796800 process.cpp:2091] Resuming (13)@127.0.0.1:52945 at 2015-04-16 19:21:01.659246848+00:00
I0416 12:21:01.659433 369479680 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.659454976+00:00
I0416 12:21:01.659456 365723648 process.cpp:2091] Resuming (14)@127.0.0.1:52945 at 2015-04-16 19:21:01.659475968+00:00
I0416 12:21:01.659478 2105078528 process.cpp:2081] Spawned process (14)@127.0.0.1:52945
I0416 12:21:01.659782 367869952 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.659796992+00:00
I0416 12:21:01.660182 2105078528 process.cpp:2081] Spawned process __limiter__(3)@127.0.0.1:52945
I0416 12:21:01.660195 365723648 process.cpp:2091] Resuming __limiter__(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.660218112+00:00
I0416 12:21:01.660320 2105078528 process.cpp:2081] Spawned process slave(2)@127.0.0.1:52945
I0416 12:21:01.660328 365723648 process.cpp:2091] Resuming slave(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.660350976+00:00
I0416 12:21:01.660370 365723648 slave.cpp:176] Slave started on 2)@127.0.0.1:52945
I0416 12:21:01.660398 365723648 slave.cpp:177] Flags at startup: --authenticatee="crammd5" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_remove_delay="6hrs" --docker_sandbox_directory="/mnt/mesos/sandbox" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/Users/osxbuildbot/buildslave/slave/dev_test_osx/dev_distcheck_build_dist/mesos-0.23.0/_build/src" --logbufsecs="0" --logging_level="INFO" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --strict="true" --switch_user="true" --version="false" --work_dir="/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/1"
W0416 12:21:01.660644 365723648 slave.cpp:180]
**************************************************
Slave bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
I0416 12:21:01.660853 365723648 slave.cpp:316] Slave resources: cpus(*):2; mem(*):10240; disk(*):233432; ports(*):[31000-32000]
I0416 12:21:01.661152 365723648 slave.cpp:346] Slave hostname: localhost
I0416 12:21:01.661175 365723648 slave.cpp:351] Slave checkpoint: true
I0416 12:21:01.661226 368406528 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.661250048+00:00
I0416 12:21:01.661245 368943104 process.cpp:2091] Resuming (11)@127.0.0.1:52945 at 2015-04-16 19:21:01.661270016+00:00
I0416 12:21:01.661363 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.661381888+00:00
I0416 12:21:01.661384 2105078528 process.cpp:2081] Spawned process (15)@127.0.0.1:52945
I0416 12:21:01.661396 367333376 process.cpp:2081] Spawned process __async_executor__(1)@127.0.0.1:52945
I0416 12:21:01.661409 367869952 process.cpp:2091] Resuming (15)@127.0.0.1:52945 at 2015-04-16 19:21:01.661426176+00:00
I0416 12:21:01.661424 368943104 process.cpp:2091] Resuming __async_executor__(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.661444864+00:00
I0416 12:21:01.661437 369479680 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.661457920+00:00
I0416 12:21:01.661505 368943104 state.cpp:35] Recovering state from '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/0/meta'
I0416 12:21:01.661540 365723648 process.cpp:2081] Spawned process __async_executor__(2)@127.0.0.1:52945
I0416 12:21:01.661530 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.661549056+00:00
I0416 12:21:01.661551 367869952 process.cpp:2091] Resuming __async_executor__(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.661587968+00:00
I0416 12:21:01.661619 2105078528 process.cpp:2081] Spawned process (16)@127.0.0.1:52945
I0416 12:21:01.661615 366796800 process.cpp:2091] Resuming (16)@127.0.0.1:52945 at 2015-04-16 19:21:01.661634816+00:00
I0416 12:21:01.661654 367869952 state.cpp:35] Recovering state from '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/1/meta'
I0416 12:21:01.661659 368943104 process.cpp:2198] Cleaning up __async_executor__(1)@127.0.0.1:52945
I0416 12:21:01.661775 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.661796096+00:00
I0416 12:21:01.661835 2105078528 containerizer.cpp:113] Using isolation: posix/cpu,posix/mem
I0416 12:21:01.661828 365723648 process.cpp:2091] Resuming slave(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.661849856+00:00
I0416 12:21:01.661828 368406528 process.cpp:2091] Resuming (6)@127.0.0.1:52945 at 2015-04-16 19:21:01.661850880+00:00
I0416 12:21:01.661852 367869952 process.cpp:2198] Cleaning up __async_executor__(2)@127.0.0.1:52945
I0416 12:21:01.661906 2105078528 process.cpp:2081] Spawned process (17)@127.0.0.1:52945
I0416 12:21:01.661903 368406528 status_update_manager.cpp:197] Recovering status update manager
I0416 12:21:01.661912 367333376 process.cpp:2091] Resuming (17)@127.0.0.1:52945 at 2015-04-16 19:21:01.661935104+00:00
I0416 12:21:01.661911 366796800 process.cpp:2091] Resuming (11)@127.0.0.1:52945 at 2015-04-16 19:21:01.661954048+00:00
I0416 12:21:01.661938 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.661967872+00:00
I0416 12:21:01.661979 366796800 status_update_manager.cpp:197] Recovering status update manager
I0416 12:21:01.661993 2105078528 process.cpp:2081] Spawned process (18)@127.0.0.1:52945
I0416 12:21:01.662013 365723648 process.cpp:2091] Resuming (18)@127.0.0.1:52945 at 2015-04-16 19:21:01.662036992+00:00
I0416 12:21:01.662022 367869952 process.cpp:2091] Resuming slave(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.662044928+00:00
I0416 12:21:01.662078 368406528 process.cpp:2091] Resuming slave(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.662102016+00:00
I0416 12:21:01.662204 369479680 process.cpp:2091] Resuming (9)@127.0.0.1:52945 at 2015-04-16 19:21:01.662226944+00:00
I0416 12:21:01.662245 366796800 process.cpp:2091] Resuming (14)@127.0.0.1:52945 at 2015-04-16 19:21:01.662259200+00:00
I0416 12:21:01.662256 369479680 containerizer.cpp:310] Recovering containerizer
I0416 12:21:01.662277 366796800 containerizer.cpp:310] Recovering containerizer
I0416 12:21:01.662314 367869952 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.662337024+00:00
I0416 12:21:01.662366 2105078528 process.cpp:2081] Spawned process (19)@127.0.0.1:52945
I0416 12:21:01.662379 368406528 process.cpp:2091] Resuming (19)@127.0.0.1:52945 at 2015-04-16 19:21:01.662401024+00:00
I0416 12:21:01.662529 365723648 process.cpp:2091] Resuming (12)@127.0.0.1:52945 at 2015-04-16 19:21:01.662550016+00:00
I0416 12:21:01.662535 367869952 process.cpp:2091] Resuming (7)@127.0.0.1:52945 at 2015-04-16 19:21:01.662552064+00:00
I0416 12:21:01.662574 368943104 process.cpp:2091] Resuming (13)@127.0.0.1:52945 at 2015-04-16 19:21:01.662606080+00:00
I0416 12:21:01.662621 367869952 process.cpp:2091] Resuming (8)@127.0.0.1:52945 at 2015-04-16 19:21:01.662644992+00:00
I0416 12:21:01.662680 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.662700800+00:00
I0416 12:21:01.662730 368943104 process.cpp:2091] Resuming (20)@127.0.0.1:52945 at 2015-04-16 19:21:01.662752000+00:00
I0416 12:21:01.662750 367333376 process.cpp:2091] Resuming (21)@127.0.0.1:52945 at 2015-04-16 19:21:01.662771968+00:00
I0416 12:21:01.662768 366796800 process.cpp:2081] Spawned process (20)@127.0.0.1:52945
I0416 12:21:01.662792 369479680 process.cpp:2081] Spawned process (21)@127.0.0.1:52945
I0416 12:21:01.662801 367869952 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.662818048+00:00
I0416 12:21:01.662976 365723648 process.cpp:2091] Resuming (9)@127.0.0.1:52945 at 2015-04-16 19:21:01.662996992+00:00
I0416 12:21:01.663003 367333376 process.cpp:2198] Cleaning up (21)@127.0.0.1:52945
I0416 12:21:01.662993 368406528 process.cpp:2091] Resuming (14)@127.0.0.1:52945 at 2015-04-16 19:21:01.663014912+00:00
I0416 12:21:01.663022 368943104 process.cpp:2198] Cleaning up (20)@127.0.0.1:52945
I0416 12:21:01.663079 366796800 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.663098112+00:00
I0416 12:21:01.663130 366796800 process.cpp:2091] Resuming slave(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.663148032+00:00
I0416 12:21:01.663162 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.663183872+00:00
I0416 12:21:01.663200 365723648 process.cpp:2091] Resuming slave(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.663222016+00:00
I0416 12:21:01.663219 367333376 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.663239936+00:00
I0416 12:21:01.663233 366796800 slave.cpp:3806] Finished recovery
I0416 12:21:01.663290 365723648 slave.cpp:3806] Finished recovery
I0416 12:21:01.663611 368943104 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.663631872+00:00
I0416 12:21:01.663646 2105078528 process.cpp:2081] Spawned process __limiter__(4)@127.0.0.1:52945
I0416 12:21:01.663648 369479680 process.cpp:2091] Resuming __limiter__(4)@127.0.0.1:52945 at 2015-04-16 19:21:01.663665920+00:00
I0416 12:21:01.663733 2105078528 process.cpp:2081] Spawned process slave(3)@127.0.0.1:52945
I0416 12:21:01.663791 367869952 process.cpp:2091] Resuming slave(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.663815936+00:00
I0416 12:21:01.663836 367869952 slave.cpp:176] Slave started on 3)@127.0.0.1:52945
I0416 12:21:01.663892 2105078528 pid.cpp:82] Attempting to parse 'master@127.0.0.1:52945' into a PID
F0416 12:21:01.664001 365723648 slave.cpp:3816] CHECK_SOME(state::checkpoint(path, bootId.get())): Failed to rename '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/0/meta/EntzLz' to '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/1/meta/boot_id': No such file or directory
*** Check failure stack trace: ***
I0416 12:21:01.663998 368406528 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.664019968+00:00
I0416 12:21:01.664113 367333376 process.cpp:2091] Resuming slave(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.664136960+00:00
I0416 12:21:01.664203 367333376 pid.cpp:82] Attempting to parse 'master@127.0.0.1:52945' into a PID
I0416 12:21:01.664191 369479680 process.cpp:2091] Resuming (6)@127.0.0.1:52945 at 2015-04-16 19:21:01.664210944+00:00
I0416 12:21:01.663877 367869952 slave.cpp:177] Flags at startup: --authenticatee="crammd5" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_remove_delay="6hrs" --docker_sandbox_directory="/mnt/mesos/sandbox" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/Users/osxbuildbot/buildslave/slave/dev_test_osx/dev_distcheck_build_dist/mesos-0.23.0/_build/src" --logbufsecs="0" --logging_level="INFO" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --strict="true" --switch_user="true" --version="false" --work_dir="/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/2"
I0416 12:21:01.664229 369479680 status_update_manager.cpp:171] Pausing sending status updates
W0416 12:21:01.664244 367869952 slave.cpp:180]
**************************************************
Slave bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
I0416 12:21:01.664242 367333376 slave.cpp:627] New master detected at master@127.0.0.1:52945
I0416 12:21:01.664312 367333376 slave.cpp:652] No credentials provided. Attempting to register without authentication
I0416 12:21:01.664378 367333376 slave.cpp:663] Detecting new master
I0416 12:21:01.664428 366796800 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.664446208+00:00
I0416 12:21:01.664546 367869952 slave.cpp:316] Slave resources: cpus(*):2; mem(*):10240; disk(*):233432; ports(*):[31000-32000]
I0416 12:21:01.664926 367869952 slave.cpp:346] Slave hostname: localhost
I0416 12:21:01.664950 367869952 slave.cpp:351] Slave checkpoint: true
I0416 12:21:01.665007 368406528 process.cpp:2091] Resuming (16)@127.0.0.1:52945 at 2015-04-16 19:21:01.665029888+00:00
I0416 12:21:01.665225 369479680 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.665245184+00:00
I0416 12:21:01.665330 367333376 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.665344000+00:00
I0416 12:21:01.665340 366796800 process.cpp:2091] Resuming __async_executor__(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.665360896+00:00
I0416 12:21:01.665375 367869952 process.cpp:2081] Spawned process __async_executor__(3)@127.0.0.1:52945
I0416 12:21:01.665475 367333376 process.cpp:2091] Resuming __async_executor__(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.665491968+00:00
I0416 12:21:01.665511 367333376 state.cpp:35] Recovering state from '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/2/meta'
I0416 12:21:01.665587 367333376 process.cpp:2198] Cleaning up __async_executor__(3)@127.0.0.1:52945
I0416 12:21:01.665627 369479680 process.cpp:2091] Resuming (16)@127.0.0.1:52945 at 2015-04-16 19:21:01.665649920+00:00
I0416 12:21:01.665645 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.665666048+00:00
I0416 12:21:01.665680 369479680 status_update_manager.cpp:197] Recovering status update manager
I0416 12:21:01.665772 368406528 process.cpp:2091] Resuming slave(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.665793792+00:00
I0416 12:21:01.665886 367333376 process.cpp:2091] Resuming (19)@127.0.0.1:52945 at 2015-04-16 19:21:01.665904896+00:00
I0416 12:21:01.665922 367333376 containerizer.cpp:310] Recovering containerizer
I0416 12:21:01.666064 369479680 process.cpp:2091] Resuming (18)@127.0.0.1:52945 at 2015-04-16 19:21:01.666081792+00:00
I0416 12:21:01.666057 368943104 process.cpp:2091] Resuming (17)@127.0.0.1:52945 at 2015-04-16 19:21:01.666080000+00:00
I0416 12:21:01.666128 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.666151168+00:00
I0416 12:21:01.666173 367333376 process.cpp:2081] Spawned process (22)@127.0.0.1:52945
I0416 12:21:01.666196 368943104 process.cpp:2091] Resuming (22)@127.0.0.1:52945 at 2015-04-16 19:21:01.666230016+00:00
I0416 12:21:01.666328 366796800 process.cpp:2091] Resuming (19)@127.0.0.1:52945 at 2015-04-16 19:21:01.666349824+00:00
I0416 12:21:01.666349 368943104 process.cpp:2198] Cleaning up (22)@127.0.0.1:52945
I0416 12:21:01.666451 367869952 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.666472960+00:00
I0416 12:21:01.666476 368406528 process.cpp:2091] Resuming slave(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.666493952+00:00
I0416 12:21:01.666559 368406528 slave.cpp:3806] Finished recovery
I0416 12:21:01.668668 368943104 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.668697088+00:00
I0416 12:21:01.668804 367869952 process.cpp:2091] Resuming slave(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.668827904+00:00
I0416 12:21:01.668913 367869952 pid.cpp:82] Attempting to parse 'master@127.0.0.1:52945' into a PID
I0416 12:21:01.668900 368406528 process.cpp:2091] Resuming (16)@127.0.0.1:52945 at 2015-04-16 19:21:01.668921856+00:00
I0416 12:21:01.668959 368406528 status_update_manager.cpp:171] Pausing sending status updates
I0416 12:21:01.668977 367869952 slave.cpp:627] New master detected at master@127.0.0.1:52945
I0416 12:21:01.669018 367869952 slave.cpp:652] No credentials provided. Attempting to register without authentication
I0416 12:21:01.669067 367869952 slave.cpp:663] Detecting new master
I0416 12:21:01.669113 368943104 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.669146112+00:00
I0416 12:21:01.672554 2105078528 process.cpp:2081] Spawned process standalone-master-detector(2)@127.0.0.1:52945
I0416 12:21:01.672559 366796800 process.cpp:2091] Resuming standalone-master-detector(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.672573184+00:00
I0416 12:21:01.672622 2105078528 process.cpp:2081] Spawned process scheduler(1)@127.0.0.1:52945
I0416 12:21:01.672631 369479680 process.cpp:2091] Resuming scheduler(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.672656128+00:00
I0416 12:21:01.672837 368406528 process.cpp:2091] Resuming standalone-master-detector(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.672858880+00:00
I0416 12:21:01.672935 369479680 process.cpp:2091] Resuming scheduler(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.672957952+00:00
I0416 12:21:01.672996 369479680 pid.cpp:82] Attempting to parse 'master@127.0.0.1:52945' into a PID
I0416 12:21:01.673032 369479680 scheduler.cpp:420] New master detected at master@127.0.0.1:52945
I0416 12:21:01.673056 369479680 scheduler.cpp:469] Authenticating with master master@127.0.0.1:52945
I0416 12:21:01.673131 369479680 process.cpp:2081] Spawned process crammd5_authenticatee(1)@127.0.0.1:52945
I0416 12:21:01.673141 367333376 process.cpp:2091] Resuming crammd5_authenticatee(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.673160192+00:00
I0416 12:21:01.673285 366796800 process.cpp:2091] Resuming standalone-master-detector(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.673306880+00:00
I0416 12:21:01.673476 367333376 authenticatee.hpp:115] Initializing client SASL
@ 0x110e9284a google::LogMessage::Fail()
@ 0x110e917dd google::LogMessage::SendToLog()
@ 0x110e924ea google::LogMessage::Flush()
@ 0x110e99348 google::LogMessageFatal::~LogMessageFatal()
I0416 12:21:01.747539 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:33.747597056+00:00
@ 0x110e92ca5 google::LogMessageFatal::~LogMessageFatal()
@ 0x10f3d33d3 _CheckFatal::~_CheckFatal()
@ 0x10f3d3025 _CheckFatal::~_CheckFatal()
@ 0x10fd94da6 mesos::internal::slave::Slave::__recover()
@ 0x10fe7f09d _ZZN7process8dispatchIN5mesos8internal5slave5SlaveERKNS_6FutureI7NothingEES7_EEvRKNS_3PIDIT_EEMSB_FvT0_ET1_ENKUlPNS_11ProcessBaseEE_clESK_
@ 0x10fe7ee7f _ZNSt3__110__function6__funcIZN7process8dispatchIN5mesos8internal5slave5SlaveERKNS2_6FutureI7NothingEESA_EEvRKNS2_3PIDIT_EEMSE_FvT0_ET1_EUlPNS2_11ProcessBaseEE_NS_9allocatorISO_EEFvSN_EEclEOSN_
@ 0x110d74e7b std::__1::function<>::operator()()
@ 0x110d5c5bf process::ProcessBase::visit()
@ 0x110de6c0e process::DispatchEvent::visit()
@ 0x10f3d0841 process::ProcessBase::serve()
@ 0x110d45abe process::ProcessManager::resume()
@ 0x110d451de process::schedule()
@ 0x7fff8f1eb268 _pthread_body
@ 0x7fff8f1eb1e5 _pthread_start
@ 0x7fff8f1e941d thread_start
I0416 12:21:01.848065 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:33.848114944+00:00
I0416 12:21:01.948398 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:33.948432896+00:00
I0416 12:21:02.048640 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.048694016+00:00
I0416 12:21:02.148989 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.149024000+00:00
I0416 12:21:02.252266 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.252296960+00:00
I0416 12:21:02.352577 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.352615168+00:00
I0416 12:21:02.452965 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.452996864+00:00
I0416 12:21:02.553390 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.553423872+00:00
I0416 12:21:02.653735 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.653764864+00:00
I0416 12:21:02.756922 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.756952064+00:00
I0416 12:21:02.857390 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.857434880+00:00
I0416 12:21:02.961252 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.961291008+00:00
I0416 12:21:03.061522 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.061554944+00:00
I0416 12:21:03.166790 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.166821120+00:00
I0416 12:21:03.267645 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.267698176+00:00
I0416 12:21:03.367985 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.368038912+00:00
I0416 12:21:03.468283 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.468316928+00:00
I0416 12:21:03.568531 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.568573952+00:00
I0416 12:21:03.668767 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.668807936+00:00
I0416 12:21:03.769104 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.769152000+00:00
I0416 12:21:03.869385 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.869432832+00:00
I0416 12:21:03.969768 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.969798144+00:00
I0416 12:21:04.074630 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.074662912+00:00
I0416 12:21:04.174993 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.175024128+00:00
I0416 12:21:04.275383 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.275419904+00:00
I0416 12:21:04.375689 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.375726848+00:00
I0416 12:21:04.476035 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.476072960+00:00
I0416 12:21:04.576369 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.576424960+00:00
I0416 12:21:04.676823 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.676861952+00:00
I0416 12:21:04.777240 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.777278976+00:00
I0416 12:21:04.877797 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.877843968+00:00
I0416 12:21:04.978090 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.978121984+00:00
I0416 12:21:05.079474 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.079504896+00:00
I0416 12:21:05.181157 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.181199104+00:00
I0416 12:21:05.281492 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.281528064+00:00
I0416 12:21:05.381813 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.381846016+00:00
I0416 12:21:05.482925 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.482966784+00:00
I0416 12:21:05.583298 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.583326208+00:00
I0416 12:21:05.683570 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.683599872+00:00
I0416 12:21:05.785339 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.785371136+00:00
I0416 12:21:05.885790 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.885846016+00:00
I0416 12:21:05.986150 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.986181120+00:00
I0416 12:21:06.086530 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.086565120+00:00
I0416 12:21:06.187046 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.187074816+00:00
I0416 12:21:06.287353 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.287381760+00:00
I0416 12:21:06.390130 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.390161920+00:00
I0416 12:21:06.490376 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.490406912+00:00
I0416 12:21:06.564841 143380480 process.cpp:2091] Resuming (1)@127.0.0.1:52923 at 2015-04-16 19:21:06.564875008+00:00
I0416 12:21:06.564909 143380480 exec.cpp:86] Committing suicide by killing the process group
I0416 12:21:06.566997 344678400 process.cpp:2091] Resuming (1)@127.0.0.1:52922 at 2015-04-16 19:21:06.567048192+00:00
I0416 12:21:06.567098 344678400 exec.cpp:86] Committing suicide by killing the process group
I0416 12:21:06.568764 136261632 process.cpp:2091] Resuming (1)@127.0.0.1:52921 at 2015-04-16 19:21:06.568818176+00:00
I0416 12:21:06.568856 136261632 exec.cpp:86] Committing suicide by killing the process group
I0416 12:21:06.590653 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.590689792+00:00
I0416 12:21:06.694011 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.694044160+00:00
I0416 12:21:06.794332 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.794361088+00:00
I0416 12:21:06.894798 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.894850048+00:00
I0416 12:21:06.995139 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.995166976+00:00
I0416 12:21:07.095384 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.095429888+00:00
I0416 12:21:07.195736 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.195770112+00:00
I0416 12:21:07.296052 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.296078848+00:00
I0416 12:21:07.396627 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.396659968+00:00
I0416 12:21:07.497102 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.497129984+00:00
I0416 12:21:07.597435 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.597464064+00:00
I0416 12:21:07.697954 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.697988864+00:00
I0416 12:21:07.798230 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.798292992+00:00
I0416 12:21:07.898574 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.898610176+00:00
I0416 12:21:08.003020 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.003076096+00:00
I0416 12:21:08.103488 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.103524864+00:00
I0416 12:21:08.203766 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.203793152+00:00
I0416 12:21:08.304029 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.304101888+00:00
I0416 12:21:08.355402 311099392 process.cpp:2048] Dropping event for process slave(1)@127.0.0.1:52842
I0416 12:21:08.404270 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.404307968+00:00
I0416 12:21:08.508119 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.508163072+00:00
I0416 12:21:08.520028 311099392 process.cpp:2048] Dropping event for process slave(2)@127.0.0.1:52842
I0416 12:21:08.608477 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.608509952+00:00
I0416 12:21:08.713762 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.713812224+00:00
I0416 12:21:08.814121 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.814168064+00:00
I0416 12:21:08.841398 311099392 process.cpp:2048] Dropping event for process slave(3)@127.0.0.1:52842
I0416 12:21:08.914386 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.914430976+00:00
I0416 12:21:09.019639 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:41.019673088+00:00
I0416 12:21:09.119954 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:41.120000000+00:00
I0416 12:21:09.224282 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:41.224335872+00:00
I0416 12:21:09.326063 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:41.326106112+00:00
../../src/tests/script.cpp:83: Failure
Failed
low_level_scheduler_pthread_test.sh terminated with signal Abort trap: 6
[ FAILED ] ExamplesTest.LowLevelSchedulerPthread (7774 ms)
{code}
was (Author: cmaloney):
Just got one of these with full backtrace:
{code}
I0416 12:21:01.673476 367333376 authenticatee.hpp:115] Initializing client SASL
@ 0x110e9284a google::LogMessage::Fail()
@ 0x110e917dd google::LogMessage::SendToLog()
@ 0x110e924ea google::LogMessage::Flush()
@ 0x110e99348 google::LogMessageFatal::~LogMessageFatal()
I0416 12:21:01.747539 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:33.747597056+00:00
@ 0x110e92ca5 google::LogMessageFatal::~LogMessageFatal()
@ 0x10f3d33d3 _CheckFatal::~_CheckFatal()
@ 0x10f3d3025 _CheckFatal::~_CheckFatal()
@ 0x10fd94da6 mesos::internal::slave::Slave::__recover()
@ 0x10fe7f09d _ZZN7process8dispatchIN5mesos8internal5slave5SlaveERKNS_6FutureI7NothingEES7_EEvRKNS_3PIDIT_EEMSB_FvT0_ET1_ENKUlPNS_11ProcessBaseEE_clESK_
@ 0x10fe7ee7f _ZNSt3__110__function6__funcIZN7process8dispatchIN5mesos8internal5slave5SlaveERKNS2_6FutureI7NothingEESA_EEvRKNS2_3PIDIT_EEMSE_FvT0_ET1_EUlPNS2_11ProcessBaseEE_NS_9allocatorISO_EEFvSN_EEclEOSN_
@ 0x110d74e7b std::__1::function<>::operator()()
@ 0x110d5c5bf process::ProcessBase::visit()
@ 0x110de6c0e process::DispatchEvent::visit()
@ 0x10f3d0841 process::ProcessBase::serve()
@ 0x110d45abe process::ProcessManager::resume()
@ 0x110d451de process::schedule()
@ 0x7fff8f1eb268 _pthread_body
@ 0x7fff8f1eb1e5 _pthread_start
@ 0x7fff8f1e941d thread_start
{code}
The full log from the test (MESOS_VERBOSE, GLOG_v=2)
{code}
[ RUN ] ExamplesTest.LowLevelSchedulerPthread
Using temporary directory '/tmp/ExamplesTest_LowLevelSchedulerPthread_vVqryS'
I0416 12:21:01.637110 2105078528 logging.cpp:177] Logging to STDERR
Enabling authentication for the scheduler
I0416 12:21:01.639566 2105078528 process.cpp:2081] Spawned process __gc__@127.0.0.1:52945
I0416 12:21:01.639770 2105078528 process.cpp:2081] Spawned process help@127.0.0.1:52945
I0416 12:21:01.639583 365723648 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.639622912+00:00
I0416 12:21:01.639777 367869952 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.639796992+00:00
I0416 12:21:01.639875 366260224 process.cpp:2091] Resuming logging@127.0.0.1:52945 at 2015-04-16 19:21:01.639906816+00:00
I0416 12:21:01.639909 2105078528 process.cpp:2081] Spawned process logging@127.0.0.1:52945
I0416 12:21:01.639978 367869952 process.cpp:2091] Resuming profiler@127.0.0.1:52945 at 2015-04-16 19:21:01.640003840+00:00
I0416 12:21:01.640033 368943104 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.640058880+00:00
I0416 12:21:01.640051 2105078528 process.cpp:2081] Spawned process profiler@127.0.0.1:52945
I0416 12:21:01.640246 368406528 process.cpp:2091] Resuming system@127.0.0.1:52945 at 2015-04-16 19:21:01.640268032+00:00
I0416 12:21:01.640236 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.640258048+00:00
I0416 12:21:01.640318 2105078528 process.cpp:2081] Spawned process system@127.0.0.1:52945
I0416 12:21:01.640321 368943104 process.cpp:2091] Resuming __limiter__(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.640336128+00:00
I0416 12:21:01.640390 368406528 process.cpp:2081] Spawned process __limiter__(1)@127.0.0.1:52945
I0416 12:21:01.640425 365723648 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.640440064+00:00
I0416 12:21:01.640472 368406528 process.cpp:2081] Spawned process metrics@127.0.0.1:52945
I0416 12:21:01.640521 367869952 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.640538880+00:00
I0416 12:21:01.640733 366796800 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.640760064+00:00
I0416 12:21:01.640913 2105078528 process.cpp:2081] Spawned process __processes__@127.0.0.1:52945
I0416 12:21:01.640919 366796800 process.cpp:2091] Resuming __processes__@127.0.0.1:52945 at 2015-04-16 19:21:01.640937984+00:00
I0416 12:21:01.640949 2105078528 process.cpp:912] libprocess is initialized on 127.0.0.1:52945 for 8 cpus
I0416 12:21:01.640971 365723648 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.640985856+00:00
W0416 12:21:01.641326 2105078528 scheduler.cpp:134]
**************************************************
Scheduler driver bound to loopback interface! Cannot communicate with remote master(s). You might want to set 'LIBPROCESS_IP' environment variable to use a routable IP address.
**************************************************
I0416 12:21:01.641348 2105078528 scheduler.cpp:149] Version: 0.23.0
I0416 12:21:01.642009 2105078528 process.cpp:2081] Spawned process hierarchical-allocator(1)@127.0.0.1:52945
I0416 12:21:01.642019 368943104 process.cpp:2091] Resuming hierarchical-allocator(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.642038016+00:00
I0416 12:21:01.642069 365723648 process.cpp:2091] Resuming files@127.0.0.1:52945 at 2015-04-16 19:21:01.642086912+00:00
I0416 12:21:01.642115 2105078528 process.cpp:2081] Spawned process files@127.0.0.1:52945
I0416 12:21:01.642194 366260224 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.642208000+00:00
I0416 12:21:01.647266 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:33.647293184+00:00
I0416 12:21:01.648769 2105078528 leveldb.cpp:176] Opened db in 2269us
I0416 12:21:01.649345 2105078528 leveldb.cpp:183] Compacted db in 555us
I0416 12:21:01.649394 2105078528 leveldb.cpp:198] Created db iterator in 27us
I0416 12:21:01.649420 2105078528 leveldb.cpp:204] Seeked to beginning of db in 14us
I0416 12:21:01.649440 2105078528 leveldb.cpp:273] Iterated through 0 keys in the db in 15us
I0416 12:21:01.649510 2105078528 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
I0416 12:21:01.650113 2105078528 process.cpp:2081] Spawned process log-replica(1)@127.0.0.1:52945
I0416 12:21:01.650157 366796800 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650183936+00:00
I0416 12:21:01.650243 368943104 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650263040+00:00
I0416 12:21:01.650296 2105078528 process.cpp:2081] Spawned process (1)@127.0.0.1:52945
I0416 12:21:01.650413 2105078528 process.cpp:2081] Spawned process log(1)@127.0.0.1:52945
I0416 12:21:01.650424 365723648 process.cpp:2091] Resuming log(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650441984+00:00
I0416 12:21:01.650574 2105078528 process.cpp:2081] Spawned process log-reader(1)@127.0.0.1:52945
I0416 12:21:01.650580 366260224 process.cpp:2091] Resuming log-reader(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650599936+00:00
I0416 12:21:01.650606 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.650621952+00:00
I0416 12:21:01.650607 367333376 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650633984+00:00
I0416 12:21:01.650658 367333376 recover.cpp:449] Starting replica recovery
I0416 12:21:01.650636 369479680 process.cpp:2091] Resuming log-writer(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650664960+00:00
I0416 12:21:01.650686 2105078528 process.cpp:2081] Spawned process log-writer(1)@127.0.0.1:52945
I0416 12:21:01.650688 365723648 process.cpp:2081] Spawned process log-recover(1)@127.0.0.1:52945
I0416 12:21:01.650837 365723648 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.650851072+00:00
I0416 12:21:01.650836 369479680 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.650855936+00:00
I0416 12:21:01.650857 366260224 process.cpp:2091] Resuming (2)@127.0.0.1:52945 at 2015-04-16 19:21:01.650881024+00:00
I0416 12:21:01.650900 2105078528 process.cpp:2081] Spawned process (2)@127.0.0.1:52945
I0416 12:21:01.650966 367333376 recover.cpp:475] Replica is in EMPTY status
I0416 12:21:01.651100 367869952 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.651115008+00:00
I0416 12:21:01.651093 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.651112192+00:00
I0416 12:21:01.651120 366260224 process.cpp:2091] Resuming log-recover-protocol(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651134976+00:00
I0416 12:21:01.651160 367333376 process.cpp:2081] Spawned process log-recover-protocol(1)@127.0.0.1:52945
I0416 12:21:01.651211 366260224 recover.cpp:131] Starting to wait for enough quorum of replicas before running recovery protocol, expected quroum size: 1
I0416 12:21:01.651298 2105078528 process.cpp:2081] Spawned process registrar(1)@127.0.0.1:52945
I0416 12:21:01.651301 365723648 process.cpp:2091] Resuming registrar(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651323136+00:00
I0416 12:21:01.651314 366796800 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651334144+00:00
I0416 12:21:01.651384 367333376 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651405056+00:00
I0416 12:21:01.651434 2105078528 process.cpp:2081] Spawned process standalone-master-detector(1)@127.0.0.1:52945
I0416 12:21:01.651576 2105078528 process.cpp:2081] Spawned process authorizer(1)@127.0.0.1:52945
I0416 12:21:01.651594 365723648 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.651608064+00:00
I0416 12:21:01.651600 368406528 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.651613952+00:00
I0416 12:21:01.651583 367869952 process.cpp:2091] Resuming authorizer(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651602944+00:00
I0416 12:21:01.651657 369479680 process.cpp:2091] Resuming __latch__(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.651691008+00:00
I0416 12:21:01.651705 366260224 process.cpp:2081] Spawned process __latch__(1)@127.0.0.1:52945
I0416 12:21:01.651888 366260224 recover.cpp:145] Broadcasting recover request to all replicas
I0416 12:21:01.651996 368406528 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.652011008+00:00
I0416 12:21:01.652019 367333376 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.652039168+00:00
I0416 12:21:01.652148 366796800 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.652165888+00:00
I0416 12:21:01.652154 367869952 process.cpp:2091] Resuming (3)@127.0.0.1:52945 at 2015-04-16 19:21:01.652169984+00:00
I0416 12:21:01.652217 368406528 process.cpp:2081] Spawned process (3)@127.0.0.1:52945
I0416 12:21:01.652369 366260224 process.cpp:2091] Resuming (3)@127.0.0.1:52945 at 2015-04-16 19:21:01.652391936+00:00
I0416 12:21:01.652400 368943104 process.cpp:2091] Resuming log-recover-protocol(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.652436992+00:00
I0416 12:21:01.652465 368943104 recover.cpp:154] Broadcast request completed
I0416 12:21:01.652479 368406528 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.652500992+00:00
I0416 12:21:01.652529 368406528 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request
I0416 12:21:01.652642 366260224 process.cpp:2091] Resuming (3)@127.0.0.1:52945 at 2015-04-16 19:21:01.652664064+00:00
I0416 12:21:01.652740 366260224 process.cpp:2198] Cleaning up (3)@127.0.0.1:52945
I0416 12:21:01.652799 368943104 recover.cpp:195] Received a recover response from a replica in EMPTY status
I0416 12:21:01.652814 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.652847104+00:00
I0416 12:21:01.652892 368406528 process.cpp:2091] Resuming __latch__(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.652925952+00:00
I0416 12:21:01.652953 368406528 process.cpp:2198] Cleaning up __latch__(1)@127.0.0.1:52945
I0416 12:21:01.653031 366796800 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.653053952+00:00
I0416 12:21:01.653064 367869952 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.653084160+00:00
I0416 12:21:01.653147 367869952 recover.cpp:566] Updating replica status to STARTING
I0416 12:21:01.653162 368943104 process.cpp:2198] Cleaning up log-recover-protocol(1)@127.0.0.1:52945
I0416 12:21:01.653259 366796800 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.653281024+00:00
I0416 12:21:01.653301 368406528 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.653321984+00:00
I0416 12:21:01.653651 368406528 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 295us
I0416 12:21:01.653693 368406528 replica.cpp:323] Persisted replica status to STARTING
I0416 12:21:01.653681 365723648 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.653703936+00:00
I0416 12:21:01.653806 365723648 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.653827072+00:00
I0416 12:21:01.653833 366260224 process.cpp:2091] Resuming master@127.0.0.1:52945 at 2015-04-16 19:21:01.653851136+00:00
I0416 12:21:01.653863 2105078528 process.cpp:2081] Spawned process master@127.0.0.1:52945
I0416 12:21:01.653874 366260224 master.cpp:361] Master 20150416-122101-16777343-52945-98969 (localhost) started on 127.0.0.1:52945
I0416 12:21:01.653918 365723648 recover.cpp:475] Replica is in STARTING status
I0416 12:21:01.654013 365723648 process.cpp:2081] Spawned process log-recover-protocol(2)@127.0.0.1:52945
I0416 12:21:01.654022 367333376 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.654042880+00:00
I0416 12:21:01.654026 368406528 process.cpp:2091] Resuming log-recover-protocol(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.654049024+00:00
I0416 12:21:01.654101 368406528 recover.cpp:131] Starting to wait for enough quorum of replicas before running recovery protocol, expected quroum size: 1
I0416 12:21:01.654170 367869952 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.654191872+00:00
I0416 12:21:01.653894 366260224 master.cpp:363] Flags at startup: --acls="permissive: false
register_frameworks {
principals {
type: SOME
values: "test-principal"
}
roles {
type: SOME
values: "*"
}
}
run_tasks {
principals {
type: SOME
values: "test-principal"
}
users {
type: SOME
values: "osxbuildbot"
}
}
" --allocation_interval="1secs" --authenticate="true" --authenticate_slaves="false" --authenticators="crammd5" --credentials="/tmp/ExamplesTest_LowLevelSchedulerPthread_vVqryS/credentials" --framework_sorter="drf" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" --registry_strict="false" --root_submissions="true" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/Users/osxbuildbot/buildslave/slave/dev_test_osx/dev_distcheck_build_dist/mesos-0.23.0/src/webui" --work_dir="/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ" --zk_session_timeout="10secs"
W0416 12:21:01.654228 366260224 master.cpp:366]
**************************************************
Master bound to loopback interface! Cannot communicate with remote schedulers or slaves. You might want to set '--ip' flag to a routable IP address.
**************************************************
I0416 12:21:01.654275 368406528 process.cpp:2081] Spawned process __latch__(2)@127.0.0.1:52945
I0416 12:21:01.654292 366260224 master.cpp:408] Master only allowing authenticated frameworks to register
I0416 12:21:01.654278 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.654299136+00:00
I0416 12:21:01.654280 366796800 process.cpp:2091] Resuming __latch__(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.654301952+00:00
I0416 12:21:01.654309 366260224 master.cpp:415] Master allowing unauthenticated slaves to register
I0416 12:21:01.654340 366260224 credentials.hpp:37] Loading credentials for authentication from '/tmp/ExamplesTest_LowLevelSchedulerPthread_vVqryS/credentials'
I0416 12:21:01.654362 368406528 recover.cpp:145] Broadcasting recover request to all replicas
I0416 12:21:01.654424 365723648 process.cpp:2091] Resuming (1)@127.0.0.1:52945 at 2015-04-16 19:21:01.654445056+00:00
I0416 12:21:01.654500 366260224 master.cpp:452] Using default 'crammd5' authenticator
I0416 12:21:01.654549 366260224 authenticator.cpp:511] Initializing server SASL
I0416 12:21:01.654570 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.654592000+00:00
I0416 12:21:01.654582 368943104 process.cpp:2091] Resuming (4)@127.0.0.1:52945 at 2015-04-16 19:21:01.654604032+00:00
I0416 12:21:01.654599 365723648 process.cpp:2081] Spawned process (4)@127.0.0.1:52945
I0416 12:21:01.654757 368943104 process.cpp:2091] Resuming (4)@127.0.0.1:52945 at 2015-04-16 19:21:01.654782976+00:00
I0416 12:21:01.654768 367869952 process.cpp:2091] Resuming log-recover-protocol(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.654791168+00:00
I0416 12:21:01.654835 367869952 recover.cpp:154] Broadcast request completed
I0416 12:21:01.654892 368406528 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.654915072+00:00
I0416 12:21:01.654942 368406528 replica.cpp:641] Replica in STARTING status received a broadcasted recover request
I0416 12:21:01.655048 367333376 process.cpp:2091] Resuming (4)@127.0.0.1:52945 at 2015-04-16 19:21:01.655097088+00:00
I0416 12:21:01.655110 366796800 process.cpp:2091] Resuming (5)@127.0.0.1:52945 at 2015-04-16 19:21:01.655124992+00:00
I0416 12:21:01.655139 366796800 process.cpp:2091] Resuming log-recover-protocol(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.655148032+00:00
I0416 12:21:01.655169 2105078528 process.cpp:2081] Spawned process (5)@127.0.0.1:52945
I0416 12:21:01.655215 366796800 recover.cpp:195] Received a recover response from a replica in STARTING status
I0416 12:21:01.655246 367333376 process.cpp:2198] Cleaning up (4)@127.0.0.1:52945
I0416 12:21:01.655344 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.655368192+00:00
I0416 12:21:01.655405 365723648 process.cpp:2091] Resuming __latch__(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.655430144+00:00
I0416 12:21:01.655441 2105078528 process.cpp:2081] Spawned process (6)@127.0.0.1:52945
I0416 12:21:01.655452 365723648 process.cpp:2198] Cleaning up __latch__(2)@127.0.0.1:52945
I0416 12:21:01.655447 367333376 process.cpp:2091] Resuming (6)@127.0.0.1:52945 at 2015-04-16 19:21:01.655467008+00:00
I0416 12:21:01.655568 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.655589120+00:00
I0416 12:21:01.655596 367333376 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.655617024+00:00
I0416 12:21:01.655603 367869952 process.cpp:2091] Resuming __fetcher__@127.0.0.1:52945 at 2015-04-16 19:21:01.655620096+00:00
I0416 12:21:01.655654 2105078528 process.cpp:2081] Spawned process __fetcher__@127.0.0.1:52945
I0416 12:21:01.655663 367333376 recover.cpp:566] Updating replica status to VOTING
I0416 12:21:01.655693 366796800 process.cpp:2198] Cleaning up log-recover-protocol(2)@127.0.0.1:52945
I0416 12:21:01.655778 365723648 process.cpp:2091] Resuming log-replica(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.655799040+00:00
I0416 12:21:01.655797 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.655819776+00:00
I0416 12:21:01.655854 2105078528 containerizer.cpp:113] Using isolation: posix/cpu,posix/mem
I0416 12:21:01.656002 2105078528 process.cpp:2081] Spawned process (7)@127.0.0.1:52945
I0416 12:21:01.655997 365723648 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 193us
I0416 12:21:01.656008 368406528 process.cpp:2091] Resuming (7)@127.0.0.1:52945 at 2015-04-16 19:21:01.656030976+00:00
I0416 12:21:01.656043 365723648 replica.cpp:323] Persisted replica status to VOTING
I0416 12:21:01.656143 2105078528 process.cpp:2081] Spawned process (8)@127.0.0.1:52945
I0416 12:21:01.656160 368943104 process.cpp:2091] Resuming (8)@127.0.0.1:52945 at 2015-04-16 19:21:01.656179968+00:00
I0416 12:21:01.656152 368406528 process.cpp:2091] Resuming log-recover(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.656173056+00:00
I0416 12:21:01.656216 368406528 recover.cpp:580] Successfully joined the Paxos group
I0416 12:21:01.656354 368406528 recover.cpp:464] Recover process terminated
I0416 12:21:01.656347 365723648 process.cpp:2091] Resuming log(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.656369920+00:00
I0416 12:21:01.656373 368406528 process.cpp:2198] Cleaning up log-recover(1)@127.0.0.1:52945
I0416 12:21:01.656404 365723648 log.cpp:358] Log recovery completed
I0416 12:21:01.656476 367869952 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.656493824+00:00
I0416 12:21:01.656512 369479680 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.656531200+00:00
I0416 12:21:01.656528 367869952 process.cpp:2091] Resuming log-reader(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.656544000+00:00
I0416 12:21:01.656625 366796800 process.cpp:2091] Resuming log-writer(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.656647936+00:00
I0416 12:21:01.656646 2105078528 process.cpp:2081] Spawned process (9)@127.0.0.1:52945
I0416 12:21:01.656658 367333376 process.cpp:2091] Resuming (9)@127.0.0.1:52945 at 2015-04-16 19:21:01.656680192+00:00
I0416 12:21:01.657052 368943104 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.657074944+00:00
I0416 12:21:01.657438 2105078528 process.cpp:2081] Spawned process __limiter__(2)@127.0.0.1:52945
I0416 12:21:01.657449 367869952 process.cpp:2091] Resuming __limiter__(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.657474048+00:00
I0416 12:21:01.657488 366796800 process.cpp:2091] Resuming monitor@127.0.0.1:52945 at 2015-04-16 19:21:01.657510144+00:00
I0416 12:21:01.657542 2105078528 process.cpp:2081] Spawned process monitor@127.0.0.1:52945
I0416 12:21:01.657567 365723648 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.657587968+00:00
I0416 12:21:01.657670 2105078528 process.cpp:2081] Spawned process slave(1)@127.0.0.1:52945
I0416 12:21:01.657703 367333376 process.cpp:2091] Resuming slave(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.657718784+00:00
I0416 12:21:01.657738 367333376 slave.cpp:176] Slave started on 1)@127.0.0.1:52945
I0416 12:21:01.657760 367333376 slave.cpp:177] Flags at startup: --authenticatee="crammd5" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_remove_delay="6hrs" --docker_sandbox_directory="/mnt/mesos/sandbox" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/Users/osxbuildbot/buildslave/slave/dev_test_osx/dev_distcheck_build_dist/mesos-0.23.0/_build/src" --logbufsecs="0" --logging_level="INFO" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --strict="true" --switch_user="true" --version="false" --work_dir="/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/0"
W0416 12:21:01.657997 367333376 slave.cpp:180]
**************************************************
Slave bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
I0416 12:21:01.658305 367333376 slave.cpp:316] Slave resources: cpus(*):2; mem(*):10240; disk(*):233432; ports(*):[31000-32000]
I0416 12:21:01.658526 2105078528 process.cpp:2081] Spawned process (10)@127.0.0.1:52945
I0416 12:21:01.658538 366796800 process.cpp:2091] Resuming (10)@127.0.0.1:52945 at 2015-04-16 19:21:01.658558976+00:00
I0416 12:21:01.658727 2105078528 process.cpp:2081] Spawned process (11)@127.0.0.1:52945
I0416 12:21:01.658721 367333376 slave.cpp:346] Slave hostname: localhost
I0416 12:21:01.658736 368943104 process.cpp:2091] Resuming (11)@127.0.0.1:52945 at 2015-04-16 19:21:01.658759168+00:00
I0416 12:21:01.658768 367333376 slave.cpp:351] Slave checkpoint: true
I0416 12:21:01.658927 2105078528 containerizer.cpp:113] Using isolation: posix/cpu,posix/mem
I0416 12:21:01.658936 367869952 process.cpp:2091] Resuming (6)@127.0.0.1:52945 at 2015-04-16 19:21:01.658972928+00:00
I0416 12:21:01.659078 2105078528 process.cpp:2081] Spawned process (12)@127.0.0.1:52945
I0416 12:21:01.659090 365723648 process.cpp:2091] Resuming (12)@127.0.0.1:52945 at 2015-04-16 19:21:01.659112960+00:00
I0416 12:21:01.659210 2105078528 process.cpp:2081] Spawned process (13)@127.0.0.1:52945
I0416 12:21:01.659231 366796800 process.cpp:2091] Resuming (13)@127.0.0.1:52945 at 2015-04-16 19:21:01.659246848+00:00
I0416 12:21:01.659433 369479680 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.659454976+00:00
I0416 12:21:01.659456 365723648 process.cpp:2091] Resuming (14)@127.0.0.1:52945 at 2015-04-16 19:21:01.659475968+00:00
I0416 12:21:01.659478 2105078528 process.cpp:2081] Spawned process (14)@127.0.0.1:52945
I0416 12:21:01.659782 367869952 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.659796992+00:00
I0416 12:21:01.660182 2105078528 process.cpp:2081] Spawned process __limiter__(3)@127.0.0.1:52945
I0416 12:21:01.660195 365723648 process.cpp:2091] Resuming __limiter__(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.660218112+00:00
I0416 12:21:01.660320 2105078528 process.cpp:2081] Spawned process slave(2)@127.0.0.1:52945
I0416 12:21:01.660328 365723648 process.cpp:2091] Resuming slave(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.660350976+00:00
I0416 12:21:01.660370 365723648 slave.cpp:176] Slave started on 2)@127.0.0.1:52945
I0416 12:21:01.660398 365723648 slave.cpp:177] Flags at startup: --authenticatee="crammd5" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_remove_delay="6hrs" --docker_sandbox_directory="/mnt/mesos/sandbox" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/Users/osxbuildbot/buildslave/slave/dev_test_osx/dev_distcheck_build_dist/mesos-0.23.0/_build/src" --logbufsecs="0" --logging_level="INFO" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --strict="true" --switch_user="true" --version="false" --work_dir="/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/1"
W0416 12:21:01.660644 365723648 slave.cpp:180]
**************************************************
Slave bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
I0416 12:21:01.660853 365723648 slave.cpp:316] Slave resources: cpus(*):2; mem(*):10240; disk(*):233432; ports(*):[31000-32000]
I0416 12:21:01.661152 365723648 slave.cpp:346] Slave hostname: localhost
I0416 12:21:01.661175 365723648 slave.cpp:351] Slave checkpoint: true
I0416 12:21:01.661226 368406528 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.661250048+00:00
I0416 12:21:01.661245 368943104 process.cpp:2091] Resuming (11)@127.0.0.1:52945 at 2015-04-16 19:21:01.661270016+00:00
I0416 12:21:01.661363 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.661381888+00:00
I0416 12:21:01.661384 2105078528 process.cpp:2081] Spawned process (15)@127.0.0.1:52945
I0416 12:21:01.661396 367333376 process.cpp:2081] Spawned process __async_executor__(1)@127.0.0.1:52945
I0416 12:21:01.661409 367869952 process.cpp:2091] Resuming (15)@127.0.0.1:52945 at 2015-04-16 19:21:01.661426176+00:00
I0416 12:21:01.661424 368943104 process.cpp:2091] Resuming __async_executor__(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.661444864+00:00
I0416 12:21:01.661437 369479680 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.661457920+00:00
I0416 12:21:01.661505 368943104 state.cpp:35] Recovering state from '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/0/meta'
I0416 12:21:01.661540 365723648 process.cpp:2081] Spawned process __async_executor__(2)@127.0.0.1:52945
I0416 12:21:01.661530 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.661549056+00:00
I0416 12:21:01.661551 367869952 process.cpp:2091] Resuming __async_executor__(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.661587968+00:00
I0416 12:21:01.661619 2105078528 process.cpp:2081] Spawned process (16)@127.0.0.1:52945
I0416 12:21:01.661615 366796800 process.cpp:2091] Resuming (16)@127.0.0.1:52945 at 2015-04-16 19:21:01.661634816+00:00
I0416 12:21:01.661654 367869952 state.cpp:35] Recovering state from '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/1/meta'
I0416 12:21:01.661659 368943104 process.cpp:2198] Cleaning up __async_executor__(1)@127.0.0.1:52945
I0416 12:21:01.661775 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.661796096+00:00
I0416 12:21:01.661835 2105078528 containerizer.cpp:113] Using isolation: posix/cpu,posix/mem
I0416 12:21:01.661828 365723648 process.cpp:2091] Resuming slave(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.661849856+00:00
I0416 12:21:01.661828 368406528 process.cpp:2091] Resuming (6)@127.0.0.1:52945 at 2015-04-16 19:21:01.661850880+00:00
I0416 12:21:01.661852 367869952 process.cpp:2198] Cleaning up __async_executor__(2)@127.0.0.1:52945
I0416 12:21:01.661906 2105078528 process.cpp:2081] Spawned process (17)@127.0.0.1:52945
I0416 12:21:01.661903 368406528 status_update_manager.cpp:197] Recovering status update manager
I0416 12:21:01.661912 367333376 process.cpp:2091] Resuming (17)@127.0.0.1:52945 at 2015-04-16 19:21:01.661935104+00:00
I0416 12:21:01.661911 366796800 process.cpp:2091] Resuming (11)@127.0.0.1:52945 at 2015-04-16 19:21:01.661954048+00:00
I0416 12:21:01.661938 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.661967872+00:00
I0416 12:21:01.661979 366796800 status_update_manager.cpp:197] Recovering status update manager
I0416 12:21:01.661993 2105078528 process.cpp:2081] Spawned process (18)@127.0.0.1:52945
I0416 12:21:01.662013 365723648 process.cpp:2091] Resuming (18)@127.0.0.1:52945 at 2015-04-16 19:21:01.662036992+00:00
I0416 12:21:01.662022 367869952 process.cpp:2091] Resuming slave(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.662044928+00:00
I0416 12:21:01.662078 368406528 process.cpp:2091] Resuming slave(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.662102016+00:00
I0416 12:21:01.662204 369479680 process.cpp:2091] Resuming (9)@127.0.0.1:52945 at 2015-04-16 19:21:01.662226944+00:00
I0416 12:21:01.662245 366796800 process.cpp:2091] Resuming (14)@127.0.0.1:52945 at 2015-04-16 19:21:01.662259200+00:00
I0416 12:21:01.662256 369479680 containerizer.cpp:310] Recovering containerizer
I0416 12:21:01.662277 366796800 containerizer.cpp:310] Recovering containerizer
I0416 12:21:01.662314 367869952 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.662337024+00:00
I0416 12:21:01.662366 2105078528 process.cpp:2081] Spawned process (19)@127.0.0.1:52945
I0416 12:21:01.662379 368406528 process.cpp:2091] Resuming (19)@127.0.0.1:52945 at 2015-04-16 19:21:01.662401024+00:00
I0416 12:21:01.662529 365723648 process.cpp:2091] Resuming (12)@127.0.0.1:52945 at 2015-04-16 19:21:01.662550016+00:00
I0416 12:21:01.662535 367869952 process.cpp:2091] Resuming (7)@127.0.0.1:52945 at 2015-04-16 19:21:01.662552064+00:00
I0416 12:21:01.662574 368943104 process.cpp:2091] Resuming (13)@127.0.0.1:52945 at 2015-04-16 19:21:01.662606080+00:00
I0416 12:21:01.662621 367869952 process.cpp:2091] Resuming (8)@127.0.0.1:52945 at 2015-04-16 19:21:01.662644992+00:00
I0416 12:21:01.662680 368406528 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.662700800+00:00
I0416 12:21:01.662730 368943104 process.cpp:2091] Resuming (20)@127.0.0.1:52945 at 2015-04-16 19:21:01.662752000+00:00
I0416 12:21:01.662750 367333376 process.cpp:2091] Resuming (21)@127.0.0.1:52945 at 2015-04-16 19:21:01.662771968+00:00
I0416 12:21:01.662768 366796800 process.cpp:2081] Spawned process (20)@127.0.0.1:52945
I0416 12:21:01.662792 369479680 process.cpp:2081] Spawned process (21)@127.0.0.1:52945
I0416 12:21:01.662801 367869952 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.662818048+00:00
I0416 12:21:01.662976 365723648 process.cpp:2091] Resuming (9)@127.0.0.1:52945 at 2015-04-16 19:21:01.662996992+00:00
I0416 12:21:01.663003 367333376 process.cpp:2198] Cleaning up (21)@127.0.0.1:52945
I0416 12:21:01.662993 368406528 process.cpp:2091] Resuming (14)@127.0.0.1:52945 at 2015-04-16 19:21:01.663014912+00:00
I0416 12:21:01.663022 368943104 process.cpp:2198] Cleaning up (20)@127.0.0.1:52945
I0416 12:21:01.663079 366796800 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.663098112+00:00
I0416 12:21:01.663130 366796800 process.cpp:2091] Resuming slave(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.663148032+00:00
I0416 12:21:01.663162 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.663183872+00:00
I0416 12:21:01.663200 365723648 process.cpp:2091] Resuming slave(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.663222016+00:00
I0416 12:21:01.663219 367333376 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.663239936+00:00
I0416 12:21:01.663233 366796800 slave.cpp:3806] Finished recovery
I0416 12:21:01.663290 365723648 slave.cpp:3806] Finished recovery
I0416 12:21:01.663611 368943104 process.cpp:2091] Resuming metrics@127.0.0.1:52945 at 2015-04-16 19:21:01.663631872+00:00
I0416 12:21:01.663646 2105078528 process.cpp:2081] Spawned process __limiter__(4)@127.0.0.1:52945
I0416 12:21:01.663648 369479680 process.cpp:2091] Resuming __limiter__(4)@127.0.0.1:52945 at 2015-04-16 19:21:01.663665920+00:00
I0416 12:21:01.663733 2105078528 process.cpp:2081] Spawned process slave(3)@127.0.0.1:52945
I0416 12:21:01.663791 367869952 process.cpp:2091] Resuming slave(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.663815936+00:00
I0416 12:21:01.663836 367869952 slave.cpp:176] Slave started on 3)@127.0.0.1:52945
I0416 12:21:01.663892 2105078528 pid.cpp:82] Attempting to parse 'master@127.0.0.1:52945' into a PID
F0416 12:21:01.664001 365723648 slave.cpp:3816] CHECK_SOME(state::checkpoint(path, bootId.get())): Failed to rename '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/0/meta/EntzLz' to '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/1/meta/boot_id': No such file or directory
*** Check failure stack trace: ***
I0416 12:21:01.663998 368406528 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.664019968+00:00
I0416 12:21:01.664113 367333376 process.cpp:2091] Resuming slave(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.664136960+00:00
I0416 12:21:01.664203 367333376 pid.cpp:82] Attempting to parse 'master@127.0.0.1:52945' into a PID
I0416 12:21:01.664191 369479680 process.cpp:2091] Resuming (6)@127.0.0.1:52945 at 2015-04-16 19:21:01.664210944+00:00
I0416 12:21:01.663877 367869952 slave.cpp:177] Flags at startup: --authenticatee="crammd5" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_remove_delay="6hrs" --docker_sandbox_directory="/mnt/mesos/sandbox" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/Users/osxbuildbot/buildslave/slave/dev_test_osx/dev_distcheck_build_dist/mesos-0.23.0/_build/src" --logbufsecs="0" --logging_level="INFO" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --strict="true" --switch_user="true" --version="false" --work_dir="/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/2"
I0416 12:21:01.664229 369479680 status_update_manager.cpp:171] Pausing sending status updates
W0416 12:21:01.664244 367869952 slave.cpp:180]
**************************************************
Slave bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address.
**************************************************
I0416 12:21:01.664242 367333376 slave.cpp:627] New master detected at master@127.0.0.1:52945
I0416 12:21:01.664312 367333376 slave.cpp:652] No credentials provided. Attempting to register without authentication
I0416 12:21:01.664378 367333376 slave.cpp:663] Detecting new master
I0416 12:21:01.664428 366796800 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.664446208+00:00
I0416 12:21:01.664546 367869952 slave.cpp:316] Slave resources: cpus(*):2; mem(*):10240; disk(*):233432; ports(*):[31000-32000]
I0416 12:21:01.664926 367869952 slave.cpp:346] Slave hostname: localhost
I0416 12:21:01.664950 367869952 slave.cpp:351] Slave checkpoint: true
I0416 12:21:01.665007 368406528 process.cpp:2091] Resuming (16)@127.0.0.1:52945 at 2015-04-16 19:21:01.665029888+00:00
I0416 12:21:01.665225 369479680 process.cpp:2091] Resuming help@127.0.0.1:52945 at 2015-04-16 19:21:01.665245184+00:00
I0416 12:21:01.665330 367333376 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.665344000+00:00
I0416 12:21:01.665340 366796800 process.cpp:2091] Resuming __async_executor__(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.665360896+00:00
I0416 12:21:01.665375 367869952 process.cpp:2081] Spawned process __async_executor__(3)@127.0.0.1:52945
I0416 12:21:01.665475 367333376 process.cpp:2091] Resuming __async_executor__(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.665491968+00:00
I0416 12:21:01.665511 367333376 state.cpp:35] Recovering state from '/var/folders/lx/kp985px11jg_0mpdwnd3ty8r0000gn/T/mesos-XXXXXX.RtaNR5cQ/2/meta'
I0416 12:21:01.665587 367333376 process.cpp:2198] Cleaning up __async_executor__(3)@127.0.0.1:52945
I0416 12:21:01.665627 369479680 process.cpp:2091] Resuming (16)@127.0.0.1:52945 at 2015-04-16 19:21:01.665649920+00:00
I0416 12:21:01.665645 368943104 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.665666048+00:00
I0416 12:21:01.665680 369479680 status_update_manager.cpp:197] Recovering status update manager
I0416 12:21:01.665772 368406528 process.cpp:2091] Resuming slave(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.665793792+00:00
I0416 12:21:01.665886 367333376 process.cpp:2091] Resuming (19)@127.0.0.1:52945 at 2015-04-16 19:21:01.665904896+00:00
I0416 12:21:01.665922 367333376 containerizer.cpp:310] Recovering containerizer
I0416 12:21:01.666064 369479680 process.cpp:2091] Resuming (18)@127.0.0.1:52945 at 2015-04-16 19:21:01.666081792+00:00
I0416 12:21:01.666057 368943104 process.cpp:2091] Resuming (17)@127.0.0.1:52945 at 2015-04-16 19:21:01.666080000+00:00
I0416 12:21:01.666128 369479680 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.666151168+00:00
I0416 12:21:01.666173 367333376 process.cpp:2081] Spawned process (22)@127.0.0.1:52945
I0416 12:21:01.666196 368943104 process.cpp:2091] Resuming (22)@127.0.0.1:52945 at 2015-04-16 19:21:01.666230016+00:00
I0416 12:21:01.666328 366796800 process.cpp:2091] Resuming (19)@127.0.0.1:52945 at 2015-04-16 19:21:01.666349824+00:00
I0416 12:21:01.666349 368943104 process.cpp:2198] Cleaning up (22)@127.0.0.1:52945
I0416 12:21:01.666451 367869952 process.cpp:2091] Resuming __gc__@127.0.0.1:52945 at 2015-04-16 19:21:01.666472960+00:00
I0416 12:21:01.666476 368406528 process.cpp:2091] Resuming slave(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.666493952+00:00
I0416 12:21:01.666559 368406528 slave.cpp:3806] Finished recovery
I0416 12:21:01.668668 368943104 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.668697088+00:00
I0416 12:21:01.668804 367869952 process.cpp:2091] Resuming slave(3)@127.0.0.1:52945 at 2015-04-16 19:21:01.668827904+00:00
I0416 12:21:01.668913 367869952 pid.cpp:82] Attempting to parse 'master@127.0.0.1:52945' into a PID
I0416 12:21:01.668900 368406528 process.cpp:2091] Resuming (16)@127.0.0.1:52945 at 2015-04-16 19:21:01.668921856+00:00
I0416 12:21:01.668959 368406528 status_update_manager.cpp:171] Pausing sending status updates
I0416 12:21:01.668977 367869952 slave.cpp:627] New master detected at master@127.0.0.1:52945
I0416 12:21:01.669018 367869952 slave.cpp:652] No credentials provided. Attempting to register without authentication
I0416 12:21:01.669067 367869952 slave.cpp:663] Detecting new master
I0416 12:21:01.669113 368943104 process.cpp:2091] Resuming standalone-master-detector(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.669146112+00:00
I0416 12:21:01.672554 2105078528 process.cpp:2081] Spawned process standalone-master-detector(2)@127.0.0.1:52945
I0416 12:21:01.672559 366796800 process.cpp:2091] Resuming standalone-master-detector(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.672573184+00:00
I0416 12:21:01.672622 2105078528 process.cpp:2081] Spawned process scheduler(1)@127.0.0.1:52945
I0416 12:21:01.672631 369479680 process.cpp:2091] Resuming scheduler(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.672656128+00:00
I0416 12:21:01.672837 368406528 process.cpp:2091] Resuming standalone-master-detector(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.672858880+00:00
I0416 12:21:01.672935 369479680 process.cpp:2091] Resuming scheduler(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.672957952+00:00
I0416 12:21:01.672996 369479680 pid.cpp:82] Attempting to parse 'master@127.0.0.1:52945' into a PID
I0416 12:21:01.673032 369479680 scheduler.cpp:420] New master detected at master@127.0.0.1:52945
I0416 12:21:01.673056 369479680 scheduler.cpp:469] Authenticating with master master@127.0.0.1:52945
I0416 12:21:01.673131 369479680 process.cpp:2081] Spawned process crammd5_authenticatee(1)@127.0.0.1:52945
I0416 12:21:01.673141 367333376 process.cpp:2091] Resuming crammd5_authenticatee(1)@127.0.0.1:52945 at 2015-04-16 19:21:01.673160192+00:00
I0416 12:21:01.673285 366796800 process.cpp:2091] Resuming standalone-master-detector(2)@127.0.0.1:52945 at 2015-04-16 19:21:01.673306880+00:00
I0416 12:21:01.673476 367333376 authenticatee.hpp:115] Initializing client SASL
@ 0x110e9284a google::LogMessage::Fail()
@ 0x110e917dd google::LogMessage::SendToLog()
@ 0x110e924ea google::LogMessage::Flush()
@ 0x110e99348 google::LogMessageFatal::~LogMessageFatal()
I0416 12:21:01.747539 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:33.747597056+00:00
@ 0x110e92ca5 google::LogMessageFatal::~LogMessageFatal()
@ 0x10f3d33d3 _CheckFatal::~_CheckFatal()
@ 0x10f3d3025 _CheckFatal::~_CheckFatal()
@ 0x10fd94da6 mesos::internal::slave::Slave::__recover()
@ 0x10fe7f09d _ZZN7process8dispatchIN5mesos8internal5slave5SlaveERKNS_6FutureI7NothingEES7_EEvRKNS_3PIDIT_EEMSB_FvT0_ET1_ENKUlPNS_11ProcessBaseEE_clESK_
@ 0x10fe7ee7f _ZNSt3__110__function6__funcIZN7process8dispatchIN5mesos8internal5slave5SlaveERKNS2_6FutureI7NothingEESA_EEvRKNS2_3PIDIT_EEMSE_FvT0_ET1_EUlPNS2_11ProcessBaseEE_NS_9allocatorISO_EEFvSN_EEclEOSN_
@ 0x110d74e7b std::__1::function<>::operator()()
@ 0x110d5c5bf process::ProcessBase::visit()
@ 0x110de6c0e process::DispatchEvent::visit()
@ 0x10f3d0841 process::ProcessBase::serve()
@ 0x110d45abe process::ProcessManager::resume()
@ 0x110d451de process::schedule()
@ 0x7fff8f1eb268 _pthread_body
@ 0x7fff8f1eb1e5 _pthread_start
@ 0x7fff8f1e941d thread_start
I0416 12:21:01.848065 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:33.848114944+00:00
I0416 12:21:01.948398 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:33.948432896+00:00
I0416 12:21:02.048640 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.048694016+00:00
I0416 12:21:02.148989 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.149024000+00:00
I0416 12:21:02.252266 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.252296960+00:00
I0416 12:21:02.352577 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.352615168+00:00
I0416 12:21:02.452965 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.452996864+00:00
I0416 12:21:02.553390 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.553423872+00:00
I0416 12:21:02.653735 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.653764864+00:00
I0416 12:21:02.756922 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.756952064+00:00
I0416 12:21:02.857390 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.857434880+00:00
I0416 12:21:02.961252 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:34.961291008+00:00
I0416 12:21:03.061522 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.061554944+00:00
I0416 12:21:03.166790 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.166821120+00:00
I0416 12:21:03.267645 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.267698176+00:00
I0416 12:21:03.367985 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.368038912+00:00
I0416 12:21:03.468283 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.468316928+00:00
I0416 12:21:03.568531 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.568573952+00:00
I0416 12:21:03.668767 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.668807936+00:00
I0416 12:21:03.769104 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.769152000+00:00
I0416 12:21:03.869385 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.869432832+00:00
I0416 12:21:03.969768 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:35.969798144+00:00
I0416 12:21:04.074630 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.074662912+00:00
I0416 12:21:04.174993 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.175024128+00:00
I0416 12:21:04.275383 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.275419904+00:00
I0416 12:21:04.375689 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.375726848+00:00
I0416 12:21:04.476035 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.476072960+00:00
I0416 12:21:04.576369 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.576424960+00:00
I0416 12:21:04.676823 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.676861952+00:00
I0416 12:21:04.777240 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.777278976+00:00
I0416 12:21:04.877797 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.877843968+00:00
I0416 12:21:04.978090 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:36.978121984+00:00
I0416 12:21:05.079474 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.079504896+00:00
I0416 12:21:05.181157 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.181199104+00:00
I0416 12:21:05.281492 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.281528064+00:00
I0416 12:21:05.381813 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.381846016+00:00
I0416 12:21:05.482925 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.482966784+00:00
I0416 12:21:05.583298 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.583326208+00:00
I0416 12:21:05.683570 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.683599872+00:00
I0416 12:21:05.785339 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.785371136+00:00
I0416 12:21:05.885790 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.885846016+00:00
I0416 12:21:05.986150 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:37.986181120+00:00
I0416 12:21:06.086530 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.086565120+00:00
I0416 12:21:06.187046 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.187074816+00:00
I0416 12:21:06.287353 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.287381760+00:00
I0416 12:21:06.390130 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.390161920+00:00
I0416 12:21:06.490376 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.490406912+00:00
I0416 12:21:06.564841 143380480 process.cpp:2091] Resuming (1)@127.0.0.1:52923 at 2015-04-16 19:21:06.564875008+00:00
I0416 12:21:06.564909 143380480 exec.cpp:86] Committing suicide by killing the process group
I0416 12:21:06.566997 344678400 process.cpp:2091] Resuming (1)@127.0.0.1:52922 at 2015-04-16 19:21:06.567048192+00:00
I0416 12:21:06.567098 344678400 exec.cpp:86] Committing suicide by killing the process group
I0416 12:21:06.568764 136261632 process.cpp:2091] Resuming (1)@127.0.0.1:52921 at 2015-04-16 19:21:06.568818176+00:00
I0416 12:21:06.568856 136261632 exec.cpp:86] Committing suicide by killing the process group
I0416 12:21:06.590653 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.590689792+00:00
I0416 12:21:06.694011 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.694044160+00:00
I0416 12:21:06.794332 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.794361088+00:00
I0416 12:21:06.894798 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.894850048+00:00
I0416 12:21:06.995139 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:38.995166976+00:00
I0416 12:21:07.095384 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.095429888+00:00
I0416 12:21:07.195736 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.195770112+00:00
I0416 12:21:07.296052 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.296078848+00:00
I0416 12:21:07.396627 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.396659968+00:00
I0416 12:21:07.497102 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.497129984+00:00
I0416 12:21:07.597435 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.597464064+00:00
I0416 12:21:07.697954 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.697988864+00:00
I0416 12:21:07.798230 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.798292992+00:00
I0416 12:21:07.898574 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:39.898610176+00:00
I0416 12:21:08.003020 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.003076096+00:00
I0416 12:21:08.103488 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.103524864+00:00
I0416 12:21:08.203766 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.203793152+00:00
I0416 12:21:08.304029 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.304101888+00:00
I0416 12:21:08.355402 311099392 process.cpp:2048] Dropping event for process slave(1)@127.0.0.1:52842
I0416 12:21:08.404270 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.404307968+00:00
I0416 12:21:08.508119 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.508163072+00:00
I0416 12:21:08.520028 311099392 process.cpp:2048] Dropping event for process slave(2)@127.0.0.1:52842
I0416 12:21:08.608477 307343360 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.608509952+00:00
I0416 12:21:08.713762 308416512 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.713812224+00:00
I0416 12:21:08.814121 307879936 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.814168064+00:00
I0416 12:21:08.841398 311099392 process.cpp:2048] Dropping event for process slave(3)@127.0.0.1:52842
I0416 12:21:08.914386 310562816 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:40.914430976+00:00
I0416 12:21:09.019639 309489664 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:41.019673088+00:00
I0416 12:21:09.119954 308953088 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:41.120000000+00:00
I0416 12:21:09.224282 306806784 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:41.224335872+00:00
I0416 12:21:09.326063 310026240 process.cpp:2091] Resuming reaper(1)@127.0.0.1:52842 at 2015-04-16 19:21:41.326106112+00:00
../../src/tests/script.cpp:83: Failure
Failed
low_level_scheduler_pthread_test.sh terminated with signal Abort trap: 6
[ FAILED ] ExamplesTest.LowLevelSchedulerPthread (7774 ms)
{code}
> Segmentation Fault in ExamplesTest.LowLevelSchedulerPthread
> -----------------------------------------------------------
>
> Key: MESOS-2144
> URL: https://issues.apache.org/jira/browse/MESOS-2144
> Project: Mesos
> Issue Type: Bug
> Components: test
> Affects Versions: 0.21.0
> Reporter: Cody Maloney
> Assignee: Yan Xu
> Priority: Minor
> Labels: flaky, twitter
>
> Occured on review bot review of: https://reviews.apache.org/r/28262/#review62333
> The review doesn't touch code related to the test (And doesn't break libprocess in general)
> [ RUN ] ExamplesTest.LowLevelSchedulerPthread
> ../../src/tests/script.cpp:83: Failure
> Failed
> low_level_scheduler_pthread_test.sh terminated with signal Segmentation fault
> [ FAILED ] ExamplesTest.LowLevelSchedulerPthread (7561 ms)
> The test
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)