You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Benjamin Mahler (JIRA)" <ji...@apache.org> on 2015/11/18 16:19:11 UTC
[jira] [Updated] (MESOS-3907) PersistentVolumeTest.SlaveRecovery
test is flaky
[ https://issues.apache.org/jira/browse/MESOS-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Mahler updated MESOS-3907:
-----------------------------------
Description:
Looks like the executor didn't re-register in time after slave restart. The Clock::settle() should've guaranteed that the executor sent re-register and slave processed it, so it's not clear why this race happened.
{noformat: title=Good Run}
[ RUN ] PersistentVolumeTest.SlaveRecovery
I1112 17:53:29.746028 29676 leveldb.cpp:176] Opened db in 2.430372ms
I1112 17:53:29.746974 29676 leveldb.cpp:183] Compacted db in 832726ns
I1112 17:53:29.747094 29676 leveldb.cpp:198] Created db iterator in 24074ns
I1112 17:53:29.747210 29676 leveldb.cpp:204] Seeked to beginning of db in 2533ns
I1112 17:53:29.747329 29676 leveldb.cpp:273] Iterated through 0 keys in the db in 786ns
I1112 17:53:29.747462 29676 replica.cpp:780] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
I1112 17:53:29.748114 29699 recover.cpp:449] Starting replica recovery
I1112 17:53:29.748440 29696 recover.cpp:475] Replica is in EMPTY status
I1112 17:53:29.749609 29708 replica.cpp:676] Replica in EMPTY status received a broadcasted recover request from (5641)@172.17.6.242:48157
I1112 17:53:29.750352 29702 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1112 17:53:29.750880 29707 recover.cpp:566] Updating replica status to STARTING
I1112 17:53:29.751597 29705 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 418265ns
I1112 17:53:29.751699 29705 replica.cpp:323] Persisted replica status to STARTING
I1112 17:53:29.752048 29710 recover.cpp:475] Replica is in STARTING status
I1112 17:53:29.752780 29710 replica.cpp:676] Replica in STARTING status received a broadcasted recover request from (5642)@172.17.6.242:48157
I1112 17:53:29.753223 29705 recover.cpp:195] Received a recover response from a replica in STARTING status
I1112 17:53:29.753618 29702 recover.cpp:566] Updating replica status to VOTING
I1112 17:53:29.754117 29696 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 357564ns
I1112 17:53:29.754144 29696 replica.cpp:323] Persisted replica status to VOTING
I1112 17:53:29.754236 29696 recover.cpp:580] Successfully joined the Paxos group
I1112 17:53:29.754380 29696 recover.cpp:464] Recover process terminated
I1112 17:53:29.755972 29705 master.cpp:367] Master 4104c1dc-cb09-41a0-8f89-339ad511ce2a (bf058f59fe00) started on 172.17.6.242:48157
I1112 17:53:29.755993 29705 master.cpp:369] Flags at startup: --acls="register_frameworks {
principals {
values: "test-principal"
}
roles {
values: "role1"
}
}
" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/JRCPBA/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --roles="role1" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.26.0/_inst/share/mesos/webui" --work_dir="/tmp/JRCPBA/master" --zk_session_timeout="10secs"
I1112 17:53:29.756337 29705 master.cpp:414] Master only allowing authenticated frameworks to register
I1112 17:53:29.756350 29705 master.cpp:419] Master only allowing authenticated slaves to register
I1112 17:53:29.756358 29705 credentials.hpp:37] Loading credentials for authentication from '/tmp/JRCPBA/credentials'
I1112 17:53:29.756618 29705 master.cpp:458] Using default 'crammd5' authenticator
I1112 17:53:29.756753 29705 master.cpp:495] Authorization enabled
I1112 17:53:29.756985 29706 whitelist_watcher.cpp:79] No whitelist given
I1112 17:53:29.757154 29698 hierarchical.cpp:151] Initialized hierarchical allocator process
I1112 17:53:29.759389 29700 master.cpp:1606] The newly elected leader is master@172.17.6.242:48157 with id 4104c1dc-cb09-41a0-8f89-339ad511ce2a
I1112 17:53:29.759423 29700 master.cpp:1619] Elected as the leading master!
I1112 17:53:29.759443 29700 master.cpp:1379] Recovering from registrar
I1112 17:53:29.759692 29704 registrar.cpp:309] Recovering registrar
I1112 17:53:29.760234 29699 log.cpp:661] Attempting to start the writer
I1112 17:53:29.761632 29700 replica.cpp:496] Replica received implicit promise request from (5643)@172.17.6.242:48157 with proposal 1
I1112 17:53:29.762080 29700 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 412844ns
I1112 17:53:29.762109 29700 replica.cpp:345] Persisted promised to 1
I1112 17:53:29.762683 29700 coordinator.cpp:240] Coordinator attempting to fill missing positions
I1112 17:53:29.763840 29697 replica.cpp:391] Replica received explicit promise request from (5644)@172.17.6.242:48157 for position 0 with proposal 2
I1112 17:53:29.764291 29697 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 407157ns
I1112 17:53:29.764317 29697 replica.cpp:715] Persisted action at 0
I1112 17:53:29.765326 29697 replica.cpp:540] Replica received write request for position 0 from (5645)@172.17.6.242:48157
I1112 17:53:29.765386 29697 leveldb.cpp:438] Reading position from leveldb took 28779ns
I1112 17:53:29.765704 29697 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 273113ns
I1112 17:53:29.765730 29697 replica.cpp:715] Persisted action at 0
I1112 17:53:29.766624 29699 replica.cpp:694] Replica received learned notice for position 0 from @0.0.0.0:0
I1112 17:53:29.766934 29699 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 268658ns
I1112 17:53:29.766963 29699 replica.cpp:715] Persisted action at 0
I1112 17:53:29.766983 29699 replica.cpp:700] Replica learned NOP action at position 0
I1112 17:53:29.767499 29699 log.cpp:677] Writer started with ending position 0
I1112 17:53:29.768618 29703 leveldb.cpp:438] Reading position from leveldb took 25919ns
I1112 17:53:29.769791 29703 registrar.cpp:342] Successfully fetched the registry (0B) in 10.054144ms
I1112 17:53:29.769908 29703 registrar.cpp:441] Applied 1 operations in 29537ns; attempting to update the 'registry'
I1112 17:53:29.771086 29699 log.cpp:685] Attempting to append 176 bytes to the log
I1112 17:53:29.771188 29710 coordinator.cpp:350] Coordinator attempting to write APPEND action at position 1
I1112 17:53:29.772028 29700 replica.cpp:540] Replica received write request for position 1 from (5646)@172.17.6.242:48157
I1112 17:53:29.772368 29700 leveldb.cpp:343] Persisting action (195 bytes) to leveldb took 303124ns
I1112 17:53:29.772455 29700 replica.cpp:715] Persisted action at 1
I1112 17:53:29.773016 29698 replica.cpp:694] Replica received learned notice for position 1 from @0.0.0.0:0
I1112 17:53:29.773453 29698 leveldb.cpp:343] Persisting action (197 bytes) to leveldb took 405803ns
I1112 17:53:29.773538 29698 replica.cpp:715] Persisted action at 1
I1112 17:53:29.773633 29698 replica.cpp:700] Replica learned APPEND action at position 1
I1112 17:53:29.774744 29702 registrar.cpp:486] Successfully updated the 'registry' in 4.68096ms
I1112 17:53:29.774878 29702 registrar.cpp:372] Successfully recovered registrar
I1112 17:53:29.775008 29708 log.cpp:704] Attempting to truncate the log to 1
I1112 17:53:29.775240 29705 master.cpp:1416] Recovered 0 slaves from the Registry (137B) ; allowing 10mins for slaves to re-register
I1112 17:53:29.775346 29702 coordinator.cpp:350] Coordinator attempting to write TRUNCATE action at position 2
I1112 17:53:29.776243 29695 replica.cpp:540] Replica received write request for position 2 from (5647)@172.17.6.242:48157
I1112 17:53:29.776782 29695 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 458517ns
I1112 17:53:29.776813 29695 replica.cpp:715] Persisted action at 2
I1112 17:53:29.777896 29699 replica.cpp:694] Replica received learned notice for position 2 from @0.0.0.0:0
I1112 17:53:29.778386 29699 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 321333ns
I1112 17:53:29.778445 29699 leveldb.cpp:401] Deleting ~1 keys from leveldb took 29781ns
I1112 17:53:29.778466 29699 replica.cpp:715] Persisted action at 2
I1112 17:53:29.778493 29699 replica.cpp:700] Replica learned TRUNCATE action at position 2
I1112 17:53:29.789324 29676 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix
W1112 17:53:29.789729 29676 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges
I1112 17:53:29.802047 29705 slave.cpp:191] Slave started on 196)@172.17.6.242:48157
I1112 17:53:29.802078 29705 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):1024" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK"
I1112 17:53:29.802577 29705 credentials.hpp:85] Loading credential for authentication from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/credential'
I1112 17:53:29.802806 29705 slave.cpp:322] Slave using credential for: test-principal
I1112 17:53:29.803010 29705 resources.cpp:474] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):1024
Trying semicolon-delimited string format instead
I1112 17:53:29.803546 29705 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
I1112 17:53:29.803680 29705 slave.cpp:400] Slave attributes: [ ]
I1112 17:53:29.803779 29705 slave.cpp:405] Slave hostname: bf058f59fe00
I1112 17:53:29.803850 29705 slave.cpp:410] Slave checkpoint: true
I1112 17:53:29.805106 29699 state.cpp:54] Recovering state from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta'
I1112 17:53:29.805594 29699 status_update_manager.cpp:202] Recovering status update manager
I1112 17:53:29.805768 29699 containerizer.cpp:385] Recovering containerizer
I1112 17:53:29.806855 29676 sched.cpp:166] Version: 0.26.0
I1112 17:53:29.807466 29698 slave.cpp:4230] Finished recovery
I1112 17:53:29.808089 29698 slave.cpp:4387] Querying resource estimator for oversubscribable resources
I1112 17:53:29.808365 29697 sched.cpp:264] New master detected at master@172.17.6.242:48157
I1112 17:53:29.808424 29698 slave.cpp:729] New master detected at master@172.17.6.242:48157
I1112 17:53:29.808478 29697 sched.cpp:320] Authenticating with master master@172.17.6.242:48157
I1112 17:53:29.808497 29697 sched.cpp:327] Using default CRAM-MD5 authenticatee
I1112 17:53:29.808511 29703 status_update_manager.cpp:176] Pausing sending status updates
I1112 17:53:29.808593 29698 slave.cpp:792] Authenticating with master master@172.17.6.242:48157
I1112 17:53:29.808616 29698 slave.cpp:797] Using default CRAM-MD5 authenticatee
I1112 17:53:29.808734 29698 slave.cpp:765] Detecting new master
I1112 17:53:29.808794 29704 authenticatee.cpp:123] Creating new client SASL connection
I1112 17:53:29.809075 29695 master.cpp:5150] Authenticating scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
I1112 17:53:29.809202 29707 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(467)@172.17.6.242:48157
I1112 17:53:29.809305 29697 authenticatee.cpp:123] Creating new client SASL connection
I1112 17:53:29.809438 29706 authenticator.cpp:100] Creating new server SASL connection
I1112 17:53:29.809612 29697 master.cpp:5150] Authenticating slave(196)@172.17.6.242:48157
I1112 17:53:29.809723 29695 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(468)@172.17.6.242:48157
I1112 17:53:29.809921 29706 authenticator.cpp:100] Creating new server SASL connection
I1112 17:53:29.810148 29706 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
I1112 17:53:29.810184 29706 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
I1112 17:53:29.810276 29706 authenticator.cpp:205] Received SASL authentication start
I1112 17:53:29.810340 29706 authenticator.cpp:327] Authentication requires more steps
I1112 17:53:29.810426 29706 authenticatee.cpp:260] Received SASL authentication step
I1112 17:53:29.810525 29706 authenticator.cpp:233] Received SASL authentication step
I1112 17:53:29.810556 29706 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1112 17:53:29.810570 29706 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I1112 17:53:29.810621 29706 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1112 17:53:29.810654 29706 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1112 17:53:29.810667 29706 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1112 17:53:29.810675 29706 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1112 17:53:29.810693 29706 authenticator.cpp:319] Authentication success
I1112 17:53:29.810796 29695 authenticatee.cpp:300] Authentication success
I1112 17:53:29.810864 29706 master.cpp:5180] Successfully authenticated principal 'test-principal' at slave(196)@172.17.6.242:48157
I1112 17:53:29.810920 29695 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(468)@172.17.6.242:48157
I1112 17:53:29.809806 29709 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
I1112 17:53:29.810995 29709 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
I1112 17:53:29.811110 29701 authenticator.cpp:205] Received SASL authentication start
I1112 17:53:29.811131 29698 slave.cpp:4401] Received oversubscribable resources from the resource estimator
I1112 17:53:29.811182 29701 authenticator.cpp:327] Authentication requires more steps
I1112 17:53:29.811270 29701 authenticatee.cpp:260] Received SASL authentication step
I1112 17:53:29.811373 29698 slave.cpp:860] Successfully authenticated with master master@172.17.6.242:48157
I1112 17:53:29.811406 29701 authenticator.cpp:233] Received SASL authentication step
I1112 17:53:29.811435 29701 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1112 17:53:29.811449 29701 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I1112 17:53:29.811491 29701 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1112 17:53:29.811522 29701 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1112 17:53:29.811514 29698 slave.cpp:1254] Will retry registration in 4.148387ms if necessary
I1112 17:53:29.811537 29701 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1112 17:53:29.811548 29701 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1112 17:53:29.811566 29701 authenticator.cpp:319] Authentication success
I1112 17:53:29.811722 29701 authenticatee.cpp:300] Authentication success
I1112 17:53:29.811812 29701 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(467)@172.17.6.242:48157
I1112 17:53:29.811847 29698 master.cpp:3859] Registering slave at slave(196)@172.17.6.242:48157 (bf058f59fe00) with id 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
I1112 17:53:29.812162 29698 master.cpp:5180] Successfully authenticated principal 'test-principal' at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
I1112 17:53:29.812445 29706 registrar.cpp:441] Applied 1 operations in 89470ns; attempting to update the 'registry'
I1112 17:53:29.813341 29710 sched.cpp:409] Successfully authenticated with master master@172.17.6.242:48157
I1112 17:53:29.813371 29710 sched.cpp:716] Sending SUBSCRIBE call to master@172.17.6.242:48157
I1112 17:53:29.813485 29710 sched.cpp:749] Will retry registration in 1.306688873secs if necessary
I1112 17:53:29.813665 29710 master.cpp:2176] Received SUBSCRIBE call for framework 'default' at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
I1112 17:53:29.813711 29698 log.cpp:685] Attempting to append 349 bytes to the log
I1112 17:53:29.813731 29710 master.cpp:1645] Authorizing framework principal 'test-principal' to receive offers for role 'role1'
I1112 17:53:29.813869 29710 coordinator.cpp:350] Coordinator attempting to write APPEND action at position 3
I1112 17:53:29.814548 29707 master.cpp:2247] Subscribing framework default with checkpointing enabled and capabilities [ ]
I1112 17:53:29.814918 29708 replica.cpp:540] Replica received write request for position 3 from (5659)@172.17.6.242:48157
I1112 17:53:29.814998 29703 hierarchical.cpp:195] Added framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:29.815057 29703 hierarchical.cpp:977] No resources available to allocate!
I1112 17:53:29.815091 29703 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:53:29.815115 29703 hierarchical.cpp:876] Performed allocation for 0 slaves in 96149ns
I1112 17:53:29.815232 29703 sched.cpp:643] Framework registered with 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:29.815317 29703 sched.cpp:657] Scheduler::registered took 60505ns
I1112 17:53:29.815325 29708 leveldb.cpp:343] Persisting action (368 bytes) to leveldb took 351340ns
I1112 17:53:29.815349 29708 replica.cpp:715] Persisted action at 3
I1112 17:53:29.817284 29708 slave.cpp:1254] Will retry registration in 17.295409ms if necessary
I1112 17:53:29.817334 29705 replica.cpp:694] Replica received learned notice for position 3 from @0.0.0.0:0
I1112 17:53:29.817575 29708 master.cpp:3847] Ignoring register slave message from slave(196)@172.17.6.242:48157 (bf058f59fe00) as admission is already in progress
I1112 17:53:29.817993 29705 leveldb.cpp:343] Persisting action (370 bytes) to leveldb took 631697ns
I1112 17:53:29.818019 29705 replica.cpp:715] Persisted action at 3
I1112 17:53:29.818042 29705 replica.cpp:700] Replica learned APPEND action at position 3
I1112 17:53:29.819962 29705 registrar.cpp:486] Successfully updated the 'registry' in 7.428864ms
I1112 17:53:29.819972 29708 log.cpp:704] Attempting to truncate the log to 3
I1112 17:53:29.821169 29704 slave.cpp:3169] Received ping from slave-observer(198)@172.17.6.242:48157
I1112 17:53:29.821400 29701 hierarchical.cpp:344] Added slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 (bf058f59fe00) with cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000] (allocated: )
I1112 17:53:29.822106 29704 coordinator.cpp:350] Coordinator attempting to write TRUNCATE action at position 4
I1112 17:53:29.822491 29701 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:53:29.822597 29701 hierarchical.cpp:892] Performed allocation for slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 in 1.075053ms
I1112 17:53:29.823355 29701 replica.cpp:540] Replica received write request for position 4 from (5660)@172.17.6.242:48157
I1112 17:53:29.823503 29705 master.cpp:3927] Registered slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00) with cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
I1112 17:53:29.823626 29707 slave.cpp:904] Registered with master master@172.17.6.242:48157; given slave ID 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
I1112 17:53:29.823667 29707 fetcher.cpp:79] Clearing fetcher cache
I1112 17:53:29.823720 29701 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 323431ns
I1112 17:53:29.823748 29701 replica.cpp:715] Persisted action at 4
I1112 17:53:29.823990 29695 status_update_manager.cpp:183] Resuming sending status updates
I1112 17:53:29.824004 29705 master.cpp:4979] Sending 1 offers to framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
I1112 17:53:29.824141 29707 slave.cpp:927] Checkpointing SlaveInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/slave.info'
I1112 17:53:29.824463 29705 sched.cpp:813] Scheduler::resourceOffers took 119643ns
I1112 17:53:29.824638 29707 slave.cpp:963] Forwarding total oversubscribed resources
I1112 17:53:29.825076 29695 master.cpp:4269] Received update of slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00) with total oversubscribed resources
I1112 17:53:29.825426 29676 resources.cpp:474] Parsing resources as JSON failed: cpus:1;mem:128;disk(role1):32
Trying semicolon-delimited string format instead
I1112 17:53:29.825443 29695 hierarchical.cpp:400] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 (bf058f59fe00) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):1024)
I1112 17:53:29.825541 29704 replica.cpp:694] Replica received learned notice for position 4 from @0.0.0.0:0
I1112 17:53:29.826035 29695 hierarchical.cpp:977] No resources available to allocate!
I1112 17:53:29.826062 29704 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 489313ns
I1112 17:53:29.826095 29695 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:53:29.826120 29695 hierarchical.cpp:892] Performed allocation for slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 in 621170ns
I1112 17:53:29.826128 29704 leveldb.cpp:401] Deleting ~2 keys from leveldb took 41561ns
I1112 17:53:29.826148 29704 replica.cpp:715] Persisted action at 4
I1112 17:53:29.826169 29704 replica.cpp:700] Replica learned TRUNCATE action at position 4
I1112 17:53:29.827690 29701 master.cpp:2915] Processing ACCEPT call for offers: [ 4104c1dc-cb09-41a0-8f89-339ad511ce2a-O0 ] on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00) for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
I1112 17:53:29.827743 29701 master.cpp:2711] Authorizing framework principal 'test-principal' to launch task 128c936c-abbb-4925-ae2f-170e01e62865 as user 'mesos'
I1112 17:53:29.829272 29709 master.cpp:3112] Applying CREATE operation for volumes disk(role1)[id1:path1]:64 from framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157 to slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:29.829699 29709 master.cpp:6205] Sending checkpointed resources disk(role1)[id1:path1]:64 to slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:29.830781 29709 master.hpp:176] Adding task 128c936c-abbb-4925-ae2f-170e01e62865 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 (bf058f59fe00)
I1112 17:53:29.831727 29700 hierarchical.cpp:513] Updated allocation of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 from cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):1024 to cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):960; disk(role1)[id1:path1]:64
I1112 17:53:29.831910 29706 slave.cpp:2274] Updated checkpointed resources from to disk(role1)[id1:path1]:64
I1112 17:53:29.832084 29709 master.cpp:3245] Launching task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:29.832653 29709 slave.cpp:1294] Got assigned task 128c936c-abbb-4925-ae2f-170e01e62865 for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:29.832810 29709 slave.cpp:4920] Checkpointing FrameworkInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/framework.info'
I1112 17:53:29.833256 29709 slave.cpp:4931] Checkpointing framework pid 'scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157' to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/framework.pid'
I1112 17:53:29.833261 29700 hierarchical.cpp:744] Recovered cpus(*):1; mem(*):896; ports(*):[31000-32000]; disk(role1):928 (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64) on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 from framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:29.833344 29700 hierarchical.cpp:781] Framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 filtered slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 for 5secs
I1112 17:53:29.833658 29709 resources.cpp:474] Parsing resources as JSON failed: cpus:0.1;mem:32
Trying semicolon-delimited string format instead
I1112 17:53:29.834321 29709 slave.cpp:1410] Launching task 128c936c-abbb-4925-ae2f-170e01e62865 for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:29.834424 29709 resources.cpp:474] Parsing resources as JSON failed: cpus:0.1;mem:32
Trying semicolon-delimited string format instead
I1112 17:53:29.835336 29709 paths.cpp:436] Trying to chown '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805' to user 'mesos'
I1112 17:53:29.844714 29709 slave.cpp:5360] Checkpointing ExecutorInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/executor.info'
I1112 17:53:29.845645 29709 slave.cpp:4999] Launching executor 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
I1112 17:53:29.846499 29704 containerizer.cpp:619] Starting container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805' for executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework '4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000'
I1112 17:53:29.846513 29709 slave.cpp:5383] Checkpointing TaskInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/tasks/128c936c-abbb-4925-ae2f-170e01e62865/task.info'
I1112 17:53:29.847183 29709 slave.cpp:1628] Queuing task '128c936c-abbb-4925-ae2f-170e01e62865' for executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:29.847409 29709 slave.cpp:682] Successfully attached file '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
I1112 17:53:29.847451 29706 posix.cpp:210] Changing the ownership of the persistent volume at '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/volumes/roles/role1/id1' with uid 1000 and gid 1000
I1112 17:53:29.856493 29706 posix.cpp:254] Adding symlink from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/volumes/roles/role1/id1' to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/path1' for persistent volume disk(role1)[id1:path1]:64 of container 4726aa7f-ee7c-41ed-a49f-f722e2d1c805
I1112 17:53:29.861801 29696 launcher.cpp:134] Forked child with pid '3079' for container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
I1112 17:53:29.862185 29696 containerizer.cpp:852] Checkpointing executor's forked pid 3079 to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/pids/forked.pid'
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1112 17:53:30.023684 3093 process.cpp:960] libprocess is initialized on 172.17.6.242:51169 for 16 cpus
I1112 17:53:30.024690 3093 logging.cpp:177] Logging to STDERR
I1112 17:53:30.027457 3093 exec.cpp:136] Version: 0.26.0
I1112 17:53:30.040992 3120 exec.cpp:186] Executor started at: executor(1)@172.17.6.242:51169 with pid 3093
I1112 17:53:30.045091 29703 slave.cpp:2405] Got registration for executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from executor(1)@172.17.6.242:51169
I1112 17:53:30.045225 29703 slave.cpp:2490] Checkpointing executor pid 'executor(1)@172.17.6.242:51169' to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/pids/libprocess.pid'
I1112 17:53:30.048463 3111 exec.cpp:210] Executor registered on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
I1112 17:53:30.048741 29703 slave.cpp:1793] Sending queued task '128c936c-abbb-4925-ae2f-170e01e62865' to executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 at executor(1)@172.17.6.242:51169
I1112 17:53:30.051081 3111 exec.cpp:222] Executor::registered took 414331ns
Registered executor on bf058f59fe00
I1112 17:53:30.053513 3119 exec.cpp:297] Executor asked to run task '128c936c-abbb-4925-ae2f-170e01e62865'
I1112 17:53:30.053709 3119 exec.cpp:306] Executor::launchTask took 137969ns
Starting task 128c936c-abbb-4925-ae2f-170e01e62865
Forked command at 3124
sh -c 'while true; do test -d path1; done'
I1112 17:53:30.058800 3119 exec.cpp:519] Executor sending status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.061161 29699 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from executor(1)@172.17.6.242:51169
I1112 17:53:30.061717 29699 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.061764 29699 status_update_manager.cpp:499] Creating StatusUpdate stream for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.063155 29699 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.064308 29699 status_update_manager.cpp:376] Forwarding update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to the slave
I1112 17:53:30.064816 29699 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to master@172.17.6.242:48157
I1112 17:53:30.065068 29699 slave.cpp:3005] Status update manager successfully handled status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.065956 29699 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to executor(1)@172.17.6.242:51169
I1112 17:53:30.065420 29707 master.cpp:4414] Status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:30.066629 29707 master.cpp:4462] Forwarding status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.066886 29707 master.cpp:6066] Updating the state of task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I1112 17:53:30.067373 29702 sched.cpp:921] Scheduler::statusUpdate took 159288ns
I1112 17:53:30.067679 29702 master.cpp:3571] Processing ACKNOWLEDGE call 2591ad57-0df4-4e31-b50f-7454091654d8 for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
I1112 17:53:30.068071 29702 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.068222 29702 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.068727 3116 exec.cpp:343] Executor received status update acknowledgement 2591ad57-0df4-4e31-b50f-7454091654d8 for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.072254 29702 slave.cpp:2345] Status update manager successfully handled status update acknowledgement (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.073086 29676 slave.cpp:601] Slave terminating
I1112 17:53:30.074015 29702 master.cpp:1083] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00) disconnected
I1112 17:53:30.074048 29702 master.cpp:2531] Disconnecting slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:30.074110 29702 master.cpp:2550] Deactivating slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:30.074353 29697 hierarchical.cpp:429] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 deactivated
I1112 17:53:30.092569 29676 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix
W1112 17:53:30.093389 29676 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges
I1112 17:53:30.099161 29706 slave.cpp:191] Slave started on 197)@172.17.6.242:48157
I1112 17:53:30.099205 29706 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):1024" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK"
I1112 17:53:30.099730 29706 credentials.hpp:85] Loading credential for authentication from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/credential'
I1112 17:53:30.100016 29706 slave.cpp:322] Slave using credential for: test-principal
I1112 17:53:30.100208 29706 resources.cpp:474] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):1024
Trying semicolon-delimited string format instead
I1112 17:53:30.100651 29706 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
I1112 17:53:30.100720 29706 slave.cpp:400] Slave attributes: [ ]
I1112 17:53:30.100733 29706 slave.cpp:405] Slave hostname: bf058f59fe00
I1112 17:53:30.100742 29706 slave.cpp:410] Slave checkpoint: true
I1112 17:53:30.102144 29710 state.cpp:54] Recovering state from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta'
I1112 17:53:30.108124 29698 fetcher.cpp:79] Clearing fetcher cache
I1112 17:53:30.108225 29698 slave.cpp:4318] Recovering framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.108312 29698 slave.cpp:5108] Recovering executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.109978 29698 status_update_manager.cpp:202] Recovering status update manager
I1112 17:53:30.110011 29698 status_update_manager.cpp:210] Recovering executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.110088 29698 status_update_manager.cpp:499] Creating StatusUpdate stream for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.110942 29703 slave.cpp:682] Successfully attached file '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
I1112 17:53:30.111120 29698 status_update_manager.cpp:802] Replaying status update stream for task 128c936c-abbb-4925-ae2f-170e01e62865
I1112 17:53:30.112169 29698 containerizer.cpp:385] Recovering containerizer
I1112 17:53:30.112259 29698 containerizer.cpp:440] Recovering container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805' for executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.114814 29706 slave.cpp:4170] Sending reconnect request to executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 at executor(1)@172.17.6.242:51169
I1112 17:53:30.117032 3116 exec.cpp:256] Received reconnect request from slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
I1112 17:53:30.117979 29695 slave.cpp:2553] Re-registering executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.119094 29700 posix.cpp:210] Changing the ownership of the persistent volume at '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/volumes/roles/role1/id1' with uid 1000 and gid 1000
I1112 17:53:30.123585 3116 exec.cpp:233] Executor re-registered on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
I1112 17:53:30.125902 3116 exec.cpp:245] Executor::reregistered took 110185ns
Re-registered executor on bf058f59fe00
I1112 17:53:30.145524 29700 hierarchical.cpp:977] No resources available to allocate!
I1112 17:53:30.145620 29700 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:53:30.145637 29709 slave.cpp:2702] Cleaning up un-reregistered executors
I1112 17:53:30.145656 29700 hierarchical.cpp:876] Performed allocation for 1 slaves in 239099ns
I1112 17:53:30.145795 29709 slave.cpp:4230] Finished recovery
I1112 17:53:30.146611 29709 slave.cpp:4387] Querying resource estimator for oversubscribable resources
I1112 17:53:30.147100 29709 slave.cpp:729] New master detected at master@172.17.6.242:48157
I1112 17:53:30.147147 29700 status_update_manager.cpp:176] Pausing sending status updates
I1112 17:53:30.147213 29709 slave.cpp:792] Authenticating with master master@172.17.6.242:48157
I1112 17:53:30.147239 29709 slave.cpp:797] Using default CRAM-MD5 authenticatee
I1112 17:53:30.147446 29709 slave.cpp:765] Detecting new master
I1112 17:53:30.147581 29709 slave.cpp:4401] Received oversubscribable resources from the resource estimator
I1112 17:53:30.147833 29709 authenticatee.cpp:123] Creating new client SASL connection
I1112 17:53:30.148392 29709 master.cpp:5150] Authenticating slave(197)@172.17.6.242:48157
I1112 17:53:30.148526 29707 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(469)@172.17.6.242:48157
I1112 17:53:30.148771 29708 authenticator.cpp:100] Creating new server SASL connection
I1112 17:53:30.149075 29704 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
I1112 17:53:30.149114 29704 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
I1112 17:53:30.149261 29708 authenticator.cpp:205] Received SASL authentication start
I1112 17:53:30.149334 29708 authenticator.cpp:327] Authentication requires more steps
I1112 17:53:30.149437 29708 authenticatee.cpp:260] Received SASL authentication step
I1112 17:53:30.149559 29708 authenticator.cpp:233] Received SASL authentication step
I1112 17:53:30.149593 29708 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1112 17:53:30.149611 29708 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I1112 17:53:30.149673 29708 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1112 17:53:30.149718 29708 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1112 17:53:30.149735 29708 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1112 17:53:30.149747 29708 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1112 17:53:30.149770 29708 authenticator.cpp:319] Authentication success
I1112 17:53:30.149871 29707 authenticatee.cpp:300] Authentication success
I1112 17:53:30.149989 29708 master.cpp:5180] Successfully authenticated principal 'test-principal' at slave(197)@172.17.6.242:48157
I1112 17:53:30.150027 29700 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(469)@172.17.6.242:48157
I1112 17:53:30.152010 29707 slave.cpp:860] Successfully authenticated with master master@172.17.6.242:48157
I1112 17:53:30.152513 29707 slave.cpp:1254] Will retry registration in 17.988166ms if necessary
I1112 17:53:30.153123 29700 master.cpp:4019] Re-registering slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:30.153628 29700 master.cpp:4207] Sending updated checkpointed resources disk(role1)[id1:path1]:64 to slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:30.153990 29700 slave.cpp:1004] Re-registered with master master@172.17.6.242:48157
I1112 17:53:30.154044 29707 hierarchical.cpp:417] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 reactivated
I1112 17:53:30.154106 29700 slave.cpp:1040] Forwarding total oversubscribed resources
I1112 17:53:30.154258 29699 status_update_manager.cpp:183] Resuming sending status updates
I1112 17:53:30.154536 29700 slave.cpp:2178] Updating framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 pid to scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
I1112 17:53:30.154603 29700 slave.cpp:2194] Checkpointing framework pid 'scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157' to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/framework.pid'
I1112 17:53:30.154841 29710 master.cpp:4269] Received update of slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00) with total oversubscribed resources
I1112 17:53:30.155228 29708 status_update_manager.cpp:183] Resuming sending status updates
I1112 17:53:30.155268 29710 hierarchical.cpp:400] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 (bf058f59fe00) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64)
I1112 17:53:30.155864 29709 master.cpp:3479] Telling slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00) to kill task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
I1112 17:53:30.155908 29700 slave.cpp:2274] Updated checkpointed resources from disk(role1)[id1:path1]:64 to disk(role1)[id1:path1]:64
I1112 17:53:30.156059 29700 slave.cpp:1821] Asked to kill task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.156313 29710 hierarchical.cpp:1162] Filtered offer with cpus(*):1; mem(*):896; ports(*):[31000-32000]; disk(role1):928 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.156425 29710 hierarchical.cpp:977] No resources available to allocate!
I1112 17:53:30.156582 29710 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:53:30.156693 29710 hierarchical.cpp:892] Performed allocation for slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 in 1.366876ms
I1112 17:53:30.158217 3108 exec.cpp:317] Executor asked to kill task '128c936c-abbb-4925-ae2f-170e01e62865'
I1112 17:53:30.158285 3108 exec.cpp:326] Executor::killTask took 29545ns
Shutting down
Sending SIGTERM to process tree at pid 3124
Killing the following process trees:
[
--- 3124 sh -c while true; do test -d path1; done
]
Command terminated with signal Terminated (pid: 3124)
I1112 17:53:30.261500 3114 exec.cpp:519] Executor sending status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.263711 29709 slave.cpp:2762] Handling status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from executor(1)@172.17.6.242:51169
I1112 17:53:30.263947 29709 slave.cpp:5298] Terminating task 128c936c-abbb-4925-ae2f-170e01e62865
I1112 17:53:30.264845 29709 posix.cpp:156] Removing symlink '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/path1' for persistent volume disk(role1)[id1:path1]:64 of container 4726aa7f-ee7c-41ed-a49f-f722e2d1c805
I1112 17:53:30.266062 29699 status_update_manager.cpp:322] Received status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.266145 29699 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.268108 29699 status_update_manager.cpp:376] Forwarding update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to the slave
I1112 17:53:30.268626 29699 slave.cpp:3087] Forwarding the update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to master@172.17.6.242:48157
I1112 17:53:30.268796 29699 slave.cpp:3005] Status update manager successfully handled status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.268828 29699 slave.cpp:3011] Sending acknowledgement for status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to executor(1)@172.17.6.242:51169
I1112 17:53:30.269331 29700 master.cpp:4414] Status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:30.269423 29700 master.cpp:4462] Forwarding status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.269665 29700 master.cpp:6066] Updating the state of task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
II1112 17:53:30.270424 29700 sched.cpp:921] Scheduler::statusUpdate took 130601ns
1112 17:53:30.270457 3114 exec.cpp:343] Executor received status update acknowledgement 01d489fd-62de-4853-9c2e-0dea985eb730 for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.270642 29703 hierarchical.cpp:744] Recovered cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: ) on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 from framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.270889 29703 master.cpp:3571] Processing ACKNOWLEDGE call 01d489fd-62de-4853-9c2e-0dea985eb730 for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
I1112 17:53:30.270972 29703 master.cpp:6132] Removing task 128c936c-abbb-4925-ae2f-170e01e62865 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00)
I1112 17:53:30.271466 29707 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.271620 29707 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.271903 29676 sched.cpp:1805] Asked to stop the driver
I1112 17:53:30.272076 29710 master.cpp:922] Master terminating
I1112 17:53:30.272251 29700 sched.cpp:1043] Stopping framework '4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000'
I1112 17:53:30.272675 29707 status_update_manager.cpp:530] Cleaning up status update stream for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.272737 29695 hierarchical.cpp:373] Removed slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
I1112 17:53:30.273172 29706 slave.cpp:2345] Status update manager successfully handled status update acknowledgement (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.273232 29706 slave.cpp:5339] Completing task 128c936c-abbb-4925-ae2f-170e01e62865
I1112 17:53:30.276796 29710 hierarchical.cpp:230] Removed framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.276867 29710 slave.cpp:3215] master@172.17.6.242:48157 exited
W1112 17:53:30.277004 29710 slave.cpp:3218] Master disconnected! Waiting for a new master to be elected
I1112 17:53:30.284138 29699 containerizer.cpp:1075] Destroying container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
I1112 17:53:30.350635 29699 containerizer.cpp:1258] Executor for container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805' has exited
I1112 17:53:30.353777 29705 slave.cpp:3553] Executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 terminated with signal Killed
I1112 17:53:30.353916 29705 slave.cpp:3657] Cleaning up executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 at executor(1)@172.17.6.242:51169
I1112 17:53:30.354750 29697 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805' for gc 6.99999589540741days in the future
I1112 17:53:30.355023 29697 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865' for gc 6.99999589335704days in the future
I1112 17:53:30.356132 29706 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805' for gc 6.99999588955852days in the future
I1112 17:53:30.356284 29706 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865' for gc 6.99999587701333days in the future
I1112 17:53:30.356299 29705 slave.cpp:3745] Cleaning up framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.356453 29706 status_update_manager.cpp:284] Closing status update streams for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
I1112 17:53:30.356506 29706 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000' for gc 6.99999587491852days in the future
I1112 17:53:30.356564 29705 slave.cpp:601] Slave terminating
I1112 17:53:30.356593 29706 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000' for gc 6.99999587395852days in the future
[ OK ] PersistentVolumeTest.SlaveRecovery (620 ms)
{noformat}
{noformat: title=Bad Run}
[ RUN ] PersistentVolumeTest.SlaveRecovery
I1112 17:49:00.333355 28097 leveldb.cpp:176] Opened db in 2.811376ms
I1112 17:49:00.336050 28097 leveldb.cpp:183] Compacted db in 2.655859ms
I1112 17:49:00.336125 28097 leveldb.cpp:198] Created db iterator in 34982ns
I1112 17:49:00.336144 28097 leveldb.cpp:204] Seeked to beginning of db in 8700ns
I1112 17:49:00.336156 28097 leveldb.cpp:273] Iterated through 0 keys in the db in 6732ns
I1112 17:49:00.336205 28097 replica.cpp:780] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
I1112 17:49:00.337661 28127 recover.cpp:449] Starting replica recovery
I1112 17:49:00.338086 28127 recover.cpp:475] Replica is in EMPTY status
I1112 17:49:00.340411 28130 replica.cpp:676] Replica in EMPTY status received a broadcasted recover request from (5611)@172.17.6.239:32961
I1112 17:49:00.340880 28130 recover.cpp:195] Received a recover response from a replica in EMPTY status
I1112 17:49:00.341501 28130 recover.cpp:566] Updating replica status to STARTING
I1112 17:49:00.342439 28121 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 724899ns
I1112 17:49:00.342470 28121 replica.cpp:323] Persisted replica status to STARTING
I1112 17:49:00.342743 28124 recover.cpp:475] Replica is in STARTING status
I1112 17:49:00.345824 28120 master.cpp:367] Master 5b272a7d-2df2-4bb6-b65f-e2633bea9888 (f4f9d084d7ba) started on 172.17.6.239:32961
I1112 17:49:00.345859 28120 master.cpp:369] Flags at startup: --acls="register_frameworks {
principals {
values: "test-principal"
}
roles {
values: "role1"
}
}
" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/XaPAlx/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --roles="role1" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.26.0/_inst/share/mesos/webui" --work_dir="/tmp/XaPAlx/master" --zk_session_timeout="10secs"
I1112 17:49:00.346254 28120 master.cpp:414] Master only allowing authenticated frameworks to register
I1112 17:49:00.346266 28120 master.cpp:419] Master only allowing authenticated slaves to register
I1112 17:49:00.346274 28120 credentials.hpp:37] Loading credentials for authentication from '/tmp/XaPAlx/credentials'
I1112 17:49:00.346552 28120 master.cpp:458] Using default 'crammd5' authenticator
I1112 17:49:00.346678 28120 master.cpp:495] Authorization enabled
I1112 17:49:00.346774 28127 replica.cpp:676] Replica in STARTING status received a broadcasted recover request from (5612)@172.17.6.239:32961
I1112 17:49:00.347045 28127 hierarchical.cpp:151] Initialized hierarchical allocator process
I1112 17:49:00.347195 28127 recover.cpp:195] Received a recover response from a replica in STARTING status
I1112 17:49:00.347899 28125 recover.cpp:566] Updating replica status to VOTING
I1112 17:49:00.347997 28119 whitelist_watcher.cpp:79] No whitelist given
I1112 17:49:00.348471 28119 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 438348ns
I1112 17:49:00.348497 28119 replica.cpp:323] Persisted replica status to VOTING
I1112 17:49:00.348598 28125 recover.cpp:580] Successfully joined the Paxos group
I1112 17:49:00.349103 28125 recover.cpp:464] Recover process terminated
I1112 17:49:00.350262 28128 master.cpp:1606] The newly elected leader is master@172.17.6.239:32961 with id 5b272a7d-2df2-4bb6-b65f-e2633bea9888
I1112 17:49:00.350395 28128 master.cpp:1619] Elected as the leading master!
I1112 17:49:00.350510 28128 master.cpp:1379] Recovering from registrar
I1112 17:49:00.350709 28120 registrar.cpp:309] Recovering registrar
I1112 17:49:00.351351 28124 log.cpp:661] Attempting to start the writer
I1112 17:49:00.354029 28124 replica.cpp:496] Replica received implicit promise request from (5613)@172.17.6.239:32961 with proposal 1
I1112 17:49:00.354639 28124 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 560313ns
I1112 17:49:00.354667 28124 replica.cpp:345] Persisted promised to 1
I1112 17:49:00.355598 28124 coordinator.cpp:240] Coordinator attempting to fill missing positions
I1112 17:49:00.357206 28124 replica.cpp:391] Replica received explicit promise request from (5614)@172.17.6.239:32961 for position 0 with proposal 2
I1112 17:49:00.357781 28124 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 478236ns
I1112 17:49:00.357807 28124 replica.cpp:715] Persisted action at 0
I1112 17:49:00.359031 28123 replica.cpp:540] Replica received write request for position 0 from (5615)@172.17.6.239:32961
I1112 17:49:00.359105 28123 leveldb.cpp:438] Reading position from leveldb took 45943ns
I1112 17:49:00.359474 28123 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 328418ns
I1112 17:49:00.359503 28123 replica.cpp:715] Persisted action at 0
I1112 17:49:00.360296 28127 replica.cpp:694] Replica received learned notice for position 0 from @0.0.0.0:0
I1112 17:49:00.360959 28127 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 642538ns
I1112 17:49:00.360986 28127 replica.cpp:715] Persisted action at 0
I1112 17:49:00.361006 28127 replica.cpp:700] Replica learned NOP action at position 0
I1112 17:49:00.361683 28129 log.cpp:677] Writer started with ending position 0
I1112 17:49:00.363149 28122 leveldb.cpp:438] Reading position from leveldb took 85104ns
I1112 17:49:00.364382 28116 registrar.cpp:342] Successfully fetched the registry (0B) in 13.476864ms
I1112 17:49:00.364588 28116 registrar.cpp:441] Applied 1 operations in 44922ns; attempting to update the 'registry'
I1112 17:49:00.365350 28116 log.cpp:685] Attempting to append 176 bytes to the log
I1112 17:49:00.365546 28127 coordinator.cpp:350] Coordinator attempting to write APPEND action at position 1
I1112 17:49:00.366474 28115 replica.cpp:540] Replica received write request for position 1 from (5616)@172.17.6.239:32961
I1112 17:49:00.367040 28115 leveldb.cpp:343] Persisting action (195 bytes) to leveldb took 537014ns
I1112 17:49:00.367069 28115 replica.cpp:715] Persisted action at 1
I1112 17:49:00.367697 28126 replica.cpp:694] Replica received learned notice for position 1 from @0.0.0.0:0
I1112 17:49:00.368448 28126 leveldb.cpp:343] Persisting action (197 bytes) to leveldb took 732384ns
I1112 17:49:00.368476 28126 replica.cpp:715] Persisted action at 1
I1112 17:49:00.368499 28126 replica.cpp:700] Replica learned APPEND action at position 1
I1112 17:49:00.370054 28126 registrar.cpp:486] Successfully updated the 'registry' in 5.339904ms
I1112 17:49:00.370199 28126 registrar.cpp:372] Successfully recovered registrar
I1112 17:49:00.370453 28126 log.cpp:704] Attempting to truncate the log to 1
I1112 17:49:00.370818 28126 master.cpp:1416] Recovered 0 slaves from the Registry (137B) ; allowing 10mins for slaves to re-register
I1112 17:49:00.370919 28126 coordinator.cpp:350] Coordinator attempting to write TRUNCATE action at position 2
I1112 17:49:00.371949 28120 replica.cpp:540] Replica received write request for position 2 from (5617)@172.17.6.239:32961
I1112 17:49:00.372375 28120 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 398089ns
I1112 17:49:00.372402 28120 replica.cpp:715] Persisted action at 2
I1112 17:49:00.373205 28121 replica.cpp:694] Replica received learned notice for position 2 from @0.0.0.0:0
I1112 17:49:00.373517 28121 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 297514ns
I1112 17:49:00.373572 28121 leveldb.cpp:401] Deleting ~1 keys from leveldb took 40498ns
I1112 17:49:00.373594 28121 replica.cpp:715] Persisted action at 2
I1112 17:49:00.373615 28121 replica.cpp:700] Replica learned TRUNCATE action at position 2
I1112 17:49:00.385612 28097 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix
W1112 17:49:00.386059 28097 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges
I1112 17:49:00.390331 28128 slave.cpp:191] Slave started on 196)@172.17.6.239:32961
I1112 17:49:00.390383 28128 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):1024" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj"
I1112 17:49:00.390844 28128 credentials.hpp:85] Loading credential for authentication from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/credential'
I1112 17:49:00.391086 28128 slave.cpp:322] Slave using credential for: test-principal
I1112 17:49:00.391239 28128 resources.cpp:474] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):1024
Trying semicolon-delimited string format instead
I1112 17:49:00.391666 28128 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
I1112 17:49:00.391737 28128 slave.cpp:400] Slave attributes: [ ]
I1112 17:49:00.391751 28128 slave.cpp:405] Slave hostname: f4f9d084d7ba
I1112 17:49:00.391758 28128 slave.cpp:410] Slave checkpoint: true
I1112 17:49:00.393064 28121 state.cpp:54] Recovering state from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta'
I1112 17:49:00.393465 28121 status_update_manager.cpp:202] Recovering status update manager
I1112 17:49:00.393633 28121 containerizer.cpp:385] Recovering containerizer
I1112 17:49:00.393672 28097 sched.cpp:166] Version: 0.26.0
I1112 17:49:00.394464 28118 sched.cpp:264] New master detected at master@172.17.6.239:32961
I1112 17:49:00.394554 28118 sched.cpp:320] Authenticating with master master@172.17.6.239:32961
I1112 17:49:00.394579 28118 sched.cpp:327] Using default CRAM-MD5 authenticatee
I1112 17:49:00.394716 28115 slave.cpp:4230] Finished recovery
I1112 17:49:00.394938 28118 authenticatee.cpp:123] Creating new client SASL connection
I1112 17:49:00.395225 28125 master.cpp:5150] Authenticating scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
I1112 17:49:00.395256 28115 slave.cpp:4387] Querying resource estimator for oversubscribable resources
I1112 17:49:00.395397 28125 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(465)@172.17.6.239:32961
I1112 17:49:00.395648 28115 slave.cpp:729] New master detected at master@172.17.6.239:32961
I1112 17:49:00.395697 28125 authenticator.cpp:100] Creating new server SASL connection
I1112 17:49:00.395706 28115 slave.cpp:792] Authenticating with master master@172.17.6.239:32961
I1112 17:49:00.395723 28115 slave.cpp:797] Using default CRAM-MD5 authenticatee
I1112 17:49:00.395828 28115 slave.cpp:765] Detecting new master
I1112 17:49:00.395895 28125 status_update_manager.cpp:176] Pausing sending status updates
I1112 17:49:00.395947 28115 slave.cpp:4401] Received oversubscribable resources from the resource estimator
I1112 17:49:00.395998 28125 authenticatee.cpp:123] Creating new client SASL connection
I1112 17:49:00.396046 28115 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
I1112 17:49:00.396080 28115 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
I1112 17:49:00.396199 28125 master.cpp:5150] Authenticating slave(196)@172.17.6.239:32961
I1112 17:49:00.396230 28115 authenticator.cpp:205] Received SASL authentication start
I1112 17:49:00.396289 28115 authenticator.cpp:327] Authentication requires more steps
I1112 17:49:00.396329 28125 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(466)@172.17.6.239:32961
I1112 17:49:00.396358 28115 authenticatee.cpp:260] Received SASL authentication step
I1112 17:49:00.396493 28115 authenticator.cpp:100] Creating new server SASL connection
I1112 17:49:00.396683 28115 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
I1112 17:49:00.396714 28115 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
I1112 17:49:00.396781 28115 authenticator.cpp:205] Received SASL authentication start
I1112 17:49:00.396822 28115 authenticator.cpp:327] Authentication requires more steps
I1112 17:49:00.396891 28115 authenticatee.cpp:260] Received SASL authentication step
I1112 17:49:00.396970 28125 authenticator.cpp:233] Received SASL authentication step
I1112 17:49:00.396999 28125 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1112 17:49:00.397011 28125 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I1112 17:49:00.397058 28125 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1112 17:49:00.397085 28125 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1112 17:49:00.397096 28125 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1112 17:49:00.397104 28125 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1112 17:49:00.397119 28125 authenticator.cpp:319] Authentication success
I1112 17:49:00.397202 28115 authenticator.cpp:233] Received SASL authentication step
I1112 17:49:00.397230 28115 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1112 17:49:00.397241 28115 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I1112 17:49:00.397267 28115 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1112 17:49:00.397289 28115 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1112 17:49:00.397300 28115 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1112 17:49:00.397307 28115 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1112 17:49:00.397321 28115 authenticator.cpp:319] Authentication success
I1112 17:49:00.397457 28115 authenticatee.cpp:300] Authentication success
I1112 17:49:00.397538 28115 master.cpp:5180] Successfully authenticated principal 'test-principal' at slave(196)@172.17.6.239:32961
I1112 17:49:00.397610 28115 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(466)@172.17.6.239:32961
I1112 17:49:00.397871 28115 slave.cpp:860] Successfully authenticated with master master@172.17.6.239:32961
I1112 17:49:00.398057 28125 authenticatee.cpp:300] Authentication success
I1112 17:49:00.398133 28125 master.cpp:5180] Successfully authenticated principal 'test-principal' at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
I1112 17:49:00.398200 28125 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(465)@172.17.6.239:32961
I1112 17:49:00.398424 28125 sched.cpp:409] Successfully authenticated with master master@172.17.6.239:32961
I1112 17:49:00.398447 28125 sched.cpp:716] Sending SUBSCRIBE call to master@172.17.6.239:32961
I1112 17:49:00.398531 28125 sched.cpp:749] Will retry registration in 281.865576ms if necessary
I1112 17:49:00.398694 28125 master.cpp:2176] Received SUBSCRIBE call for framework 'default' at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
I1112 17:49:00.398718 28125 master.cpp:1645] Authorizing framework principal 'test-principal' to receive offers for role 'role1'
I1112 17:49:00.399085 28115 slave.cpp:1254] Will retry registration in 13.451406ms if necessary
I1112 17:49:00.400010 28122 master.cpp:2247] Subscribing framework default with checkpointing enabled and capabilities [ ]
I1112 17:49:00.400645 28122 master.cpp:3859] Registering slave at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) with id 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
I1112 17:49:00.401038 28122 hierarchical.cpp:195] Added framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.401165 28122 hierarchical.cpp:977] No resources available to allocate!
I1112 17:49:00.401298 28122 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:49:00.401407 28122 hierarchical.cpp:876] Performed allocation for 0 slaves in 290991ns
I1112 17:49:00.401165 28124 sched.cpp:643] Framework registered with 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.401517 28130 registrar.cpp:441] Applied 1 operations in 91430ns; attempting to update the 'registry'
I1112 17:49:00.401669 28124 sched.cpp:657] Scheduler::registered took 75312ns
I1112 17:49:00.402467 28118 log.cpp:685] Attempting to append 349 bytes to the log
I1112 17:49:00.402775 28118 coordinator.cpp:350] Coordinator attempting to write APPEND action at position 3
I1112 17:49:00.403746 28118 replica.cpp:540] Replica received write request for position 3 from (5629)@172.17.6.239:32961
I1112 17:49:00.404013 28118 leveldb.cpp:343] Persisting action (368 bytes) to leveldb took 236794ns
I1112 17:49:00.404042 28118 replica.cpp:715] Persisted action at 3
I1112 17:49:00.404981 28115 replica.cpp:694] Replica received learned notice for position 3 from @0.0.0.0:0
I1112 17:49:00.405659 28115 leveldb.cpp:343] Persisting action (370 bytes) to leveldb took 639583ns
I1112 17:49:00.405690 28115 replica.cpp:715] Persisted action at 3
I1112 17:49:00.405716 28115 replica.cpp:700] Replica learned APPEND action at position 3
I1112 17:49:00.408030 28115 registrar.cpp:486] Successfully updated the 'registry' in 6.365952ms
I1112 17:49:00.409021 28130 log.cpp:704] Attempting to truncate the log to 3
I1112 17:49:00.409071 28124 slave.cpp:3169] Received ping from slave-observer(197)@172.17.6.239:32961
I1112 17:49:00.409194 28124 coordinator.cpp:350] Coordinator attempting to write TRUNCATE action at position 4
I1112 17:49:00.409203 28121 hierarchical.cpp:344] Added slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 (f4f9d084d7ba) with cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000] (allocated: )
I1112 17:49:00.409489 28115 master.cpp:3927] Registered slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) with cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
I1112 17:49:00.409637 28115 slave.cpp:904] Registered with master master@172.17.6.239:32961; given slave ID 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
I1112 17:49:00.409657 28115 fetcher.cpp:79] Clearing fetcher cache
I1112 17:49:00.409976 28130 status_update_manager.cpp:183] Resuming sending status updates
I1112 17:49:00.410019 28121 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:49:00.410056 28121 hierarchical.cpp:892] Performed allocation for slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 in 807103ns
I1112 17:49:00.410110 28115 slave.cpp:927] Checkpointing SlaveInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/slave.info'
I1112 17:49:00.410204 28121 replica.cpp:540] Replica received write request for position 4 from (5630)@172.17.6.239:32961
I1112 17:49:00.410537 28130 master.cpp:4979] Sending 1 offers to framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
I1112 17:49:00.410555 28115 slave.cpp:963] Forwarding total oversubscribed resources
I1112 17:49:00.410729 28121 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 502328ns
I1112 17:49:00.410755 28121 replica.cpp:715] Persisted action at 4
I1112 17:49:00.410876 28115 master.cpp:4269] Received update of slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) with total oversubscribed resources
I1112 17:49:00.411187 28115 hierarchical.cpp:400] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 (f4f9d084d7ba) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):1024)
I1112 17:49:00.411337 28130 sched.cpp:813] Scheduler::resourceOffers took 448533ns
I1112 17:49:00.411788 28115 hierarchical.cpp:977] No resources available to allocate!
I1112 17:49:00.411802 28097 resources.cpp:474] Parsing resources as JSON failed: cpus:1;mem:128;disk(role1):32
Trying semicolon-delimited string format instead
I1112 17:49:00.411922 28115 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:49:00.412221 28123 replica.cpp:694] Replica received learned notice for position 4 from @0.0.0.0:0
I1112 17:49:00.412698 28115 hierarchical.cpp:892] Performed allocation for slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 in 1.374186ms
I1112 17:49:00.413398 28123 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 713860ns
I1112 17:49:00.413476 28123 leveldb.cpp:401] Deleting ~2 keys from leveldb took 57815ns
I1112 17:49:00.413503 28123 replica.cpp:715] Persisted action at 4
I1112 17:49:00.413532 28123 replica.cpp:700] Replica learned TRUNCATE action at position 4
I1112 17:49:00.415221 28123 master.cpp:2915] Processing ACCEPT call for offers: [ 5b272a7d-2df2-4bb6-b65f-e2633bea9888-O0 ] on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
I1112 17:49:00.415282 28123 master.cpp:2711] Authorizing framework principal 'test-principal' to launch task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 as user 'mesos'
I1112 17:49:00.416872 28123 master.cpp:3112] Applying CREATE operation for volumes disk(role1)[id1:path1]:64 from framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961 to slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.417266 28123 master.cpp:6205] Sending checkpointed resources disk(role1)[id1:path1]:64 to slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.418344 28123 master.hpp:176] Adding task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 (f4f9d084d7ba)
I1112 17:49:00.418550 28123 master.cpp:3245] Launching task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.422189 28121 slave.cpp:2274] Updated checkpointed resources from to disk(role1)[id1:path1]:64
I1112 17:49:00.422458 28121 slave.cpp:1294] Got assigned task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.422623 28121 slave.cpp:4920] Checkpointing FrameworkInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/framework.info'
I1112 17:49:00.423600 28123 hierarchical.cpp:513] Updated allocation of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 from cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):1024 to cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):960; disk(role1)[id1:path1]:64
I1112 17:49:00.424105 28123 hierarchical.cpp:744] Recovered cpus(*):1; mem(*):896; ports(*):[31000-32000]; disk(role1):928 (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64) on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 from framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.424186 28123 hierarchical.cpp:781] Framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 filtered slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 for 5secs
I1112 17:49:00.424402 28121 slave.cpp:4931] Checkpointing framework pid 'scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961' to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/framework.pid'
I1112 17:49:00.424839 28121 resources.cpp:474] Parsing resources as JSON failed: cpus:0.1;mem:32
Trying semicolon-delimited string format instead
I1112 17:49:00.425565 28121 slave.cpp:1410] Launching task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.425664 28121 resources.cpp:474] Parsing resources as JSON failed: cpus:0.1;mem:32
Trying semicolon-delimited string format instead
I1112 17:49:00.426517 28121 paths.cpp:436] Trying to chown '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' to user 'mesos'
I1112 17:49:00.437039 28121 slave.cpp:5360] Checkpointing ExecutorInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/executor.info'
I1112 17:49:00.438006 28121 slave.cpp:4999] Launching executor 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
I1112 17:49:00.438782 28121 slave.cpp:5383] Checkpointing TaskInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc/tasks/2b35f3fc-373c-4b5f-bbe6-dc3279117145/task.info'
I1112 17:49:00.439314 28124 containerizer.cpp:619] Starting container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' for executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework '5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000'
I1112 17:49:00.439545 28121 slave.cpp:1628] Queuing task '2b35f3fc-373c-4b5f-bbe6-dc3279117145' for executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.439808 28121 slave.cpp:682] Successfully attached file '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
I1112 17:49:00.440218 28119 posix.cpp:210] Changing the ownership of the persistent volume at '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/volumes/roles/role1/id1' with uid 1000 and gid 1000
I1112 17:49:00.449080 28119 posix.cpp:254] Adding symlink from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/volumes/roles/role1/id1' to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc/path1' for persistent volume disk(role1)[id1:path1]:64 of container b9dbc84e-57fa-49d5-840f-8f6d9f0233fc
I1112 17:49:00.453404 28122 launcher.cpp:134] Forked child with pid '1486' for container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
I1112 17:49:00.453604 28122 containerizer.cpp:852] Checkpointing executor's forked pid 1486 to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc/pids/forked.pid'
2015-11-12 17:49:00,593:28097(0x2ad09360b700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:44265] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1112 17:49:00.604712 1500 process.cpp:960] libprocess is initialized on 172.17.6.239:46865 for 16 cpus
I1112 17:49:00.605552 1500 logging.cpp:177] Logging to STDERR
I1112 17:49:00.608021 1500 exec.cpp:136] Version: 0.26.0
I1112 17:49:00.619716 1521 exec.cpp:186] Executor started at: executor(1)@172.17.6.239:46865 with pid 1500
I1112 17:49:00.622460 28127 slave.cpp:2405] Got registration for executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from executor(1)@172.17.6.239:46865
I1112 17:49:00.623016 28127 slave.cpp:2490] Checkpointing executor pid 'executor(1)@172.17.6.239:46865' to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc/pids/libprocess.pid'
I1112 17:49:00.625865 28121 slave.cpp:1793] Sending queued task '2b35f3fc-373c-4b5f-bbe6-dc3279117145' to executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
I1112 17:49:00.627167 1519 exec.cpp:210] Executor registered on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
I1112 17:49:00.628980 1519 exec.cpp:222] Executor::registered took 358581ns
Registered executor on f4f9d084d7ba
I1112 17:49:00.631170 1525 exec.cpp:297] Executor asked to run task '2b35f3fc-373c-4b5f-bbe6-dc3279117145'
I1112 17:49:00.631330 1525 exec.cpp:306] Executor::launchTask took 136825ns
Starting task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
Forked command at 1531
sh -c 'while true; do test -d path1; done'
I1112 17:49:00.635591 1526 exec.cpp:519] Executor sending status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.637117 28120 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from executor(1)@172.17.6.239:46865
I1112 17:49:00.637634 28120 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.637676 28120 status_update_manager.cpp:499] Creating StatusUpdate stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.638445 28120 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.639673 28120 status_update_manager.cpp:376] Forwarding update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to the slave
I1112 17:49:00.640115 28122 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to master@172.17.6.239:32961
I1112 17:49:00.640601 28122 slave.cpp:3005] Status update manager successfully handled status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.640805 28129 master.cpp:4414] Status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.640857 28129 master.cpp:4462] Forwarding status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.640720 28122 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to executor(1)@172.17.6.239:46865
I1112 17:49:00.641095 28129 master.cpp:6066] Updating the state of task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I1112 17:49:00.641374 28116 sched.cpp:921] Scheduler::statusUpdate took 141504ns
I1112 17:49:00.642500 28128 master.cpp:3571] Processing ACKNOWLEDGE call 24a2b427-4799-4be0-9cb8-d5bf525b32c6 for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
I1112 17:49:00.643096 28116 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.643277 28116 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.644330 28116 slave.cpp:2345] Status update manager successfully handled status update acknowledgement (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.645215 28097 slave.cpp:601] Slave terminating
I1112 17:49:00.646117 1517 exec.cpp:343] Executor received status update acknowledgement 24a2b427-4799-4be0-9cb8-d5bf525b32c6 for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.646978 28116 master.cpp:1083] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) disconnected
I1112 17:49:00.647253 28116 master.cpp:2531] Disconnecting slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.647650 28116 master.cpp:2550] Deactivating slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.648510 28124 hierarchical.cpp:429] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 deactivated
I1112 17:49:00.652220 28097 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix
W1112 17:49:00.652614 28097 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges
I1112 17:49:00.658809 28118 slave.cpp:191] Slave started on 197)@172.17.6.239:32961
I1112 17:49:00.658880 28118 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):1024" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj"
I1112 17:49:00.660478 28118 credentials.hpp:85] Loading credential for authentication from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/credential'
I1112 17:49:00.662098 28118 slave.cpp:322] Slave using credential for: test-principal
I1112 17:49:00.662643 28118 resources.cpp:474] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):1024
Trying semicolon-delimited string format instead
I1112 17:49:00.663300 28118 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
I1112 17:49:00.663388 28118 slave.cpp:400] Slave attributes: [ ]
I1112 17:49:00.663406 28118 slave.cpp:405] Slave hostname: f4f9d084d7ba
I1112 17:49:00.663419 28118 slave.cpp:410] Slave checkpoint: true
I1112 17:49:00.669028 28115 state.cpp:54] Recovering state from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta'
I1112 17:49:00.675704 28121 fetcher.cpp:79] Clearing fetcher cache
I1112 17:49:00.675799 28121 slave.cpp:4318] Recovering framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.675891 28121 slave.cpp:5108] Recovering executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.677217 28119 status_update_manager.cpp:202] Recovering status update manager
I1112 17:49:00.677250 28119 status_update_manager.cpp:210] Recovering executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.677326 28119 status_update_manager.cpp:499] Creating StatusUpdate stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.677896 28119 status_update_manager.cpp:802] Replaying status update stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
I1112 17:49:00.678973 28121 slave.cpp:682] Successfully attached file '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
I1112 17:49:00.679440 28121 containerizer.cpp:385] Recovering containerizer
I1112 17:49:00.679527 28121 containerizer.cpp:440] Recovering container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' for executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.685341 28115 slave.cpp:4170] Sending reconnect request to executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
I1112 17:49:00.690004 1529 exec.cpp:256] Received reconnect request from slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
I1112 17:49:00.703845 28122 hierarchical.cpp:977] No resources available to allocate!
I1112 17:49:00.703917 28122 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:49:00.704025 28122 hierarchical.cpp:876] Performed allocation for 1 slaves in 271350ns
I1112 17:49:00.703974 28116 slave.cpp:2702] Cleaning up un-reregistered executors
I1112 17:49:00.704107 28116 slave.cpp:2720] Killing un-reregistered executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
I1112 17:49:00.704262 28116 slave.cpp:4230] Finished recovery
I1112 17:49:00.704397 28123 containerizer.cpp:1075] Destroying container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
I1112 17:49:00.706696 28116 slave.cpp:4387] Querying resource estimator for oversubscribable resources
I1112 17:49:00.707142 28127 slave.cpp:4401] Received oversubscribable resources from the resource estimator
W1112 17:49:00.707540 28127 slave.cpp:2546] Shutting down executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 because the slave is not in recovery mode
I1112 17:49:00.708133 28116 slave.cpp:729] New master detected at master@172.17.6.239:32961
I1112 17:49:00.708228 28116 slave.cpp:792] Authenticating with master master@172.17.6.239:32961
I1112 17:49:00.708253 28116 slave.cpp:797] Using default CRAM-MD5 authenticatee
I1112 17:49:00.708421 28116 slave.cpp:765] Detecting new master
I1112 17:49:00.708533 28116 status_update_manager.cpp:176] Pausing sending status updates
I1112 17:49:00.708673 28116 authenticatee.cpp:123] Creating new client SASL connection
I1112 17:49:00.709913 28116 master.cpp:5150] Authenticating slave(197)@172.17.6.239:32961
I1112 17:49:00.710218 28116 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(467)@172.17.6.239:32961
I1112 17:49:00.710438 1516 exec.cpp:383] Executor asked to shutdown
I1112 17:49:00.710582 1516 exec.cpp:398] Executor::shutdown took 35481ns
I1112 17:49:00.710598 28116 authenticator.cpp:100] Creating new server SASL connection
I1112 17:49:00.710705 1516 exec.cpp:82] Scheduling shutdown of the executor
Shutting down
Sending SIGTERM to process tree at pid 1531
I1112 17:49:00.710830 28116 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
I1112 17:49:00.710866 28116 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
I1112 17:49:00.712005 28116 authenticator.cpp:205] Received SASL authentication start
I1112 17:49:00.712069 28116 authenticator.cpp:327] Authentication requires more steps
I1112 17:49:00.712214 28124 authenticatee.cpp:260] Received SASL authentication step
I1112 17:49:00.712357 28116 authenticator.cpp:233] Received SASL authentication step
I1112 17:49:00.712389 28116 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1112 17:49:00.712412 28116 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I1112 17:49:00.712467 28116 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1112 17:49:00.712507 28116 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1112 17:49:00.712524 28116 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1112 17:49:00.712534 28116 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1112 17:49:00.712555 28116 authenticator.cpp:319] Authentication success
I1112 17:49:00.712708 28128 authenticatee.cpp:300] Authentication success
I1112 17:49:00.721004 28122 master.cpp:5180] Successfully authenticated principal 'test-principal' at slave(197)@172.17.6.239:32961
I1112 17:49:00.721352 28117 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(467)@172.17.6.239:32961
Killing the following process trees:
[
--- 1531 sh -c while true; do test -d path1; done
]
I1112 17:49:00.721968 28128 slave.cpp:860] Successfully authenticated with master master@172.17.6.239:32961
I1112 17:49:00.723456 28130 master.cpp:4019] Re-registering slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.724056 28126 hierarchical.cpp:417] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 reactivated
I1112 17:49:00.723914 28130 master.cpp:4207] Sending updated checkpointed resources disk(role1)[id1:path1]:64 to slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.724306 28128 slave.cpp:1254] Will retry registration in 17.384714ms if necessary
I1112 17:49:00.726639 28128 slave.cpp:1004] Re-registered with master master@172.17.6.239:32961
I1112 17:49:00.726775 28122 master.cpp:3479] Telling slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba) to kill task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
I1112 17:49:00.726794 28127 status_update_manager.cpp:183] Resuming sending status updates
I1112 17:49:00.726913 28128 slave.cpp:1040] Forwarding total oversubscribed resources
I1112 17:49:00.727146 28126 master.cpp:4269] Received update of slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba) with total oversubscribed resources
I1112 17:49:00.727447 28126 hierarchical.cpp:400] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 (f4f9d084d7ba) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64)
I1112 17:49:00.728248 28126 hierarchical.cpp:1162] Filtered offer with cpus(*):1; mem(*):896; ports(*):[31000-32000]; disk(role1):928 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.728301 28126 hierarchical.cpp:977] No resources available to allocate!
I1112 17:49:00.728350 28126 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:49:00.728374 28126 hierarchical.cpp:892] Performed allocation for slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 in 865336ns
I1112 17:49:00.728529 28128 slave.cpp:2178] Updating framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 pid to scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
I1112 17:49:00.728587 28128 slave.cpp:2194] Checkpointing framework pid 'scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961' to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/framework.pid'
I1112 17:49:00.729202 28119 status_update_manager.cpp:183] Resuming sending status updates
I1112 17:49:00.729709 28128 slave.cpp:2274] Updated checkpointed resources from disk(role1)[id1:path1]:64 to disk(role1)[id1:path1]:64
I1112 17:49:00.729821 28128 slave.cpp:1821] Asked to kill task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
W1112 17:49:00.729884 28128 slave.cpp:1948] Ignoring kill task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 because the executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865 is terminating/terminated
I1112 17:49:00.804975 28117 containerizer.cpp:1258] Executor for container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' has exited
I1112 17:49:00.807514 28117 slave.cpp:3553] Executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 terminated with signal Killed
I1112 17:49:00.807701 28117 slave.cpp:2762] Handling status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from @0.0.0.0:0
I1112 17:49:00.807884 28117 slave.cpp:5298] Terminating task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
W1112 17:49:00.808377 28117 containerizer.cpp:967] Ignoring update for unknown container: b9dbc84e-57fa-49d5-840f-8f6d9f0233fc
I1112 17:49:00.808811 28117 status_update_manager.cpp:322] Received status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.808878 28117 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.814139 28117 status_update_manager.cpp:376] Forwarding update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to the slave
I1112 17:49:00.814621 28122 slave.cpp:3087] Forwarding the update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to master@172.17.6.239:32961
I1112 17:49:00.814888 28122 slave.cpp:3005] Status update manager successfully handled status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.815172 28117 master.cpp:4414] Status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.815218 28117 master.cpp:4462] Forwarding status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.815402 28117 master.cpp:6066] Updating the state of task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (latest state: TASK_LOST, status update state: TASK_LOST)
I1112 17:49:00.815636 28122 sched.cpp:921] Scheduler::statusUpdate took 135083ns
I1112 17:49:00.816069 28117 master.cpp:3571] Processing ACKNOWLEDGE call 0c911b27-4aa4-4435-965d-1b4b5033a40c for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
I1112 17:49:00.816118 28122 hierarchical.cpp:744] Recovered cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: ) on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 from framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.816128 28117 master.cpp:6132] Removing task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.816721 28127 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.817196 28127 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
../../src/tests/persistent_volume_tests.cpp:709: Failure
Value of: status2.get().state()
Actual: TASK_LOST
Expected: TASK_KILLED
I1112 17:49:00.818148 28127 status_update_manager.cpp:530] Cleaning up status update stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.818176 28097 sched.cpp:1805] Asked to stop the driver
I1112 17:49:00.818294 28097 master.cpp:922] Master terminating
I1112 17:49:00.818581 28127 sched.cpp:1043] Stopping framework '5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000'
I1112 17:49:00.818915 28127 slave.cpp:2345] Status update manager successfully handled status update acknowledgement (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.819017 28118 hierarchical.cpp:373] Removed slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
I1112 17:49:00.819088 28127 slave.cpp:5339] Completing task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
I1112 17:49:00.819233 28127 slave.cpp:3657] Cleaning up executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
I1112 17:49:00.819138 28118 hierarchical.cpp:230] Removed framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.819979 28124 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' for gc 6.99999051222222days in the future
I1112 17:49:00.820180 28124 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145' for gc 6.99999050987259days in the future
I1112 17:49:00.820304 28124 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' for gc 6.99999050741926days in the future
I1112 17:49:00.820425 28124 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145' for gc 6.99999050551111days in the future
I1112 17:49:00.820487 28127 slave.cpp:3745] Cleaning up framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.820775 28117 status_update_manager.cpp:284] Closing status update streams for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.820827 28117 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000' for gc 6.99999050024296days in the future
I1112 17:49:00.821035 28117 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000' for gc 6.99999049925926days in the future
I1112 17:49:00.821982 28124 slave.cpp:3215] master@172.17.6.239:32961 exited
W1112 17:49:00.822018 28124 slave.cpp:3218] Master disconnected! Waiting for a new master to be elected
I1112 17:49:00.829666 28097 slave.cpp:601] Slave terminating
[ FAILED ] PersistentVolumeTest.SlaveRecovery (508 ms)
{noformat}
was:
Looks like the executor didn't re-register in time after slave restart. The Clock::settle() should've guaranteed that the executor sent re-register and slave processed it, so it's not clear why this race happened.
{code}
I1112 17:49:00.663300 28118 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
I1112 17:49:00.663388 28118 slave.cpp:400] Slave attributes: [ ]
I1112 17:49:00.663406 28118 slave.cpp:405] Slave hostname: f4f9d084d7ba
I1112 17:49:00.663419 28118 slave.cpp:410] Slave checkpoint: true
I1112 17:49:00.669028 28115 state.cpp:54] Recovering state from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta'
I1112 17:49:00.675704 28121 fetcher.cpp:79] Clearing fetcher cache
I1112 17:49:00.675799 28121 slave.cpp:4318] Recovering framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.675891 28121 slave.cpp:5108] Recovering executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.677217 28119 status_update_manager.cpp:202] Recovering status update manager
I1112 17:49:00.677250 28119 status_update_manager.cpp:210] Recovering executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.677326 28119 status_update_manager.cpp:499] Creating StatusUpdate stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.677896 28119 status_update_manager.cpp:802] Replaying status update stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
I1112 17:49:00.678973 28121 slave.cpp:682] Successfully attached file '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
I1112 17:49:00.679440 28121 containerizer.cpp:385] Recovering containerizer
I1112 17:49:00.679527 28121 containerizer.cpp:440] Recovering container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' for executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.685341 28115 slave.cpp:4170] Sending reconnect request to executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
I1112 17:49:00.690004 1529 exec.cpp:256] Received reconnect request from slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
I1112 17:49:00.703845 28122 hierarchical.cpp:977] No resources available to allocate!
I1112 17:49:00.703917 28122 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:49:00.704025 28122 hierarchical.cpp:876] Performed allocation for 1 slaves in 271350ns
I1112 17:49:00.703974 28116 slave.cpp:2702] Cleaning up un-reregistered executors
I1112 17:49:00.704107 28116 slave.cpp:2720] Killing un-reregistered executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
I1112 17:49:00.704262 28116 slave.cpp:4230] Finished recovery
I1112 17:49:00.704397 28123 containerizer.cpp:1075] Destroying container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
I1112 17:49:00.706696 28116 slave.cpp:4387] Querying resource estimator for oversubscribable resources
I1112 17:49:00.707142 28127 slave.cpp:4401] Received oversubscribable resources from the resource estimator
W1112 17:49:00.707540 28127 slave.cpp:2546] Shutting down executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 because the slave is not in recovery mode
I1112 17:49:00.708133 28116 slave.cpp:729] New master detected at master@172.17.6.239:32961
I1112 17:49:00.708228 28116 slave.cpp:792] Authenticating with master master@172.17.6.239:32961
I1112 17:49:00.708253 28116 slave.cpp:797] Using default CRAM-MD5 authenticatee
I1112 17:49:00.708421 28116 slave.cpp:765] Detecting new master
I1112 17:49:00.708533 28116 status_update_manager.cpp:176] Pausing sending status updates
I1112 17:49:00.708673 28116 authenticatee.cpp:123] Creating new client SASL connection
I1112 17:49:00.709913 28116 master.cpp:5150] Authenticating slave(197)@172.17.6.239:32961
I1112 17:49:00.710218 28116 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(467)@172.17.6.239:32961
I1112 17:49:00.710438 1516 exec.cpp:383] Executor asked to shutdown
I1112 17:49:00.710582 1516 exec.cpp:398] Executor::shutdown took 35481ns
I1112 17:49:00.710598 28116 authenticator.cpp:100] Creating new server SASL connection
I1112 17:49:00.710705 1516 exec.cpp:82] Scheduling shutdown of the executor
Shutting down
Sending SIGTERM to process tree at pid 1531
I1112 17:49:00.710830 28116 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
I1112 17:49:00.710866 28116 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
I1112 17:49:00.712005 28116 authenticator.cpp:205] Received SASL authentication start
I1112 17:49:00.712069 28116 authenticator.cpp:327] Authentication requires more steps
I1112 17:49:00.712214 28124 authenticatee.cpp:260] Received SASL authentication step
I1112 17:49:00.712357 28116 authenticator.cpp:233] Received SASL authentication step
I1112 17:49:00.712389 28116 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1112 17:49:00.712412 28116 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I1112 17:49:00.712467 28116 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1112 17:49:00.712507 28116 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1112 17:49:00.712524 28116 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1112 17:49:00.712534 28116 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1112 17:49:00.712555 28116 authenticator.cpp:319] Authentication success
I1112 17:49:00.712708 28128 authenticatee.cpp:300] Authentication success
I1112 17:49:00.721004 28122 master.cpp:5180] Successfully authenticated principal 'test-principal' at slave(197)@172.17.6.239:32961
I1112 17:49:00.721352 28117 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(467)@172.17.6.239:32961
Killing the following process trees:
[
--- 1531 sh -c while true; do test -d path1; done
]
I1112 17:49:00.721968 28128 slave.cpp:860] Successfully authenticated with master master@172.17.6.239:32961
I1112 17:49:00.723456 28130 master.cpp:4019] Re-registering slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.724056 28126 hierarchical.cpp:417] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 reactivated
I1112 17:49:00.723914 28130 master.cpp:4207] Sending updated checkpointed resources disk(role1)[id1:path1]:64 to slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.724306 28128 slave.cpp:1254] Will retry registration in 17.384714ms if necessary
I1112 17:49:00.726639 28128 slave.cpp:1004] Re-registered with master master@172.17.6.239:32961
I1112 17:49:00.726775 28122 master.cpp:3479] Telling slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba) to kill task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
I1112 17:49:00.726794 28127 status_update_manager.cpp:183] Resuming sending status updates
I1112 17:49:00.726913 28128 slave.cpp:1040] Forwarding total oversubscribed resources
I1112 17:49:00.727146 28126 master.cpp:4269] Received update of slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba) with total oversubscribed resources
I1112 17:49:00.727447 28126 hierarchical.cpp:400] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 (f4f9d084d7ba) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64)
I1112 17:49:00.728248 28126 hierarchical.cpp:1162] Filtered offer with cpus(*):1; mem(*):896; ports(*):[31000-32000]; disk(role1):928 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.728301 28126 hierarchical.cpp:977] No resources available to allocate!
I1112 17:49:00.728350 28126 hierarchical.cpp:1070] No inverse offers to send out!
I1112 17:49:00.728374 28126 hierarchical.cpp:892] Performed allocation for slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 in 865336ns
I1112 17:49:00.728529 28128 slave.cpp:2178] Updating framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 pid to scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
I1112 17:49:00.728587 28128 slave.cpp:2194] Checkpointing framework pid 'scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961' to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/framework.pid'
I1112 17:49:00.729202 28119 status_update_manager.cpp:183] Resuming sending status updates
I1112 17:49:00.729709 28128 slave.cpp:2274] Updated checkpointed resources from disk(role1)[id1:path1]:64 to disk(role1)[id1:path1]:64
I1112 17:49:00.729821 28128 slave.cpp:1821] Asked to kill task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
W1112 17:49:00.729884 28128 slave.cpp:1948] Ignoring kill task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 because the executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865 is terminating/terminated
I1112 17:49:00.804975 28117 containerizer.cpp:1258] Executor for container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' has exited
I1112 17:49:00.807514 28117 slave.cpp:3553] Executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 terminated with signal Killed
I1112 17:49:00.807701 28117 slave.cpp:2762] Handling status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from @0.0.0.0:0
I1112 17:49:00.807884 28117 slave.cpp:5298] Terminating task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
W1112 17:49:00.808377 28117 containerizer.cpp:967] Ignoring update for unknown container: b9dbc84e-57fa-49d5-840f-8f6d9f0233fc
I1112 17:49:00.808811 28117 status_update_manager.cpp:322] Received status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.808878 28117 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.814139 28117 status_update_manager.cpp:376] Forwarding update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to the slave
I1112 17:49:00.814621 28122 slave.cpp:3087] Forwarding the update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to master@172.17.6.239:32961
I1112 17:49:00.814888 28122 slave.cpp:3005] Status update manager successfully handled status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.815172 28117 master.cpp:4414] Status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.815218 28117 master.cpp:4462] Forwarding status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.815402 28117 master.cpp:6066] Updating the state of task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (latest state: TASK_LOST, status update state: TASK_LOST)
I1112 17:49:00.815636 28122 sched.cpp:921] Scheduler::statusUpdate took 135083ns
I1112 17:49:00.816069 28117 master.cpp:3571] Processing ACKNOWLEDGE call 0c911b27-4aa4-4435-965d-1b4b5033a40c for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
I1112 17:49:00.816118 28122 hierarchical.cpp:744] Recovered cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: ) on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 from framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.816128 28117 master.cpp:6132] Removing task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba)
I1112 17:49:00.816721 28127 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
I1112 17:49:00.817196 28127 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
../../src/tests/persistent_volume_tests.cpp:709: Failure
Value of: status2.get().state()
Actual: TASK_LOST
Expected: TASK_KILLED
{code}
> PersistentVolumeTest.SlaveRecovery test is flaky
> ------------------------------------------------
>
> Key: MESOS-3907
> URL: https://issues.apache.org/jira/browse/MESOS-3907
> Project: Mesos
> Issue Type: Bug
> Environment: ASF CI
> Reporter: Vinod Kone
>
> Looks like the executor didn't re-register in time after slave restart. The Clock::settle() should've guaranteed that the executor sent re-register and slave processed it, so it's not clear why this race happened.
> {noformat: title=Good Run}
> [ RUN ] PersistentVolumeTest.SlaveRecovery
> I1112 17:53:29.746028 29676 leveldb.cpp:176] Opened db in 2.430372ms
> I1112 17:53:29.746974 29676 leveldb.cpp:183] Compacted db in 832726ns
> I1112 17:53:29.747094 29676 leveldb.cpp:198] Created db iterator in 24074ns
> I1112 17:53:29.747210 29676 leveldb.cpp:204] Seeked to beginning of db in 2533ns
> I1112 17:53:29.747329 29676 leveldb.cpp:273] Iterated through 0 keys in the db in 786ns
> I1112 17:53:29.747462 29676 replica.cpp:780] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
> I1112 17:53:29.748114 29699 recover.cpp:449] Starting replica recovery
> I1112 17:53:29.748440 29696 recover.cpp:475] Replica is in EMPTY status
> I1112 17:53:29.749609 29708 replica.cpp:676] Replica in EMPTY status received a broadcasted recover request from (5641)@172.17.6.242:48157
> I1112 17:53:29.750352 29702 recover.cpp:195] Received a recover response from a replica in EMPTY status
> I1112 17:53:29.750880 29707 recover.cpp:566] Updating replica status to STARTING
> I1112 17:53:29.751597 29705 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 418265ns
> I1112 17:53:29.751699 29705 replica.cpp:323] Persisted replica status to STARTING
> I1112 17:53:29.752048 29710 recover.cpp:475] Replica is in STARTING status
> I1112 17:53:29.752780 29710 replica.cpp:676] Replica in STARTING status received a broadcasted recover request from (5642)@172.17.6.242:48157
> I1112 17:53:29.753223 29705 recover.cpp:195] Received a recover response from a replica in STARTING status
> I1112 17:53:29.753618 29702 recover.cpp:566] Updating replica status to VOTING
> I1112 17:53:29.754117 29696 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 357564ns
> I1112 17:53:29.754144 29696 replica.cpp:323] Persisted replica status to VOTING
> I1112 17:53:29.754236 29696 recover.cpp:580] Successfully joined the Paxos group
> I1112 17:53:29.754380 29696 recover.cpp:464] Recover process terminated
> I1112 17:53:29.755972 29705 master.cpp:367] Master 4104c1dc-cb09-41a0-8f89-339ad511ce2a (bf058f59fe00) started on 172.17.6.242:48157
> I1112 17:53:29.755993 29705 master.cpp:369] Flags at startup: --acls="register_frameworks {
> principals {
> values: "test-principal"
> }
> roles {
> values: "role1"
> }
> }
> " --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/JRCPBA/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --roles="role1" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.26.0/_inst/share/mesos/webui" --work_dir="/tmp/JRCPBA/master" --zk_session_timeout="10secs"
> I1112 17:53:29.756337 29705 master.cpp:414] Master only allowing authenticated frameworks to register
> I1112 17:53:29.756350 29705 master.cpp:419] Master only allowing authenticated slaves to register
> I1112 17:53:29.756358 29705 credentials.hpp:37] Loading credentials for authentication from '/tmp/JRCPBA/credentials'
> I1112 17:53:29.756618 29705 master.cpp:458] Using default 'crammd5' authenticator
> I1112 17:53:29.756753 29705 master.cpp:495] Authorization enabled
> I1112 17:53:29.756985 29706 whitelist_watcher.cpp:79] No whitelist given
> I1112 17:53:29.757154 29698 hierarchical.cpp:151] Initialized hierarchical allocator process
> I1112 17:53:29.759389 29700 master.cpp:1606] The newly elected leader is master@172.17.6.242:48157 with id 4104c1dc-cb09-41a0-8f89-339ad511ce2a
> I1112 17:53:29.759423 29700 master.cpp:1619] Elected as the leading master!
> I1112 17:53:29.759443 29700 master.cpp:1379] Recovering from registrar
> I1112 17:53:29.759692 29704 registrar.cpp:309] Recovering registrar
> I1112 17:53:29.760234 29699 log.cpp:661] Attempting to start the writer
> I1112 17:53:29.761632 29700 replica.cpp:496] Replica received implicit promise request from (5643)@172.17.6.242:48157 with proposal 1
> I1112 17:53:29.762080 29700 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 412844ns
> I1112 17:53:29.762109 29700 replica.cpp:345] Persisted promised to 1
> I1112 17:53:29.762683 29700 coordinator.cpp:240] Coordinator attempting to fill missing positions
> I1112 17:53:29.763840 29697 replica.cpp:391] Replica received explicit promise request from (5644)@172.17.6.242:48157 for position 0 with proposal 2
> I1112 17:53:29.764291 29697 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 407157ns
> I1112 17:53:29.764317 29697 replica.cpp:715] Persisted action at 0
> I1112 17:53:29.765326 29697 replica.cpp:540] Replica received write request for position 0 from (5645)@172.17.6.242:48157
> I1112 17:53:29.765386 29697 leveldb.cpp:438] Reading position from leveldb took 28779ns
> I1112 17:53:29.765704 29697 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 273113ns
> I1112 17:53:29.765730 29697 replica.cpp:715] Persisted action at 0
> I1112 17:53:29.766624 29699 replica.cpp:694] Replica received learned notice for position 0 from @0.0.0.0:0
> I1112 17:53:29.766934 29699 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 268658ns
> I1112 17:53:29.766963 29699 replica.cpp:715] Persisted action at 0
> I1112 17:53:29.766983 29699 replica.cpp:700] Replica learned NOP action at position 0
> I1112 17:53:29.767499 29699 log.cpp:677] Writer started with ending position 0
> I1112 17:53:29.768618 29703 leveldb.cpp:438] Reading position from leveldb took 25919ns
> I1112 17:53:29.769791 29703 registrar.cpp:342] Successfully fetched the registry (0B) in 10.054144ms
> I1112 17:53:29.769908 29703 registrar.cpp:441] Applied 1 operations in 29537ns; attempting to update the 'registry'
> I1112 17:53:29.771086 29699 log.cpp:685] Attempting to append 176 bytes to the log
> I1112 17:53:29.771188 29710 coordinator.cpp:350] Coordinator attempting to write APPEND action at position 1
> I1112 17:53:29.772028 29700 replica.cpp:540] Replica received write request for position 1 from (5646)@172.17.6.242:48157
> I1112 17:53:29.772368 29700 leveldb.cpp:343] Persisting action (195 bytes) to leveldb took 303124ns
> I1112 17:53:29.772455 29700 replica.cpp:715] Persisted action at 1
> I1112 17:53:29.773016 29698 replica.cpp:694] Replica received learned notice for position 1 from @0.0.0.0:0
> I1112 17:53:29.773453 29698 leveldb.cpp:343] Persisting action (197 bytes) to leveldb took 405803ns
> I1112 17:53:29.773538 29698 replica.cpp:715] Persisted action at 1
> I1112 17:53:29.773633 29698 replica.cpp:700] Replica learned APPEND action at position 1
> I1112 17:53:29.774744 29702 registrar.cpp:486] Successfully updated the 'registry' in 4.68096ms
> I1112 17:53:29.774878 29702 registrar.cpp:372] Successfully recovered registrar
> I1112 17:53:29.775008 29708 log.cpp:704] Attempting to truncate the log to 1
> I1112 17:53:29.775240 29705 master.cpp:1416] Recovered 0 slaves from the Registry (137B) ; allowing 10mins for slaves to re-register
> I1112 17:53:29.775346 29702 coordinator.cpp:350] Coordinator attempting to write TRUNCATE action at position 2
> I1112 17:53:29.776243 29695 replica.cpp:540] Replica received write request for position 2 from (5647)@172.17.6.242:48157
> I1112 17:53:29.776782 29695 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 458517ns
> I1112 17:53:29.776813 29695 replica.cpp:715] Persisted action at 2
> I1112 17:53:29.777896 29699 replica.cpp:694] Replica received learned notice for position 2 from @0.0.0.0:0
> I1112 17:53:29.778386 29699 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 321333ns
> I1112 17:53:29.778445 29699 leveldb.cpp:401] Deleting ~1 keys from leveldb took 29781ns
> I1112 17:53:29.778466 29699 replica.cpp:715] Persisted action at 2
> I1112 17:53:29.778493 29699 replica.cpp:700] Replica learned TRUNCATE action at position 2
> I1112 17:53:29.789324 29676 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix
> W1112 17:53:29.789729 29676 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges
> I1112 17:53:29.802047 29705 slave.cpp:191] Slave started on 196)@172.17.6.242:48157
> I1112 17:53:29.802078 29705 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):1024" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK"
> I1112 17:53:29.802577 29705 credentials.hpp:85] Loading credential for authentication from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/credential'
> I1112 17:53:29.802806 29705 slave.cpp:322] Slave using credential for: test-principal
> I1112 17:53:29.803010 29705 resources.cpp:474] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):1024
> Trying semicolon-delimited string format instead
> I1112 17:53:29.803546 29705 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
> I1112 17:53:29.803680 29705 slave.cpp:400] Slave attributes: [ ]
> I1112 17:53:29.803779 29705 slave.cpp:405] Slave hostname: bf058f59fe00
> I1112 17:53:29.803850 29705 slave.cpp:410] Slave checkpoint: true
> I1112 17:53:29.805106 29699 state.cpp:54] Recovering state from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta'
> I1112 17:53:29.805594 29699 status_update_manager.cpp:202] Recovering status update manager
> I1112 17:53:29.805768 29699 containerizer.cpp:385] Recovering containerizer
> I1112 17:53:29.806855 29676 sched.cpp:166] Version: 0.26.0
> I1112 17:53:29.807466 29698 slave.cpp:4230] Finished recovery
> I1112 17:53:29.808089 29698 slave.cpp:4387] Querying resource estimator for oversubscribable resources
> I1112 17:53:29.808365 29697 sched.cpp:264] New master detected at master@172.17.6.242:48157
> I1112 17:53:29.808424 29698 slave.cpp:729] New master detected at master@172.17.6.242:48157
> I1112 17:53:29.808478 29697 sched.cpp:320] Authenticating with master master@172.17.6.242:48157
> I1112 17:53:29.808497 29697 sched.cpp:327] Using default CRAM-MD5 authenticatee
> I1112 17:53:29.808511 29703 status_update_manager.cpp:176] Pausing sending status updates
> I1112 17:53:29.808593 29698 slave.cpp:792] Authenticating with master master@172.17.6.242:48157
> I1112 17:53:29.808616 29698 slave.cpp:797] Using default CRAM-MD5 authenticatee
> I1112 17:53:29.808734 29698 slave.cpp:765] Detecting new master
> I1112 17:53:29.808794 29704 authenticatee.cpp:123] Creating new client SASL connection
> I1112 17:53:29.809075 29695 master.cpp:5150] Authenticating scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
> I1112 17:53:29.809202 29707 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(467)@172.17.6.242:48157
> I1112 17:53:29.809305 29697 authenticatee.cpp:123] Creating new client SASL connection
> I1112 17:53:29.809438 29706 authenticator.cpp:100] Creating new server SASL connection
> I1112 17:53:29.809612 29697 master.cpp:5150] Authenticating slave(196)@172.17.6.242:48157
> I1112 17:53:29.809723 29695 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(468)@172.17.6.242:48157
> I1112 17:53:29.809921 29706 authenticator.cpp:100] Creating new server SASL connection
> I1112 17:53:29.810148 29706 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
> I1112 17:53:29.810184 29706 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
> I1112 17:53:29.810276 29706 authenticator.cpp:205] Received SASL authentication start
> I1112 17:53:29.810340 29706 authenticator.cpp:327] Authentication requires more steps
> I1112 17:53:29.810426 29706 authenticatee.cpp:260] Received SASL authentication step
> I1112 17:53:29.810525 29706 authenticator.cpp:233] Received SASL authentication step
> I1112 17:53:29.810556 29706 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
> I1112 17:53:29.810570 29706 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
> I1112 17:53:29.810621 29706 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
> I1112 17:53:29.810654 29706 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
> I1112 17:53:29.810667 29706 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I1112 17:53:29.810675 29706 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I1112 17:53:29.810693 29706 authenticator.cpp:319] Authentication success
> I1112 17:53:29.810796 29695 authenticatee.cpp:300] Authentication success
> I1112 17:53:29.810864 29706 master.cpp:5180] Successfully authenticated principal 'test-principal' at slave(196)@172.17.6.242:48157
> I1112 17:53:29.810920 29695 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(468)@172.17.6.242:48157
> I1112 17:53:29.809806 29709 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
> I1112 17:53:29.810995 29709 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
> I1112 17:53:29.811110 29701 authenticator.cpp:205] Received SASL authentication start
> I1112 17:53:29.811131 29698 slave.cpp:4401] Received oversubscribable resources from the resource estimator
> I1112 17:53:29.811182 29701 authenticator.cpp:327] Authentication requires more steps
> I1112 17:53:29.811270 29701 authenticatee.cpp:260] Received SASL authentication step
> I1112 17:53:29.811373 29698 slave.cpp:860] Successfully authenticated with master master@172.17.6.242:48157
> I1112 17:53:29.811406 29701 authenticator.cpp:233] Received SASL authentication step
> I1112 17:53:29.811435 29701 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
> I1112 17:53:29.811449 29701 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
> I1112 17:53:29.811491 29701 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
> I1112 17:53:29.811522 29701 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
> I1112 17:53:29.811514 29698 slave.cpp:1254] Will retry registration in 4.148387ms if necessary
> I1112 17:53:29.811537 29701 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I1112 17:53:29.811548 29701 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I1112 17:53:29.811566 29701 authenticator.cpp:319] Authentication success
> I1112 17:53:29.811722 29701 authenticatee.cpp:300] Authentication success
> I1112 17:53:29.811812 29701 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(467)@172.17.6.242:48157
> I1112 17:53:29.811847 29698 master.cpp:3859] Registering slave at slave(196)@172.17.6.242:48157 (bf058f59fe00) with id 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
> I1112 17:53:29.812162 29698 master.cpp:5180] Successfully authenticated principal 'test-principal' at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
> I1112 17:53:29.812445 29706 registrar.cpp:441] Applied 1 operations in 89470ns; attempting to update the 'registry'
> I1112 17:53:29.813341 29710 sched.cpp:409] Successfully authenticated with master master@172.17.6.242:48157
> I1112 17:53:29.813371 29710 sched.cpp:716] Sending SUBSCRIBE call to master@172.17.6.242:48157
> I1112 17:53:29.813485 29710 sched.cpp:749] Will retry registration in 1.306688873secs if necessary
> I1112 17:53:29.813665 29710 master.cpp:2176] Received SUBSCRIBE call for framework 'default' at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
> I1112 17:53:29.813711 29698 log.cpp:685] Attempting to append 349 bytes to the log
> I1112 17:53:29.813731 29710 master.cpp:1645] Authorizing framework principal 'test-principal' to receive offers for role 'role1'
> I1112 17:53:29.813869 29710 coordinator.cpp:350] Coordinator attempting to write APPEND action at position 3
> I1112 17:53:29.814548 29707 master.cpp:2247] Subscribing framework default with checkpointing enabled and capabilities [ ]
> I1112 17:53:29.814918 29708 replica.cpp:540] Replica received write request for position 3 from (5659)@172.17.6.242:48157
> I1112 17:53:29.814998 29703 hierarchical.cpp:195] Added framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:29.815057 29703 hierarchical.cpp:977] No resources available to allocate!
> I1112 17:53:29.815091 29703 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:53:29.815115 29703 hierarchical.cpp:876] Performed allocation for 0 slaves in 96149ns
> I1112 17:53:29.815232 29703 sched.cpp:643] Framework registered with 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:29.815317 29703 sched.cpp:657] Scheduler::registered took 60505ns
> I1112 17:53:29.815325 29708 leveldb.cpp:343] Persisting action (368 bytes) to leveldb took 351340ns
> I1112 17:53:29.815349 29708 replica.cpp:715] Persisted action at 3
> I1112 17:53:29.817284 29708 slave.cpp:1254] Will retry registration in 17.295409ms if necessary
> I1112 17:53:29.817334 29705 replica.cpp:694] Replica received learned notice for position 3 from @0.0.0.0:0
> I1112 17:53:29.817575 29708 master.cpp:3847] Ignoring register slave message from slave(196)@172.17.6.242:48157 (bf058f59fe00) as admission is already in progress
> I1112 17:53:29.817993 29705 leveldb.cpp:343] Persisting action (370 bytes) to leveldb took 631697ns
> I1112 17:53:29.818019 29705 replica.cpp:715] Persisted action at 3
> I1112 17:53:29.818042 29705 replica.cpp:700] Replica learned APPEND action at position 3
> I1112 17:53:29.819962 29705 registrar.cpp:486] Successfully updated the 'registry' in 7.428864ms
> I1112 17:53:29.819972 29708 log.cpp:704] Attempting to truncate the log to 3
> I1112 17:53:29.821169 29704 slave.cpp:3169] Received ping from slave-observer(198)@172.17.6.242:48157
> I1112 17:53:29.821400 29701 hierarchical.cpp:344] Added slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 (bf058f59fe00) with cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000] (allocated: )
> I1112 17:53:29.822106 29704 coordinator.cpp:350] Coordinator attempting to write TRUNCATE action at position 4
> I1112 17:53:29.822491 29701 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:53:29.822597 29701 hierarchical.cpp:892] Performed allocation for slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 in 1.075053ms
> I1112 17:53:29.823355 29701 replica.cpp:540] Replica received write request for position 4 from (5660)@172.17.6.242:48157
> I1112 17:53:29.823503 29705 master.cpp:3927] Registered slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00) with cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
> I1112 17:53:29.823626 29707 slave.cpp:904] Registered with master master@172.17.6.242:48157; given slave ID 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
> I1112 17:53:29.823667 29707 fetcher.cpp:79] Clearing fetcher cache
> I1112 17:53:29.823720 29701 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 323431ns
> I1112 17:53:29.823748 29701 replica.cpp:715] Persisted action at 4
> I1112 17:53:29.823990 29695 status_update_manager.cpp:183] Resuming sending status updates
> I1112 17:53:29.824004 29705 master.cpp:4979] Sending 1 offers to framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
> I1112 17:53:29.824141 29707 slave.cpp:927] Checkpointing SlaveInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/slave.info'
> I1112 17:53:29.824463 29705 sched.cpp:813] Scheduler::resourceOffers took 119643ns
> I1112 17:53:29.824638 29707 slave.cpp:963] Forwarding total oversubscribed resources
> I1112 17:53:29.825076 29695 master.cpp:4269] Received update of slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00) with total oversubscribed resources
> I1112 17:53:29.825426 29676 resources.cpp:474] Parsing resources as JSON failed: cpus:1;mem:128;disk(role1):32
> Trying semicolon-delimited string format instead
> I1112 17:53:29.825443 29695 hierarchical.cpp:400] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 (bf058f59fe00) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):1024)
> I1112 17:53:29.825541 29704 replica.cpp:694] Replica received learned notice for position 4 from @0.0.0.0:0
> I1112 17:53:29.826035 29695 hierarchical.cpp:977] No resources available to allocate!
> I1112 17:53:29.826062 29704 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 489313ns
> I1112 17:53:29.826095 29695 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:53:29.826120 29695 hierarchical.cpp:892] Performed allocation for slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 in 621170ns
> I1112 17:53:29.826128 29704 leveldb.cpp:401] Deleting ~2 keys from leveldb took 41561ns
> I1112 17:53:29.826148 29704 replica.cpp:715] Persisted action at 4
> I1112 17:53:29.826169 29704 replica.cpp:700] Replica learned TRUNCATE action at position 4
> I1112 17:53:29.827690 29701 master.cpp:2915] Processing ACCEPT call for offers: [ 4104c1dc-cb09-41a0-8f89-339ad511ce2a-O0 ] on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00) for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
> I1112 17:53:29.827743 29701 master.cpp:2711] Authorizing framework principal 'test-principal' to launch task 128c936c-abbb-4925-ae2f-170e01e62865 as user 'mesos'
> I1112 17:53:29.829272 29709 master.cpp:3112] Applying CREATE operation for volumes disk(role1)[id1:path1]:64 from framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157 to slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:29.829699 29709 master.cpp:6205] Sending checkpointed resources disk(role1)[id1:path1]:64 to slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:29.830781 29709 master.hpp:176] Adding task 128c936c-abbb-4925-ae2f-170e01e62865 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 (bf058f59fe00)
> I1112 17:53:29.831727 29700 hierarchical.cpp:513] Updated allocation of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 from cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):1024 to cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):960; disk(role1)[id1:path1]:64
> I1112 17:53:29.831910 29706 slave.cpp:2274] Updated checkpointed resources from to disk(role1)[id1:path1]:64
> I1112 17:53:29.832084 29709 master.cpp:3245] Launching task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:29.832653 29709 slave.cpp:1294] Got assigned task 128c936c-abbb-4925-ae2f-170e01e62865 for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:29.832810 29709 slave.cpp:4920] Checkpointing FrameworkInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/framework.info'
> I1112 17:53:29.833256 29709 slave.cpp:4931] Checkpointing framework pid 'scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157' to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/framework.pid'
> I1112 17:53:29.833261 29700 hierarchical.cpp:744] Recovered cpus(*):1; mem(*):896; ports(*):[31000-32000]; disk(role1):928 (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64) on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 from framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:29.833344 29700 hierarchical.cpp:781] Framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 filtered slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 for 5secs
> I1112 17:53:29.833658 29709 resources.cpp:474] Parsing resources as JSON failed: cpus:0.1;mem:32
> Trying semicolon-delimited string format instead
> I1112 17:53:29.834321 29709 slave.cpp:1410] Launching task 128c936c-abbb-4925-ae2f-170e01e62865 for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:29.834424 29709 resources.cpp:474] Parsing resources as JSON failed: cpus:0.1;mem:32
> Trying semicolon-delimited string format instead
> I1112 17:53:29.835336 29709 paths.cpp:436] Trying to chown '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805' to user 'mesos'
> I1112 17:53:29.844714 29709 slave.cpp:5360] Checkpointing ExecutorInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/executor.info'
> I1112 17:53:29.845645 29709 slave.cpp:4999] Launching executor 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
> I1112 17:53:29.846499 29704 containerizer.cpp:619] Starting container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805' for executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework '4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000'
> I1112 17:53:29.846513 29709 slave.cpp:5383] Checkpointing TaskInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/tasks/128c936c-abbb-4925-ae2f-170e01e62865/task.info'
> I1112 17:53:29.847183 29709 slave.cpp:1628] Queuing task '128c936c-abbb-4925-ae2f-170e01e62865' for executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:29.847409 29709 slave.cpp:682] Successfully attached file '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
> I1112 17:53:29.847451 29706 posix.cpp:210] Changing the ownership of the persistent volume at '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/volumes/roles/role1/id1' with uid 1000 and gid 1000
> I1112 17:53:29.856493 29706 posix.cpp:254] Adding symlink from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/volumes/roles/role1/id1' to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/path1' for persistent volume disk(role1)[id1:path1]:64 of container 4726aa7f-ee7c-41ed-a49f-f722e2d1c805
> I1112 17:53:29.861801 29696 launcher.cpp:134] Forked child with pid '3079' for container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
> I1112 17:53:29.862185 29696 containerizer.cpp:852] Checkpointing executor's forked pid 3079 to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/pids/forked.pid'
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1112 17:53:30.023684 3093 process.cpp:960] libprocess is initialized on 172.17.6.242:51169 for 16 cpus
> I1112 17:53:30.024690 3093 logging.cpp:177] Logging to STDERR
> I1112 17:53:30.027457 3093 exec.cpp:136] Version: 0.26.0
> I1112 17:53:30.040992 3120 exec.cpp:186] Executor started at: executor(1)@172.17.6.242:51169 with pid 3093
> I1112 17:53:30.045091 29703 slave.cpp:2405] Got registration for executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from executor(1)@172.17.6.242:51169
> I1112 17:53:30.045225 29703 slave.cpp:2490] Checkpointing executor pid 'executor(1)@172.17.6.242:51169' to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/pids/libprocess.pid'
> I1112 17:53:30.048463 3111 exec.cpp:210] Executor registered on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
> I1112 17:53:30.048741 29703 slave.cpp:1793] Sending queued task '128c936c-abbb-4925-ae2f-170e01e62865' to executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 at executor(1)@172.17.6.242:51169
> I1112 17:53:30.051081 3111 exec.cpp:222] Executor::registered took 414331ns
> Registered executor on bf058f59fe00
> I1112 17:53:30.053513 3119 exec.cpp:297] Executor asked to run task '128c936c-abbb-4925-ae2f-170e01e62865'
> I1112 17:53:30.053709 3119 exec.cpp:306] Executor::launchTask took 137969ns
> Starting task 128c936c-abbb-4925-ae2f-170e01e62865
> Forked command at 3124
> sh -c 'while true; do test -d path1; done'
> I1112 17:53:30.058800 3119 exec.cpp:519] Executor sending status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.061161 29699 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from executor(1)@172.17.6.242:51169
> I1112 17:53:30.061717 29699 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.061764 29699 status_update_manager.cpp:499] Creating StatusUpdate stream for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.063155 29699 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.064308 29699 status_update_manager.cpp:376] Forwarding update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to the slave
> I1112 17:53:30.064816 29699 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to master@172.17.6.242:48157
> I1112 17:53:30.065068 29699 slave.cpp:3005] Status update manager successfully handled status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.065956 29699 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to executor(1)@172.17.6.242:51169
> I1112 17:53:30.065420 29707 master.cpp:4414] Status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:30.066629 29707 master.cpp:4462] Forwarding status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.066886 29707 master.cpp:6066] Updating the state of task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
> I1112 17:53:30.067373 29702 sched.cpp:921] Scheduler::statusUpdate took 159288ns
> I1112 17:53:30.067679 29702 master.cpp:3571] Processing ACKNOWLEDGE call 2591ad57-0df4-4e31-b50f-7454091654d8 for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
> I1112 17:53:30.068071 29702 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.068222 29702 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.068727 3116 exec.cpp:343] Executor received status update acknowledgement 2591ad57-0df4-4e31-b50f-7454091654d8 for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.072254 29702 slave.cpp:2345] Status update manager successfully handled status update acknowledgement (UUID: 2591ad57-0df4-4e31-b50f-7454091654d8) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.073086 29676 slave.cpp:601] Slave terminating
> I1112 17:53:30.074015 29702 master.cpp:1083] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00) disconnected
> I1112 17:53:30.074048 29702 master.cpp:2531] Disconnecting slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:30.074110 29702 master.cpp:2550] Deactivating slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:30.074353 29697 hierarchical.cpp:429] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 deactivated
> I1112 17:53:30.092569 29676 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix
> W1112 17:53:30.093389 29676 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges
> I1112 17:53:30.099161 29706 slave.cpp:191] Slave started on 197)@172.17.6.242:48157
> I1112 17:53:30.099205 29706 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):1024" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK"
> I1112 17:53:30.099730 29706 credentials.hpp:85] Loading credential for authentication from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/credential'
> I1112 17:53:30.100016 29706 slave.cpp:322] Slave using credential for: test-principal
> I1112 17:53:30.100208 29706 resources.cpp:474] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):1024
> Trying semicolon-delimited string format instead
> I1112 17:53:30.100651 29706 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
> I1112 17:53:30.100720 29706 slave.cpp:400] Slave attributes: [ ]
> I1112 17:53:30.100733 29706 slave.cpp:405] Slave hostname: bf058f59fe00
> I1112 17:53:30.100742 29706 slave.cpp:410] Slave checkpoint: true
> I1112 17:53:30.102144 29710 state.cpp:54] Recovering state from '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta'
> I1112 17:53:30.108124 29698 fetcher.cpp:79] Clearing fetcher cache
> I1112 17:53:30.108225 29698 slave.cpp:4318] Recovering framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.108312 29698 slave.cpp:5108] Recovering executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.109978 29698 status_update_manager.cpp:202] Recovering status update manager
> I1112 17:53:30.110011 29698 status_update_manager.cpp:210] Recovering executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.110088 29698 status_update_manager.cpp:499] Creating StatusUpdate stream for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.110942 29703 slave.cpp:682] Successfully attached file '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
> I1112 17:53:30.111120 29698 status_update_manager.cpp:802] Replaying status update stream for task 128c936c-abbb-4925-ae2f-170e01e62865
> I1112 17:53:30.112169 29698 containerizer.cpp:385] Recovering containerizer
> I1112 17:53:30.112259 29698 containerizer.cpp:440] Recovering container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805' for executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.114814 29706 slave.cpp:4170] Sending reconnect request to executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 at executor(1)@172.17.6.242:51169
> I1112 17:53:30.117032 3116 exec.cpp:256] Received reconnect request from slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
> I1112 17:53:30.117979 29695 slave.cpp:2553] Re-registering executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.119094 29700 posix.cpp:210] Changing the ownership of the persistent volume at '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/volumes/roles/role1/id1' with uid 1000 and gid 1000
> I1112 17:53:30.123585 3116 exec.cpp:233] Executor re-registered on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
> I1112 17:53:30.125902 3116 exec.cpp:245] Executor::reregistered took 110185ns
> Re-registered executor on bf058f59fe00
> I1112 17:53:30.145524 29700 hierarchical.cpp:977] No resources available to allocate!
> I1112 17:53:30.145620 29700 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:53:30.145637 29709 slave.cpp:2702] Cleaning up un-reregistered executors
> I1112 17:53:30.145656 29700 hierarchical.cpp:876] Performed allocation for 1 slaves in 239099ns
> I1112 17:53:30.145795 29709 slave.cpp:4230] Finished recovery
> I1112 17:53:30.146611 29709 slave.cpp:4387] Querying resource estimator for oversubscribable resources
> I1112 17:53:30.147100 29709 slave.cpp:729] New master detected at master@172.17.6.242:48157
> I1112 17:53:30.147147 29700 status_update_manager.cpp:176] Pausing sending status updates
> I1112 17:53:30.147213 29709 slave.cpp:792] Authenticating with master master@172.17.6.242:48157
> I1112 17:53:30.147239 29709 slave.cpp:797] Using default CRAM-MD5 authenticatee
> I1112 17:53:30.147446 29709 slave.cpp:765] Detecting new master
> I1112 17:53:30.147581 29709 slave.cpp:4401] Received oversubscribable resources from the resource estimator
> I1112 17:53:30.147833 29709 authenticatee.cpp:123] Creating new client SASL connection
> I1112 17:53:30.148392 29709 master.cpp:5150] Authenticating slave(197)@172.17.6.242:48157
> I1112 17:53:30.148526 29707 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(469)@172.17.6.242:48157
> I1112 17:53:30.148771 29708 authenticator.cpp:100] Creating new server SASL connection
> I1112 17:53:30.149075 29704 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
> I1112 17:53:30.149114 29704 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
> I1112 17:53:30.149261 29708 authenticator.cpp:205] Received SASL authentication start
> I1112 17:53:30.149334 29708 authenticator.cpp:327] Authentication requires more steps
> I1112 17:53:30.149437 29708 authenticatee.cpp:260] Received SASL authentication step
> I1112 17:53:30.149559 29708 authenticator.cpp:233] Received SASL authentication step
> I1112 17:53:30.149593 29708 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
> I1112 17:53:30.149611 29708 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
> I1112 17:53:30.149673 29708 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
> I1112 17:53:30.149718 29708 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'bf058f59fe00' server FQDN: 'bf058f59fe00' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
> I1112 17:53:30.149735 29708 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I1112 17:53:30.149747 29708 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I1112 17:53:30.149770 29708 authenticator.cpp:319] Authentication success
> I1112 17:53:30.149871 29707 authenticatee.cpp:300] Authentication success
> I1112 17:53:30.149989 29708 master.cpp:5180] Successfully authenticated principal 'test-principal' at slave(197)@172.17.6.242:48157
> I1112 17:53:30.150027 29700 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(469)@172.17.6.242:48157
> I1112 17:53:30.152010 29707 slave.cpp:860] Successfully authenticated with master master@172.17.6.242:48157
> I1112 17:53:30.152513 29707 slave.cpp:1254] Will retry registration in 17.988166ms if necessary
> I1112 17:53:30.153123 29700 master.cpp:4019] Re-registering slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(196)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:30.153628 29700 master.cpp:4207] Sending updated checkpointed resources disk(role1)[id1:path1]:64 to slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:30.153990 29700 slave.cpp:1004] Re-registered with master master@172.17.6.242:48157
> I1112 17:53:30.154044 29707 hierarchical.cpp:417] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 reactivated
> I1112 17:53:30.154106 29700 slave.cpp:1040] Forwarding total oversubscribed resources
> I1112 17:53:30.154258 29699 status_update_manager.cpp:183] Resuming sending status updates
> I1112 17:53:30.154536 29700 slave.cpp:2178] Updating framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 pid to scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
> I1112 17:53:30.154603 29700 slave.cpp:2194] Checkpointing framework pid 'scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157' to '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/framework.pid'
> I1112 17:53:30.154841 29710 master.cpp:4269] Received update of slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00) with total oversubscribed resources
> I1112 17:53:30.155228 29708 status_update_manager.cpp:183] Resuming sending status updates
> I1112 17:53:30.155268 29710 hierarchical.cpp:400] Slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 (bf058f59fe00) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64)
> I1112 17:53:30.155864 29709 master.cpp:3479] Telling slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00) to kill task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157
> I1112 17:53:30.155908 29700 slave.cpp:2274] Updated checkpointed resources from disk(role1)[id1:path1]:64 to disk(role1)[id1:path1]:64
> I1112 17:53:30.156059 29700 slave.cpp:1821] Asked to kill task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.156313 29710 hierarchical.cpp:1162] Filtered offer with cpus(*):1; mem(*):896; ports(*):[31000-32000]; disk(role1):928 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.156425 29710 hierarchical.cpp:977] No resources available to allocate!
> I1112 17:53:30.156582 29710 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:53:30.156693 29710 hierarchical.cpp:892] Performed allocation for slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 in 1.366876ms
> I1112 17:53:30.158217 3108 exec.cpp:317] Executor asked to kill task '128c936c-abbb-4925-ae2f-170e01e62865'
> I1112 17:53:30.158285 3108 exec.cpp:326] Executor::killTask took 29545ns
> Shutting down
> Sending SIGTERM to process tree at pid 3124
> Killing the following process trees:
> [
> --- 3124 sh -c while true; do test -d path1; done
> ]
> Command terminated with signal Terminated (pid: 3124)
> I1112 17:53:30.261500 3114 exec.cpp:519] Executor sending status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.263711 29709 slave.cpp:2762] Handling status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from executor(1)@172.17.6.242:51169
> I1112 17:53:30.263947 29709 slave.cpp:5298] Terminating task 128c936c-abbb-4925-ae2f-170e01e62865
> I1112 17:53:30.264845 29709 posix.cpp:156] Removing symlink '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805/path1' for persistent volume disk(role1)[id1:path1]:64 of container 4726aa7f-ee7c-41ed-a49f-f722e2d1c805
> I1112 17:53:30.266062 29699 status_update_manager.cpp:322] Received status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.266145 29699 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.268108 29699 status_update_manager.cpp:376] Forwarding update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to the slave
> I1112 17:53:30.268626 29699 slave.cpp:3087] Forwarding the update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to master@172.17.6.242:48157
> I1112 17:53:30.268796 29699 slave.cpp:3005] Status update manager successfully handled status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.268828 29699 slave.cpp:3011] Sending acknowledgement for status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 to executor(1)@172.17.6.242:51169
> I1112 17:53:30.269331 29700 master.cpp:4414] Status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 from slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:30.269423 29700 master.cpp:4462] Forwarding status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.269665 29700 master.cpp:6066] Updating the state of task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
> II1112 17:53:30.270424 29700 sched.cpp:921] Scheduler::statusUpdate took 130601ns
> 1112 17:53:30.270457 3114 exec.cpp:343] Executor received status update acknowledgement 01d489fd-62de-4853-9c2e-0dea985eb730 for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.270642 29703 hierarchical.cpp:744] Recovered cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: ) on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 from framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.270889 29703 master.cpp:3571] Processing ACKNOWLEDGE call 01d489fd-62de-4853-9c2e-0dea985eb730 for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 (default) at scheduler-d3c3b08a-4b7e-4706-9a01-f4d2e511dba1@172.17.6.242:48157 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
> I1112 17:53:30.270972 29703 master.cpp:6132] Removing task 128c936c-abbb-4925-ae2f-170e01e62865 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 on slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0 at slave(197)@172.17.6.242:48157 (bf058f59fe00)
> I1112 17:53:30.271466 29707 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.271620 29707 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_KILLED (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.271903 29676 sched.cpp:1805] Asked to stop the driver
> I1112 17:53:30.272076 29710 master.cpp:922] Master terminating
> I1112 17:53:30.272251 29700 sched.cpp:1043] Stopping framework '4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000'
> I1112 17:53:30.272675 29707 status_update_manager.cpp:530] Cleaning up status update stream for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.272737 29695 hierarchical.cpp:373] Removed slave 4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0
> I1112 17:53:30.273172 29706 slave.cpp:2345] Status update manager successfully handled status update acknowledgement (UUID: 01d489fd-62de-4853-9c2e-0dea985eb730) for task 128c936c-abbb-4925-ae2f-170e01e62865 of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.273232 29706 slave.cpp:5339] Completing task 128c936c-abbb-4925-ae2f-170e01e62865
> I1112 17:53:30.276796 29710 hierarchical.cpp:230] Removed framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.276867 29710 slave.cpp:3215] master@172.17.6.242:48157 exited
> W1112 17:53:30.277004 29710 slave.cpp:3218] Master disconnected! Waiting for a new master to be elected
> I1112 17:53:30.284138 29699 containerizer.cpp:1075] Destroying container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805'
> I1112 17:53:30.350635 29699 containerizer.cpp:1258] Executor for container '4726aa7f-ee7c-41ed-a49f-f722e2d1c805' has exited
> I1112 17:53:30.353777 29705 slave.cpp:3553] Executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 terminated with signal Killed
> I1112 17:53:30.353916 29705 slave.cpp:3657] Cleaning up executor '128c936c-abbb-4925-ae2f-170e01e62865' of framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000 at executor(1)@172.17.6.242:51169
> I1112 17:53:30.354750 29697 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805' for gc 6.99999589540741days in the future
> I1112 17:53:30.355023 29697 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865' for gc 6.99999589335704days in the future
> I1112 17:53:30.356132 29706 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865/runs/4726aa7f-ee7c-41ed-a49f-f722e2d1c805' for gc 6.99999588955852days in the future
> I1112 17:53:30.356284 29706 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000/executors/128c936c-abbb-4925-ae2f-170e01e62865' for gc 6.99999587701333days in the future
> I1112 17:53:30.356299 29705 slave.cpp:3745] Cleaning up framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.356453 29706 status_update_manager.cpp:284] Closing status update streams for framework 4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000
> I1112 17:53:30.356506 29706 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000' for gc 6.99999587491852days in the future
> I1112 17:53:30.356564 29705 slave.cpp:601] Slave terminating
> I1112 17:53:30.356593 29706 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_nEafzK/meta/slaves/4104c1dc-cb09-41a0-8f89-339ad511ce2a-S0/frameworks/4104c1dc-cb09-41a0-8f89-339ad511ce2a-0000' for gc 6.99999587395852days in the future
> [ OK ] PersistentVolumeTest.SlaveRecovery (620 ms)
> {noformat}
> {noformat: title=Bad Run}
> [ RUN ] PersistentVolumeTest.SlaveRecovery
> I1112 17:49:00.333355 28097 leveldb.cpp:176] Opened db in 2.811376ms
> I1112 17:49:00.336050 28097 leveldb.cpp:183] Compacted db in 2.655859ms
> I1112 17:49:00.336125 28097 leveldb.cpp:198] Created db iterator in 34982ns
> I1112 17:49:00.336144 28097 leveldb.cpp:204] Seeked to beginning of db in 8700ns
> I1112 17:49:00.336156 28097 leveldb.cpp:273] Iterated through 0 keys in the db in 6732ns
> I1112 17:49:00.336205 28097 replica.cpp:780] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
> I1112 17:49:00.337661 28127 recover.cpp:449] Starting replica recovery
> I1112 17:49:00.338086 28127 recover.cpp:475] Replica is in EMPTY status
> I1112 17:49:00.340411 28130 replica.cpp:676] Replica in EMPTY status received a broadcasted recover request from (5611)@172.17.6.239:32961
> I1112 17:49:00.340880 28130 recover.cpp:195] Received a recover response from a replica in EMPTY status
> I1112 17:49:00.341501 28130 recover.cpp:566] Updating replica status to STARTING
> I1112 17:49:00.342439 28121 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 724899ns
> I1112 17:49:00.342470 28121 replica.cpp:323] Persisted replica status to STARTING
> I1112 17:49:00.342743 28124 recover.cpp:475] Replica is in STARTING status
> I1112 17:49:00.345824 28120 master.cpp:367] Master 5b272a7d-2df2-4bb6-b65f-e2633bea9888 (f4f9d084d7ba) started on 172.17.6.239:32961
> I1112 17:49:00.345859 28120 master.cpp:369] Flags at startup: --acls="register_frameworks {
> principals {
> values: "test-principal"
> }
> roles {
> values: "role1"
> }
> }
> " --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/XaPAlx/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="25secs" --registry_strict="true" --roles="role1" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.26.0/_inst/share/mesos/webui" --work_dir="/tmp/XaPAlx/master" --zk_session_timeout="10secs"
> I1112 17:49:00.346254 28120 master.cpp:414] Master only allowing authenticated frameworks to register
> I1112 17:49:00.346266 28120 master.cpp:419] Master only allowing authenticated slaves to register
> I1112 17:49:00.346274 28120 credentials.hpp:37] Loading credentials for authentication from '/tmp/XaPAlx/credentials'
> I1112 17:49:00.346552 28120 master.cpp:458] Using default 'crammd5' authenticator
> I1112 17:49:00.346678 28120 master.cpp:495] Authorization enabled
> I1112 17:49:00.346774 28127 replica.cpp:676] Replica in STARTING status received a broadcasted recover request from (5612)@172.17.6.239:32961
> I1112 17:49:00.347045 28127 hierarchical.cpp:151] Initialized hierarchical allocator process
> I1112 17:49:00.347195 28127 recover.cpp:195] Received a recover response from a replica in STARTING status
> I1112 17:49:00.347899 28125 recover.cpp:566] Updating replica status to VOTING
> I1112 17:49:00.347997 28119 whitelist_watcher.cpp:79] No whitelist given
> I1112 17:49:00.348471 28119 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 438348ns
> I1112 17:49:00.348497 28119 replica.cpp:323] Persisted replica status to VOTING
> I1112 17:49:00.348598 28125 recover.cpp:580] Successfully joined the Paxos group
> I1112 17:49:00.349103 28125 recover.cpp:464] Recover process terminated
> I1112 17:49:00.350262 28128 master.cpp:1606] The newly elected leader is master@172.17.6.239:32961 with id 5b272a7d-2df2-4bb6-b65f-e2633bea9888
> I1112 17:49:00.350395 28128 master.cpp:1619] Elected as the leading master!
> I1112 17:49:00.350510 28128 master.cpp:1379] Recovering from registrar
> I1112 17:49:00.350709 28120 registrar.cpp:309] Recovering registrar
> I1112 17:49:00.351351 28124 log.cpp:661] Attempting to start the writer
> I1112 17:49:00.354029 28124 replica.cpp:496] Replica received implicit promise request from (5613)@172.17.6.239:32961 with proposal 1
> I1112 17:49:00.354639 28124 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 560313ns
> I1112 17:49:00.354667 28124 replica.cpp:345] Persisted promised to 1
> I1112 17:49:00.355598 28124 coordinator.cpp:240] Coordinator attempting to fill missing positions
> I1112 17:49:00.357206 28124 replica.cpp:391] Replica received explicit promise request from (5614)@172.17.6.239:32961 for position 0 with proposal 2
> I1112 17:49:00.357781 28124 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 478236ns
> I1112 17:49:00.357807 28124 replica.cpp:715] Persisted action at 0
> I1112 17:49:00.359031 28123 replica.cpp:540] Replica received write request for position 0 from (5615)@172.17.6.239:32961
> I1112 17:49:00.359105 28123 leveldb.cpp:438] Reading position from leveldb took 45943ns
> I1112 17:49:00.359474 28123 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 328418ns
> I1112 17:49:00.359503 28123 replica.cpp:715] Persisted action at 0
> I1112 17:49:00.360296 28127 replica.cpp:694] Replica received learned notice for position 0 from @0.0.0.0:0
> I1112 17:49:00.360959 28127 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 642538ns
> I1112 17:49:00.360986 28127 replica.cpp:715] Persisted action at 0
> I1112 17:49:00.361006 28127 replica.cpp:700] Replica learned NOP action at position 0
> I1112 17:49:00.361683 28129 log.cpp:677] Writer started with ending position 0
> I1112 17:49:00.363149 28122 leveldb.cpp:438] Reading position from leveldb took 85104ns
> I1112 17:49:00.364382 28116 registrar.cpp:342] Successfully fetched the registry (0B) in 13.476864ms
> I1112 17:49:00.364588 28116 registrar.cpp:441] Applied 1 operations in 44922ns; attempting to update the 'registry'
> I1112 17:49:00.365350 28116 log.cpp:685] Attempting to append 176 bytes to the log
> I1112 17:49:00.365546 28127 coordinator.cpp:350] Coordinator attempting to write APPEND action at position 1
> I1112 17:49:00.366474 28115 replica.cpp:540] Replica received write request for position 1 from (5616)@172.17.6.239:32961
> I1112 17:49:00.367040 28115 leveldb.cpp:343] Persisting action (195 bytes) to leveldb took 537014ns
> I1112 17:49:00.367069 28115 replica.cpp:715] Persisted action at 1
> I1112 17:49:00.367697 28126 replica.cpp:694] Replica received learned notice for position 1 from @0.0.0.0:0
> I1112 17:49:00.368448 28126 leveldb.cpp:343] Persisting action (197 bytes) to leveldb took 732384ns
> I1112 17:49:00.368476 28126 replica.cpp:715] Persisted action at 1
> I1112 17:49:00.368499 28126 replica.cpp:700] Replica learned APPEND action at position 1
> I1112 17:49:00.370054 28126 registrar.cpp:486] Successfully updated the 'registry' in 5.339904ms
> I1112 17:49:00.370199 28126 registrar.cpp:372] Successfully recovered registrar
> I1112 17:49:00.370453 28126 log.cpp:704] Attempting to truncate the log to 1
> I1112 17:49:00.370818 28126 master.cpp:1416] Recovered 0 slaves from the Registry (137B) ; allowing 10mins for slaves to re-register
> I1112 17:49:00.370919 28126 coordinator.cpp:350] Coordinator attempting to write TRUNCATE action at position 2
> I1112 17:49:00.371949 28120 replica.cpp:540] Replica received write request for position 2 from (5617)@172.17.6.239:32961
> I1112 17:49:00.372375 28120 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 398089ns
> I1112 17:49:00.372402 28120 replica.cpp:715] Persisted action at 2
> I1112 17:49:00.373205 28121 replica.cpp:694] Replica received learned notice for position 2 from @0.0.0.0:0
> I1112 17:49:00.373517 28121 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 297514ns
> I1112 17:49:00.373572 28121 leveldb.cpp:401] Deleting ~1 keys from leveldb took 40498ns
> I1112 17:49:00.373594 28121 replica.cpp:715] Persisted action at 2
> I1112 17:49:00.373615 28121 replica.cpp:700] Replica learned TRUNCATE action at position 2
> I1112 17:49:00.385612 28097 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix
> W1112 17:49:00.386059 28097 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges
> I1112 17:49:00.390331 28128 slave.cpp:191] Slave started on 196)@172.17.6.239:32961
> I1112 17:49:00.390383 28128 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):1024" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj"
> I1112 17:49:00.390844 28128 credentials.hpp:85] Loading credential for authentication from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/credential'
> I1112 17:49:00.391086 28128 slave.cpp:322] Slave using credential for: test-principal
> I1112 17:49:00.391239 28128 resources.cpp:474] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):1024
> Trying semicolon-delimited string format instead
> I1112 17:49:00.391666 28128 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
> I1112 17:49:00.391737 28128 slave.cpp:400] Slave attributes: [ ]
> I1112 17:49:00.391751 28128 slave.cpp:405] Slave hostname: f4f9d084d7ba
> I1112 17:49:00.391758 28128 slave.cpp:410] Slave checkpoint: true
> I1112 17:49:00.393064 28121 state.cpp:54] Recovering state from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta'
> I1112 17:49:00.393465 28121 status_update_manager.cpp:202] Recovering status update manager
> I1112 17:49:00.393633 28121 containerizer.cpp:385] Recovering containerizer
> I1112 17:49:00.393672 28097 sched.cpp:166] Version: 0.26.0
> I1112 17:49:00.394464 28118 sched.cpp:264] New master detected at master@172.17.6.239:32961
> I1112 17:49:00.394554 28118 sched.cpp:320] Authenticating with master master@172.17.6.239:32961
> I1112 17:49:00.394579 28118 sched.cpp:327] Using default CRAM-MD5 authenticatee
> I1112 17:49:00.394716 28115 slave.cpp:4230] Finished recovery
> I1112 17:49:00.394938 28118 authenticatee.cpp:123] Creating new client SASL connection
> I1112 17:49:00.395225 28125 master.cpp:5150] Authenticating scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
> I1112 17:49:00.395256 28115 slave.cpp:4387] Querying resource estimator for oversubscribable resources
> I1112 17:49:00.395397 28125 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(465)@172.17.6.239:32961
> I1112 17:49:00.395648 28115 slave.cpp:729] New master detected at master@172.17.6.239:32961
> I1112 17:49:00.395697 28125 authenticator.cpp:100] Creating new server SASL connection
> I1112 17:49:00.395706 28115 slave.cpp:792] Authenticating with master master@172.17.6.239:32961
> I1112 17:49:00.395723 28115 slave.cpp:797] Using default CRAM-MD5 authenticatee
> I1112 17:49:00.395828 28115 slave.cpp:765] Detecting new master
> I1112 17:49:00.395895 28125 status_update_manager.cpp:176] Pausing sending status updates
> I1112 17:49:00.395947 28115 slave.cpp:4401] Received oversubscribable resources from the resource estimator
> I1112 17:49:00.395998 28125 authenticatee.cpp:123] Creating new client SASL connection
> I1112 17:49:00.396046 28115 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
> I1112 17:49:00.396080 28115 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
> I1112 17:49:00.396199 28125 master.cpp:5150] Authenticating slave(196)@172.17.6.239:32961
> I1112 17:49:00.396230 28115 authenticator.cpp:205] Received SASL authentication start
> I1112 17:49:00.396289 28115 authenticator.cpp:327] Authentication requires more steps
> I1112 17:49:00.396329 28125 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(466)@172.17.6.239:32961
> I1112 17:49:00.396358 28115 authenticatee.cpp:260] Received SASL authentication step
> I1112 17:49:00.396493 28115 authenticator.cpp:100] Creating new server SASL connection
> I1112 17:49:00.396683 28115 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
> I1112 17:49:00.396714 28115 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
> I1112 17:49:00.396781 28115 authenticator.cpp:205] Received SASL authentication start
> I1112 17:49:00.396822 28115 authenticator.cpp:327] Authentication requires more steps
> I1112 17:49:00.396891 28115 authenticatee.cpp:260] Received SASL authentication step
> I1112 17:49:00.396970 28125 authenticator.cpp:233] Received SASL authentication step
> I1112 17:49:00.396999 28125 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
> I1112 17:49:00.397011 28125 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
> I1112 17:49:00.397058 28125 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
> I1112 17:49:00.397085 28125 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
> I1112 17:49:00.397096 28125 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I1112 17:49:00.397104 28125 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I1112 17:49:00.397119 28125 authenticator.cpp:319] Authentication success
> I1112 17:49:00.397202 28115 authenticator.cpp:233] Received SASL authentication step
> I1112 17:49:00.397230 28115 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
> I1112 17:49:00.397241 28115 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
> I1112 17:49:00.397267 28115 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
> I1112 17:49:00.397289 28115 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
> I1112 17:49:00.397300 28115 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I1112 17:49:00.397307 28115 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I1112 17:49:00.397321 28115 authenticator.cpp:319] Authentication success
> I1112 17:49:00.397457 28115 authenticatee.cpp:300] Authentication success
> I1112 17:49:00.397538 28115 master.cpp:5180] Successfully authenticated principal 'test-principal' at slave(196)@172.17.6.239:32961
> I1112 17:49:00.397610 28115 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(466)@172.17.6.239:32961
> I1112 17:49:00.397871 28115 slave.cpp:860] Successfully authenticated with master master@172.17.6.239:32961
> I1112 17:49:00.398057 28125 authenticatee.cpp:300] Authentication success
> I1112 17:49:00.398133 28125 master.cpp:5180] Successfully authenticated principal 'test-principal' at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
> I1112 17:49:00.398200 28125 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(465)@172.17.6.239:32961
> I1112 17:49:00.398424 28125 sched.cpp:409] Successfully authenticated with master master@172.17.6.239:32961
> I1112 17:49:00.398447 28125 sched.cpp:716] Sending SUBSCRIBE call to master@172.17.6.239:32961
> I1112 17:49:00.398531 28125 sched.cpp:749] Will retry registration in 281.865576ms if necessary
> I1112 17:49:00.398694 28125 master.cpp:2176] Received SUBSCRIBE call for framework 'default' at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
> I1112 17:49:00.398718 28125 master.cpp:1645] Authorizing framework principal 'test-principal' to receive offers for role 'role1'
> I1112 17:49:00.399085 28115 slave.cpp:1254] Will retry registration in 13.451406ms if necessary
> I1112 17:49:00.400010 28122 master.cpp:2247] Subscribing framework default with checkpointing enabled and capabilities [ ]
> I1112 17:49:00.400645 28122 master.cpp:3859] Registering slave at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) with id 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
> I1112 17:49:00.401038 28122 hierarchical.cpp:195] Added framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.401165 28122 hierarchical.cpp:977] No resources available to allocate!
> I1112 17:49:00.401298 28122 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:49:00.401407 28122 hierarchical.cpp:876] Performed allocation for 0 slaves in 290991ns
> I1112 17:49:00.401165 28124 sched.cpp:643] Framework registered with 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.401517 28130 registrar.cpp:441] Applied 1 operations in 91430ns; attempting to update the 'registry'
> I1112 17:49:00.401669 28124 sched.cpp:657] Scheduler::registered took 75312ns
> I1112 17:49:00.402467 28118 log.cpp:685] Attempting to append 349 bytes to the log
> I1112 17:49:00.402775 28118 coordinator.cpp:350] Coordinator attempting to write APPEND action at position 3
> I1112 17:49:00.403746 28118 replica.cpp:540] Replica received write request for position 3 from (5629)@172.17.6.239:32961
> I1112 17:49:00.404013 28118 leveldb.cpp:343] Persisting action (368 bytes) to leveldb took 236794ns
> I1112 17:49:00.404042 28118 replica.cpp:715] Persisted action at 3
> I1112 17:49:00.404981 28115 replica.cpp:694] Replica received learned notice for position 3 from @0.0.0.0:0
> I1112 17:49:00.405659 28115 leveldb.cpp:343] Persisting action (370 bytes) to leveldb took 639583ns
> I1112 17:49:00.405690 28115 replica.cpp:715] Persisted action at 3
> I1112 17:49:00.405716 28115 replica.cpp:700] Replica learned APPEND action at position 3
> I1112 17:49:00.408030 28115 registrar.cpp:486] Successfully updated the 'registry' in 6.365952ms
> I1112 17:49:00.409021 28130 log.cpp:704] Attempting to truncate the log to 3
> I1112 17:49:00.409071 28124 slave.cpp:3169] Received ping from slave-observer(197)@172.17.6.239:32961
> I1112 17:49:00.409194 28124 coordinator.cpp:350] Coordinator attempting to write TRUNCATE action at position 4
> I1112 17:49:00.409203 28121 hierarchical.cpp:344] Added slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 (f4f9d084d7ba) with cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000] (allocated: )
> I1112 17:49:00.409489 28115 master.cpp:3927] Registered slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) with cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
> I1112 17:49:00.409637 28115 slave.cpp:904] Registered with master master@172.17.6.239:32961; given slave ID 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
> I1112 17:49:00.409657 28115 fetcher.cpp:79] Clearing fetcher cache
> I1112 17:49:00.409976 28130 status_update_manager.cpp:183] Resuming sending status updates
> I1112 17:49:00.410019 28121 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:49:00.410056 28121 hierarchical.cpp:892] Performed allocation for slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 in 807103ns
> I1112 17:49:00.410110 28115 slave.cpp:927] Checkpointing SlaveInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/slave.info'
> I1112 17:49:00.410204 28121 replica.cpp:540] Replica received write request for position 4 from (5630)@172.17.6.239:32961
> I1112 17:49:00.410537 28130 master.cpp:4979] Sending 1 offers to framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
> I1112 17:49:00.410555 28115 slave.cpp:963] Forwarding total oversubscribed resources
> I1112 17:49:00.410729 28121 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 502328ns
> I1112 17:49:00.410755 28121 replica.cpp:715] Persisted action at 4
> I1112 17:49:00.410876 28115 master.cpp:4269] Received update of slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) with total oversubscribed resources
> I1112 17:49:00.411187 28115 hierarchical.cpp:400] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 (f4f9d084d7ba) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):1024)
> I1112 17:49:00.411337 28130 sched.cpp:813] Scheduler::resourceOffers took 448533ns
> I1112 17:49:00.411788 28115 hierarchical.cpp:977] No resources available to allocate!
> I1112 17:49:00.411802 28097 resources.cpp:474] Parsing resources as JSON failed: cpus:1;mem:128;disk(role1):32
> Trying semicolon-delimited string format instead
> I1112 17:49:00.411922 28115 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:49:00.412221 28123 replica.cpp:694] Replica received learned notice for position 4 from @0.0.0.0:0
> I1112 17:49:00.412698 28115 hierarchical.cpp:892] Performed allocation for slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 in 1.374186ms
> I1112 17:49:00.413398 28123 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 713860ns
> I1112 17:49:00.413476 28123 leveldb.cpp:401] Deleting ~2 keys from leveldb took 57815ns
> I1112 17:49:00.413503 28123 replica.cpp:715] Persisted action at 4
> I1112 17:49:00.413532 28123 replica.cpp:700] Replica learned TRUNCATE action at position 4
> I1112 17:49:00.415221 28123 master.cpp:2915] Processing ACCEPT call for offers: [ 5b272a7d-2df2-4bb6-b65f-e2633bea9888-O0 ] on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
> I1112 17:49:00.415282 28123 master.cpp:2711] Authorizing framework principal 'test-principal' to launch task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 as user 'mesos'
> I1112 17:49:00.416872 28123 master.cpp:3112] Applying CREATE operation for volumes disk(role1)[id1:path1]:64 from framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961 to slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.417266 28123 master.cpp:6205] Sending checkpointed resources disk(role1)[id1:path1]:64 to slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.418344 28123 master.hpp:176] Adding task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 (f4f9d084d7ba)
> I1112 17:49:00.418550 28123 master.cpp:3245] Launching task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.422189 28121 slave.cpp:2274] Updated checkpointed resources from to disk(role1)[id1:path1]:64
> I1112 17:49:00.422458 28121 slave.cpp:1294] Got assigned task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.422623 28121 slave.cpp:4920] Checkpointing FrameworkInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/framework.info'
> I1112 17:49:00.423600 28123 hierarchical.cpp:513] Updated allocation of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 from cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):1024 to cpus(*):2; mem(*):1024; ports(*):[31000-32000]; disk(role1):960; disk(role1)[id1:path1]:64
> I1112 17:49:00.424105 28123 hierarchical.cpp:744] Recovered cpus(*):1; mem(*):896; ports(*):[31000-32000]; disk(role1):928 (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64) on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 from framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.424186 28123 hierarchical.cpp:781] Framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 filtered slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 for 5secs
> I1112 17:49:00.424402 28121 slave.cpp:4931] Checkpointing framework pid 'scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961' to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/framework.pid'
> I1112 17:49:00.424839 28121 resources.cpp:474] Parsing resources as JSON failed: cpus:0.1;mem:32
> Trying semicolon-delimited string format instead
> I1112 17:49:00.425565 28121 slave.cpp:1410] Launching task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.425664 28121 resources.cpp:474] Parsing resources as JSON failed: cpus:0.1;mem:32
> Trying semicolon-delimited string format instead
> I1112 17:49:00.426517 28121 paths.cpp:436] Trying to chown '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' to user 'mesos'
> I1112 17:49:00.437039 28121 slave.cpp:5360] Checkpointing ExecutorInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/executor.info'
> I1112 17:49:00.438006 28121 slave.cpp:4999] Launching executor 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
> I1112 17:49:00.438782 28121 slave.cpp:5383] Checkpointing TaskInfo to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc/tasks/2b35f3fc-373c-4b5f-bbe6-dc3279117145/task.info'
> I1112 17:49:00.439314 28124 containerizer.cpp:619] Starting container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' for executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework '5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000'
> I1112 17:49:00.439545 28121 slave.cpp:1628] Queuing task '2b35f3fc-373c-4b5f-bbe6-dc3279117145' for executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.439808 28121 slave.cpp:682] Successfully attached file '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
> I1112 17:49:00.440218 28119 posix.cpp:210] Changing the ownership of the persistent volume at '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/volumes/roles/role1/id1' with uid 1000 and gid 1000
> I1112 17:49:00.449080 28119 posix.cpp:254] Adding symlink from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/volumes/roles/role1/id1' to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc/path1' for persistent volume disk(role1)[id1:path1]:64 of container b9dbc84e-57fa-49d5-840f-8f6d9f0233fc
> I1112 17:49:00.453404 28122 launcher.cpp:134] Forked child with pid '1486' for container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
> I1112 17:49:00.453604 28122 containerizer.cpp:852] Checkpointing executor's forked pid 1486 to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc/pids/forked.pid'
> 2015-11-12 17:49:00,593:28097(0x2ad09360b700):ZOO_ERROR@handle_socket_error_msg@1697: Socket [127.0.0.1:44265] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I1112 17:49:00.604712 1500 process.cpp:960] libprocess is initialized on 172.17.6.239:46865 for 16 cpus
> I1112 17:49:00.605552 1500 logging.cpp:177] Logging to STDERR
> I1112 17:49:00.608021 1500 exec.cpp:136] Version: 0.26.0
> I1112 17:49:00.619716 1521 exec.cpp:186] Executor started at: executor(1)@172.17.6.239:46865 with pid 1500
> I1112 17:49:00.622460 28127 slave.cpp:2405] Got registration for executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from executor(1)@172.17.6.239:46865
> I1112 17:49:00.623016 28127 slave.cpp:2490] Checkpointing executor pid 'executor(1)@172.17.6.239:46865' to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc/pids/libprocess.pid'
> I1112 17:49:00.625865 28121 slave.cpp:1793] Sending queued task '2b35f3fc-373c-4b5f-bbe6-dc3279117145' to executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
> I1112 17:49:00.627167 1519 exec.cpp:210] Executor registered on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
> I1112 17:49:00.628980 1519 exec.cpp:222] Executor::registered took 358581ns
> Registered executor on f4f9d084d7ba
> I1112 17:49:00.631170 1525 exec.cpp:297] Executor asked to run task '2b35f3fc-373c-4b5f-bbe6-dc3279117145'
> I1112 17:49:00.631330 1525 exec.cpp:306] Executor::launchTask took 136825ns
> Starting task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
> Forked command at 1531
> sh -c 'while true; do test -d path1; done'
> I1112 17:49:00.635591 1526 exec.cpp:519] Executor sending status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.637117 28120 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from executor(1)@172.17.6.239:46865
> I1112 17:49:00.637634 28120 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.637676 28120 status_update_manager.cpp:499] Creating StatusUpdate stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.638445 28120 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.639673 28120 status_update_manager.cpp:376] Forwarding update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to the slave
> I1112 17:49:00.640115 28122 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to master@172.17.6.239:32961
> I1112 17:49:00.640601 28122 slave.cpp:3005] Status update manager successfully handled status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.640805 28129 master.cpp:4414] Status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.640857 28129 master.cpp:4462] Forwarding status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.640720 28122 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to executor(1)@172.17.6.239:46865
> I1112 17:49:00.641095 28129 master.cpp:6066] Updating the state of task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
> I1112 17:49:00.641374 28116 sched.cpp:921] Scheduler::statusUpdate took 141504ns
> I1112 17:49:00.642500 28128 master.cpp:3571] Processing ACKNOWLEDGE call 24a2b427-4799-4be0-9cb8-d5bf525b32c6 for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
> I1112 17:49:00.643096 28116 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.643277 28116 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.644330 28116 slave.cpp:2345] Status update manager successfully handled status update acknowledgement (UUID: 24a2b427-4799-4be0-9cb8-d5bf525b32c6) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.645215 28097 slave.cpp:601] Slave terminating
> I1112 17:49:00.646117 1517 exec.cpp:343] Executor received status update acknowledgement 24a2b427-4799-4be0-9cb8-d5bf525b32c6 for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.646978 28116 master.cpp:1083] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba) disconnected
> I1112 17:49:00.647253 28116 master.cpp:2531] Disconnecting slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.647650 28116 master.cpp:2550] Deactivating slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.648510 28124 hierarchical.cpp:429] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 deactivated
> I1112 17:49:00.652220 28097 containerizer.cpp:143] Using isolation: posix/cpu,posix/mem,filesystem/posix
> W1112 17:49:00.652614 28097 backend.cpp:50] Failed to create 'bind' backend: BindBackend requires root privileges
> I1112 17:49:00.658809 28118 slave.cpp:191] Slave started on 197)@172.17.6.239:32961
> I1112 17:49:00.658880 28118 slave.cpp:192] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" --docker_kill_orphans="true" --docker_local_archives_dir="/tmp/mesos/images/docker" --docker_puller="local" --docker_puller_timeout="60" --docker_registry="registry-1.docker.io" --docker_registry_port="443" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.26.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk(role1):1024" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj"
> I1112 17:49:00.660478 28118 credentials.hpp:85] Loading credential for authentication from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/credential'
> I1112 17:49:00.662098 28118 slave.cpp:322] Slave using credential for: test-principal
> I1112 17:49:00.662643 28118 resources.cpp:474] Parsing resources as JSON failed: cpus:2;mem:1024;disk(role1):1024
> Trying semicolon-delimited string format instead
> I1112 17:49:00.663300 28118 slave.cpp:392] Slave resources: cpus(*):2; mem(*):1024; disk(role1):1024; ports(*):[31000-32000]
> I1112 17:49:00.663388 28118 slave.cpp:400] Slave attributes: [ ]
> I1112 17:49:00.663406 28118 slave.cpp:405] Slave hostname: f4f9d084d7ba
> I1112 17:49:00.663419 28118 slave.cpp:410] Slave checkpoint: true
> I1112 17:49:00.669028 28115 state.cpp:54] Recovering state from '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta'
> I1112 17:49:00.675704 28121 fetcher.cpp:79] Clearing fetcher cache
> I1112 17:49:00.675799 28121 slave.cpp:4318] Recovering framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.675891 28121 slave.cpp:5108] Recovering executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.677217 28119 status_update_manager.cpp:202] Recovering status update manager
> I1112 17:49:00.677250 28119 status_update_manager.cpp:210] Recovering executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.677326 28119 status_update_manager.cpp:499] Creating StatusUpdate stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.677896 28119 status_update_manager.cpp:802] Replaying status update stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
> I1112 17:49:00.678973 28121 slave.cpp:682] Successfully attached file '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
> I1112 17:49:00.679440 28121 containerizer.cpp:385] Recovering containerizer
> I1112 17:49:00.679527 28121 containerizer.cpp:440] Recovering container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' for executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.685341 28115 slave.cpp:4170] Sending reconnect request to executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
> I1112 17:49:00.690004 1529 exec.cpp:256] Received reconnect request from slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
> I1112 17:49:00.703845 28122 hierarchical.cpp:977] No resources available to allocate!
> I1112 17:49:00.703917 28122 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:49:00.704025 28122 hierarchical.cpp:876] Performed allocation for 1 slaves in 271350ns
> I1112 17:49:00.703974 28116 slave.cpp:2702] Cleaning up un-reregistered executors
> I1112 17:49:00.704107 28116 slave.cpp:2720] Killing un-reregistered executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
> I1112 17:49:00.704262 28116 slave.cpp:4230] Finished recovery
> I1112 17:49:00.704397 28123 containerizer.cpp:1075] Destroying container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc'
> I1112 17:49:00.706696 28116 slave.cpp:4387] Querying resource estimator for oversubscribable resources
> I1112 17:49:00.707142 28127 slave.cpp:4401] Received oversubscribable resources from the resource estimator
> W1112 17:49:00.707540 28127 slave.cpp:2546] Shutting down executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 because the slave is not in recovery mode
> I1112 17:49:00.708133 28116 slave.cpp:729] New master detected at master@172.17.6.239:32961
> I1112 17:49:00.708228 28116 slave.cpp:792] Authenticating with master master@172.17.6.239:32961
> I1112 17:49:00.708253 28116 slave.cpp:797] Using default CRAM-MD5 authenticatee
> I1112 17:49:00.708421 28116 slave.cpp:765] Detecting new master
> I1112 17:49:00.708533 28116 status_update_manager.cpp:176] Pausing sending status updates
> I1112 17:49:00.708673 28116 authenticatee.cpp:123] Creating new client SASL connection
> I1112 17:49:00.709913 28116 master.cpp:5150] Authenticating slave(197)@172.17.6.239:32961
> I1112 17:49:00.710218 28116 authenticator.cpp:415] Starting authentication session for crammd5_authenticatee(467)@172.17.6.239:32961
> I1112 17:49:00.710438 1516 exec.cpp:383] Executor asked to shutdown
> I1112 17:49:00.710582 1516 exec.cpp:398] Executor::shutdown took 35481ns
> I1112 17:49:00.710598 28116 authenticator.cpp:100] Creating new server SASL connection
> I1112 17:49:00.710705 1516 exec.cpp:82] Scheduling shutdown of the executor
> Shutting down
> Sending SIGTERM to process tree at pid 1531
> I1112 17:49:00.710830 28116 authenticatee.cpp:214] Received SASL authentication mechanisms: CRAM-MD5
> I1112 17:49:00.710866 28116 authenticatee.cpp:240] Attempting to authenticate with mechanism 'CRAM-MD5'
> I1112 17:49:00.712005 28116 authenticator.cpp:205] Received SASL authentication start
> I1112 17:49:00.712069 28116 authenticator.cpp:327] Authentication requires more steps
> I1112 17:49:00.712214 28124 authenticatee.cpp:260] Received SASL authentication step
> I1112 17:49:00.712357 28116 authenticator.cpp:233] Received SASL authentication step
> I1112 17:49:00.712389 28116 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
> I1112 17:49:00.712412 28116 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
> I1112 17:49:00.712467 28116 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
> I1112 17:49:00.712507 28116 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'f4f9d084d7ba' server FQDN: 'f4f9d084d7ba' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
> I1112 17:49:00.712524 28116 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I1112 17:49:00.712534 28116 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I1112 17:49:00.712555 28116 authenticator.cpp:319] Authentication success
> I1112 17:49:00.712708 28128 authenticatee.cpp:300] Authentication success
> I1112 17:49:00.721004 28122 master.cpp:5180] Successfully authenticated principal 'test-principal' at slave(197)@172.17.6.239:32961
> I1112 17:49:00.721352 28117 authenticator.cpp:433] Authentication session cleanup for crammd5_authenticatee(467)@172.17.6.239:32961
> Killing the following process trees:
> [
> --- 1531 sh -c while true; do test -d path1; done
> ]
> I1112 17:49:00.721968 28128 slave.cpp:860] Successfully authenticated with master master@172.17.6.239:32961
> I1112 17:49:00.723456 28130 master.cpp:4019] Re-registering slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(196)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.724056 28126 hierarchical.cpp:417] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 reactivated
> I1112 17:49:00.723914 28130 master.cpp:4207] Sending updated checkpointed resources disk(role1)[id1:path1]:64 to slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.724306 28128 slave.cpp:1254] Will retry registration in 17.384714ms if necessary
> I1112 17:49:00.726639 28128 slave.cpp:1004] Re-registered with master master@172.17.6.239:32961
> I1112 17:49:00.726775 28122 master.cpp:3479] Telling slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba) to kill task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
> I1112 17:49:00.726794 28127 status_update_manager.cpp:183] Resuming sending status updates
> I1112 17:49:00.726913 28128 slave.cpp:1040] Forwarding total oversubscribed resources
> I1112 17:49:00.727146 28126 master.cpp:4269] Received update of slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba) with total oversubscribed resources
> I1112 17:49:00.727447 28126 hierarchical.cpp:400] Slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 (f4f9d084d7ba) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64)
> I1112 17:49:00.728248 28126 hierarchical.cpp:1162] Filtered offer with cpus(*):1; mem(*):896; ports(*):[31000-32000]; disk(role1):928 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.728301 28126 hierarchical.cpp:977] No resources available to allocate!
> I1112 17:49:00.728350 28126 hierarchical.cpp:1070] No inverse offers to send out!
> I1112 17:49:00.728374 28126 hierarchical.cpp:892] Performed allocation for slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 in 865336ns
> I1112 17:49:00.728529 28128 slave.cpp:2178] Updating framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 pid to scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961
> I1112 17:49:00.728587 28128 slave.cpp:2194] Checkpointing framework pid 'scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961' to '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/framework.pid'
> I1112 17:49:00.729202 28119 status_update_manager.cpp:183] Resuming sending status updates
> I1112 17:49:00.729709 28128 slave.cpp:2274] Updated checkpointed resources from disk(role1)[id1:path1]:64 to disk(role1)[id1:path1]:64
> I1112 17:49:00.729821 28128 slave.cpp:1821] Asked to kill task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> W1112 17:49:00.729884 28128 slave.cpp:1948] Ignoring kill task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 because the executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865 is terminating/terminated
> I1112 17:49:00.804975 28117 containerizer.cpp:1258] Executor for container 'b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' has exited
> I1112 17:49:00.807514 28117 slave.cpp:3553] Executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 terminated with signal Killed
> I1112 17:49:00.807701 28117 slave.cpp:2762] Handling status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from @0.0.0.0:0
> I1112 17:49:00.807884 28117 slave.cpp:5298] Terminating task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
> W1112 17:49:00.808377 28117 containerizer.cpp:967] Ignoring update for unknown container: b9dbc84e-57fa-49d5-840f-8f6d9f0233fc
> I1112 17:49:00.808811 28117 status_update_manager.cpp:322] Received status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.808878 28117 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.814139 28117 status_update_manager.cpp:376] Forwarding update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to the slave
> I1112 17:49:00.814621 28122 slave.cpp:3087] Forwarding the update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 to master@172.17.6.239:32961
> I1112 17:49:00.814888 28122 slave.cpp:3005] Status update manager successfully handled status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.815172 28117 master.cpp:4414] Status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 from slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.815218 28117 master.cpp:4462] Forwarding status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.815402 28117 master.cpp:6066] Updating the state of task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (latest state: TASK_LOST, status update state: TASK_LOST)
> I1112 17:49:00.815636 28122 sched.cpp:921] Scheduler::statusUpdate took 135083ns
> I1112 17:49:00.816069 28117 master.cpp:3571] Processing ACKNOWLEDGE call 0c911b27-4aa4-4435-965d-1b4b5033a40c for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 (default) at scheduler-7193507a-d32c-48fe-bfc2-1f594ff8a63a@172.17.6.239:32961 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
> I1112 17:49:00.816118 28122 hierarchical.cpp:744] Recovered cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 (total: cpus(*):2; mem(*):1024; disk(role1):960; ports(*):[31000-32000]; disk(role1)[id1:path1]:64, allocated: ) on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 from framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.816128 28117 master.cpp:6132] Removing task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 with resources cpus(*):1; mem(*):128; disk(role1):32; disk(role1)[id1:path1]:64 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 on slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0 at slave(197)@172.17.6.239:32961 (f4f9d084d7ba)
> I1112 17:49:00.816721 28127 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.817196 28127 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_LOST (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> ../../src/tests/persistent_volume_tests.cpp:709: Failure
> Value of: status2.get().state()
> Actual: TASK_LOST
> Expected: TASK_KILLED
> I1112 17:49:00.818148 28127 status_update_manager.cpp:530] Cleaning up status update stream for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.818176 28097 sched.cpp:1805] Asked to stop the driver
> I1112 17:49:00.818294 28097 master.cpp:922] Master terminating
> I1112 17:49:00.818581 28127 sched.cpp:1043] Stopping framework '5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000'
> I1112 17:49:00.818915 28127 slave.cpp:2345] Status update manager successfully handled status update acknowledgement (UUID: 0c911b27-4aa4-4435-965d-1b4b5033a40c) for task 2b35f3fc-373c-4b5f-bbe6-dc3279117145 of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.819017 28118 hierarchical.cpp:373] Removed slave 5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0
> I1112 17:49:00.819088 28127 slave.cpp:5339] Completing task 2b35f3fc-373c-4b5f-bbe6-dc3279117145
> I1112 17:49:00.819233 28127 slave.cpp:3657] Cleaning up executor '2b35f3fc-373c-4b5f-bbe6-dc3279117145' of framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000 at executor(1)@172.17.6.239:46865
> I1112 17:49:00.819138 28118 hierarchical.cpp:230] Removed framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.819979 28124 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' for gc 6.99999051222222days in the future
> I1112 17:49:00.820180 28124 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145' for gc 6.99999050987259days in the future
> I1112 17:49:00.820304 28124 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145/runs/b9dbc84e-57fa-49d5-840f-8f6d9f0233fc' for gc 6.99999050741926days in the future
> I1112 17:49:00.820425 28124 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000/executors/2b35f3fc-373c-4b5f-bbe6-dc3279117145' for gc 6.99999050551111days in the future
> I1112 17:49:00.820487 28127 slave.cpp:3745] Cleaning up framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.820775 28117 status_update_manager.cpp:284] Closing status update streams for framework 5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000
> I1112 17:49:00.820827 28117 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000' for gc 6.99999050024296days in the future
> I1112 17:49:00.821035 28117 gc.cpp:56] Scheduling '/tmp/PersistentVolumeTest_SlaveRecovery_LLnwwj/meta/slaves/5b272a7d-2df2-4bb6-b65f-e2633bea9888-S0/frameworks/5b272a7d-2df2-4bb6-b65f-e2633bea9888-0000' for gc 6.99999049925926days in the future
> I1112 17:49:00.821982 28124 slave.cpp:3215] master@172.17.6.239:32961 exited
> W1112 17:49:00.822018 28124 slave.cpp:3218] Master disconnected! Waiting for a new master to be elected
> I1112 17:49:00.829666 28097 slave.cpp:601] Slave terminating
> [ FAILED ] PersistentVolumeTest.SlaveRecovery (508 ms)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)