You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Till Toenshoff (JIRA)" <ji...@apache.org> on 2017/03/06 18:34:32 UTC
[jira] [Created] (MESOS-7213)
SlaveRecoveryTest/0.RecoverUnregisteredHTTPExecutor fails.
Till Toenshoff created MESOS-7213:
-------------------------------------
Summary: SlaveRecoveryTest/0.RecoverUnregisteredHTTPExecutor fails.
Key: MESOS-7213
URL: https://issues.apache.org/jira/browse/MESOS-7213
Project: Mesos
Issue Type: Bug
Components: tests
Affects Versions: 1.1.1
Environment: Debian 8, SSL/libevent build
Reporter: Till Toenshoff
The following happened while testing 1.1.1-rc2; may be flaky.
{noformat}
[ RUN ] SlaveRecoveryTest/0.RecoverUnregisteredHTTPExecutor
I0306 14:16:42.640406 27141 cluster.cpp:158] Creating default 'local' authorizer
I0306 14:16:42.648387 27141 leveldb.cpp:174] Opened db in 7.851169ms
I0306 14:16:42.649245 27141 leveldb.cpp:181] Compacted db in 832265ns
I0306 14:16:42.649266 27141 leveldb.cpp:196] Created db iterator in 4269ns
I0306 14:16:42.649271 27141 leveldb.cpp:202] Seeked to beginning of db in 840ns
I0306 14:16:42.649276 27141 leveldb.cpp:271] Iterated through 0 keys in the db in 448ns
I0306 14:16:42.649291 27141 replica.cpp:776] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
I0306 14:16:42.649471 27163 recover.cpp:451] Starting replica recovery
I0306 14:16:42.649528 27166 recover.cpp:477] Replica is in EMPTY status
I0306 14:16:42.649864 27166 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from __req_res__(5541)@10.99.136.60:39312
I0306 14:16:42.649952 27160 recover.cpp:197] Received a recover response from a replica in EMPTY status
I0306 14:16:42.650060 27164 recover.cpp:568] Updating replica status to STARTING
I0306 14:16:42.650842 27160 master.cpp:380] Master 81fb2ed1-6c17-4dfb-a44f-160cfde9741e (ip-10-99-136-60.ec2.internal) started on 10.99.136.60:39312
I0306 14:16:42.650862 27160 master.cpp:382] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/syBZyN/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="100secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/syBZyN/master" --zk_session_timeout="10secs"
I0306 14:16:42.651005 27160 master.cpp:432] Master only allowing authenticated frameworks to register
I0306 14:16:42.651010 27160 master.cpp:446] Master only allowing authenticated agents to register
I0306 14:16:42.651012 27160 master.cpp:459] Master only allowing authenticated HTTP frameworks to register
I0306 14:16:42.651016 27160 credentials.hpp:37] Loading credentials for authentication from '/tmp/syBZyN/credentials'
I0306 14:16:42.651876 27160 master.cpp:504] Using default 'crammd5' authenticator
I0306 14:16:42.651919 27160 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-master-readonly'
I0306 14:16:42.652004 27160 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-master-readwrite'
I0306 14:16:42.652042 27160 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-master-scheduler'
I0306 14:16:42.652107 27160 master.cpp:584] Authorization enabled
I0306 14:16:42.652220 27161 whitelist_watcher.cpp:77] No whitelist given
I0306 14:16:42.652228 27165 hierarchical.cpp:149] Initialized hierarchical allocator process
I0306 14:16:42.652447 27162 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 2.311958ms
I0306 14:16:42.652469 27162 replica.cpp:320] Persisted replica status to STARTING
I0306 14:16:42.652573 27162 recover.cpp:477] Replica is in STARTING status
I0306 14:16:42.652951 27162 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from __req_res__(5542)@10.99.136.60:39312
I0306 14:16:42.652984 27163 master.cpp:2017] Elected as the leading master!
I0306 14:16:42.652999 27163 master.cpp:1560] Recovering from registrar
I0306 14:16:42.653074 27164 registrar.cpp:329] Recovering registrar
I0306 14:16:42.653125 27165 recover.cpp:197] Received a recover response from a replica in STARTING status
I0306 14:16:42.653259 27161 recover.cpp:568] Updating replica status to VOTING
I0306 14:16:42.654013 27161 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 662866ns
I0306 14:16:42.654029 27161 replica.cpp:320] Persisted replica status to VOTING
I0306 14:16:42.654060 27161 recover.cpp:582] Successfully joined the Paxos group
I0306 14:16:42.654132 27161 recover.cpp:466] Recover process terminated
I0306 14:16:42.654265 27163 log.cpp:553] Attempting to start the writer
I0306 14:16:42.654656 27160 replica.cpp:493] Replica received implicit promise request from __req_res__(5543)@10.99.136.60:39312 with proposal 1
I0306 14:16:42.655340 27160 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 660605ns
I0306 14:16:42.655356 27160 replica.cpp:342] Persisted promised to 1
I0306 14:16:42.655604 27167 coordinator.cpp:238] Coordinator attempting to fill missing positions
I0306 14:16:42.656023 27167 replica.cpp:388] Replica received explicit promise request from __req_res__(5544)@10.99.136.60:39312 for position 0 with proposal 2
I0306 14:16:42.656772 27167 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 725495ns
I0306 14:16:42.656790 27167 replica.cpp:708] Persisted action NOP at position 0
I0306 14:16:42.657176 27167 replica.cpp:537] Replica received write request for position 0 from __req_res__(5545)@10.99.136.60:39312
I0306 14:16:42.657212 27167 leveldb.cpp:436] Reading position from leveldb took 13243ns
I0306 14:16:42.657894 27167 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 665260ns
I0306 14:16:42.657912 27167 replica.cpp:708] Persisted action NOP at position 0
I0306 14:16:42.658164 27161 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0
I0306 14:16:42.658838 27161 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 648537ns
I0306 14:16:42.658854 27161 replica.cpp:708] Persisted action NOP at position 0
I0306 14:16:42.659024 27163 log.cpp:569] Writer started with ending position 0
I0306 14:16:42.659308 27162 leveldb.cpp:436] Reading position from leveldb took 8970ns
I0306 14:16:42.659523 27162 registrar.cpp:362] Successfully fetched the registry (0B) in 6.4192ms
I0306 14:16:42.659561 27162 registrar.cpp:461] Applied 1 operations in 5090ns; attempting to update the registry
I0306 14:16:42.659735 27166 log.cpp:577] Attempting to append 207 bytes to the log
I0306 14:16:42.659795 27167 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1
I0306 14:16:42.660092 27164 replica.cpp:537] Replica received write request for position 1 from __req_res__(5546)@10.99.136.60:39312
I0306 14:16:42.660751 27164 leveldb.cpp:341] Persisting action (226 bytes) to leveldb took 635316ns
I0306 14:16:42.660768 27164 replica.cpp:708] Persisted action APPEND at position 1
I0306 14:16:42.661036 27162 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0
I0306 14:16:42.661697 27162 leveldb.cpp:341] Persisting action (228 bytes) to leveldb took 641896ns
I0306 14:16:42.661715 27162 replica.cpp:708] Persisted action APPEND at position 1
I0306 14:16:42.661908 27163 registrar.cpp:506] Successfully updated the registry in 2.329088ms
I0306 14:16:42.661974 27163 registrar.cpp:392] Successfully recovered registrar
I0306 14:16:42.662014 27164 log.cpp:596] Attempting to truncate the log to 1
I0306 14:16:42.662068 27165 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2
I0306 14:16:42.662142 27166 master.cpp:1676] Recovered 0 agents from the registry (168B); allowing 10mins for agents to re-register
I0306 14:16:42.662266 27162 hierarchical.cpp:176] Skipping recovery of hierarchical allocator: nothing to recover
I0306 14:16:42.662377 27161 replica.cpp:537] Replica received write request for position 2 from __req_res__(5547)@10.99.136.60:39312
I0306 14:16:42.663030 27161 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 630392ns
I0306 14:16:42.663048 27161 replica.cpp:708] Persisted action TRUNCATE at position 2
I0306 14:16:42.663291 27165 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0
I0306 14:16:42.663934 27165 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 622983ns
I0306 14:16:42.663959 27165 leveldb.cpp:399] Deleting ~1 keys from leveldb took 9877ns
I0306 14:16:42.663965 27165 replica.cpp:708] Persisted action TRUNCATE at position 2
I0306 14:16:42.703475 27141 containerizer.cpp:200] Using isolation: cgroups/cpu,cgroups/mem,filesystem/posix,network/cni
I0306 14:16:42.706837 27141 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0306 14:16:42.718296 27141 cluster.cpp:435] Creating default 'local' authorizer
I0306 14:16:42.718719 27166 slave.cpp:208] Mesos agent started on @10.99.136.60:39312
I0306 14:16:42.718734 27166 slave.cpp:209] Flags at startup: --acls="" --agent_subsystems="memory,cpuacct" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="true" --http_credentials="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" --launcher="linux" --launcher_dir="/home/admin/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-debian-8/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh"
I0306 14:16:42.718966 27166 slave.cpp:229] Moving agent process into its own cgroup for subsystem: memory
I0306 14:16:42.719632 27141 sched.cpp:226] Version: 1.1.1
I0306 14:16:42.719807 27160 sched.cpp:330] New master detected at master@10.99.136.60:39312
I0306 14:16:42.719841 27160 sched.cpp:396] Authenticating with master master@10.99.136.60:39312
I0306 14:16:42.719847 27160 sched.cpp:403] Using default CRAM-MD5 authenticatee
I0306 14:16:42.719944 27161 authenticatee.cpp:121] Creating new client SASL connection
I0306 14:16:42.720304 27161 master.cpp:6690] Authenticating scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312
I0306 14:16:42.720366 27162 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(827)@10.99.136.60:39312
I0306 14:16:42.720453 27160 authenticator.cpp:98] Creating new server SASL connection
I0306 14:16:42.720757 27161 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0306 14:16:42.720777 27161 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0306 14:16:42.720806 27161 authenticator.cpp:204] Received SASL authentication start
I0306 14:16:42.720844 27161 authenticator.cpp:326] Authentication requires more steps
I0306 14:16:42.720885 27161 authenticatee.cpp:259] Received SASL authentication step
I0306 14:16:42.720950 27163 authenticator.cpp:232] Received SASL authentication step
I0306 14:16:42.720973 27163 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-99-136-60.ec2.internal' server FQDN: 'ip-10-99-136-60.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I0306 14:16:42.720978 27163 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0306 14:16:42.720986 27163 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0306 14:16:42.720993 27163 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-99-136-60.ec2.internal' server FQDN: 'ip-10-99-136-60.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I0306 14:16:42.720995 27163 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0306 14:16:42.720999 27163 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0306 14:16:42.721007 27163 authenticator.cpp:318] Authentication success
I0306 14:16:42.721051 27163 authenticatee.cpp:299] Authentication success
I0306 14:16:42.721086 27161 master.cpp:6720] Successfully authenticated principal 'test-principal' at scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312
I0306 14:16:42.721127 27162 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(827)@10.99.136.60:39312
I0306 14:16:42.721165 27163 sched.cpp:502] Successfully authenticated with master master@10.99.136.60:39312
I0306 14:16:42.721179 27163 sched.cpp:820] Sending SUBSCRIBE call to master@10.99.136.60:39312
I0306 14:16:42.721261 27163 sched.cpp:853] Will retry registration in 1.583809875secs if necessary
I0306 14:16:42.721310 27167 master.cpp:2596] Received SUBSCRIBE call for framework 'default' at scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312
I0306 14:16:42.721333 27167 master.cpp:2053] Authorizing framework principal 'test-principal' to receive offers for role '*'
I0306 14:16:42.721454 27162 master.cpp:2672] Subscribing framework default with checkpointing enabled and capabilities [ ]
I0306 14:16:42.721623 27167 hierarchical.cpp:275] Added framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.721624 27160 sched.cpp:743] Framework registered with 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.721648 27167 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.721665 27167 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.721669 27160 sched.cpp:757] Scheduler::registered took 10506ns
I0306 14:16:42.721676 27167 hierarchical.cpp:1286] Performed allocation for 0 agents in 36301ns
I0306 14:16:42.723873 27166 slave.cpp:229] Moving agent process into its own cgroup for subsystem: cpuacct
I0306 14:16:42.726965 27166 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/credential'
I0306 14:16:42.727025 27166 slave.cpp:346] Agent using credential for: test-principal
I0306 14:16:42.727039 27166 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/http_credentials'
I0306 14:16:42.727095 27166 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
I0306 14:16:42.727141 27166 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
I0306 14:16:42.727638 27166 slave.cpp:533] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I0306 14:16:42.727671 27166 slave.cpp:541] Agent attributes: [ ]
I0306 14:16:42.727675 27166 slave.cpp:546] Agent hostname: ip-10-99-136-60.ec2.internal
I0306 14:16:42.727994 27166 state.cpp:57] Recovering state from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta'
I0306 14:16:42.728076 27163 status_update_manager.cpp:203] Recovering status update manager
I0306 14:16:42.728175 27167 containerizer.cpp:555] Recovering containerizer
I0306 14:16:42.783991 27166 provisioner.cpp:253] Provisioner recovery complete
I0306 14:16:42.784154 27163 slave.cpp:5281] Finished recovery
I0306 14:16:42.784397 27163 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.784534 27161 slave.cpp:915] New master detected at master@10.99.136.60:39312
I0306 14:16:42.784560 27161 slave.cpp:974] Authenticating with master master@10.99.136.60:39312
I0306 14:16:42.784540 27160 status_update_manager.cpp:177] Pausing sending status updates
I0306 14:16:42.784571 27161 slave.cpp:985] Using default CRAM-MD5 authenticatee
I0306 14:16:42.784605 27161 slave.cpp:947] Detecting new master
I0306 14:16:42.784646 27166 authenticatee.cpp:121] Creating new client SASL connection
I0306 14:16:42.784765 27161 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.785102 27166 master.cpp:6690] Authenticating agent@10.99.136.60:39312
I0306 14:16:42.785166 27161 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(828)@10.99.136.60:39312
I0306 14:16:42.785225 27161 authenticator.cpp:98] Creating new server SASL connection
I0306 14:16:42.785554 27161 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0306 14:16:42.785580 27161 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0306 14:16:42.785614 27161 authenticator.cpp:204] Received SASL authentication start
I0306 14:16:42.785660 27161 authenticator.cpp:326] Authentication requires more steps
I0306 14:16:42.785701 27161 authenticatee.cpp:259] Received SASL authentication step
I0306 14:16:42.785776 27165 authenticator.cpp:232] Received SASL authentication step
I0306 14:16:42.785809 27165 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-99-136-60.ec2.internal' server FQDN: 'ip-10-99-136-60.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I0306 14:16:42.785817 27165 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0306 14:16:42.785827 27165 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0306 14:16:42.785840 27165 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-99-136-60.ec2.internal' server FQDN: 'ip-10-99-136-60.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I0306 14:16:42.785856 27165 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0306 14:16:42.785863 27165 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0306 14:16:42.785877 27165 authenticator.cpp:318] Authentication success
I0306 14:16:42.785943 27167 authenticatee.cpp:299] Authentication success
I0306 14:16:42.785958 27166 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(828)@10.99.136.60:39312
I0306 14:16:42.785980 27165 master.cpp:6720] Successfully authenticated principal 'test-principal' at agent@10.99.136.60:39312
I0306 14:16:42.786044 27167 slave.cpp:1069] Successfully authenticated with master master@10.99.136.60:39312
I0306 14:16:42.786108 27167 slave.cpp:1475] Will retry registration in 14.784705ms if necessary
I0306 14:16:42.786175 27161 master.cpp:5113] Registering agent at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal) with id 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0
I0306 14:16:42.786298 27161 registrar.cpp:461] Applied 1 operations in 13620ns; attempting to update the registry
I0306 14:16:42.786489 27166 log.cpp:577] Attempting to append 392 bytes to the log
I0306 14:16:42.786545 27166 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3
I0306 14:16:42.786849 27164 replica.cpp:537] Replica received write request for position 3 from __req_res__(5548)@10.99.136.60:39312
I0306 14:16:42.787546 27164 leveldb.cpp:341] Persisting action (411 bytes) to leveldb took 667874ns
I0306 14:16:42.787565 27164 replica.cpp:708] Persisted action APPEND at position 3
I0306 14:16:42.787844 27164 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0
I0306 14:16:42.788560 27164 leveldb.cpp:341] Persisting action (413 bytes) to leveldb took 694452ns
I0306 14:16:42.788578 27164 replica.cpp:708] Persisted action APPEND at position 3
I0306 14:16:42.788902 27167 registrar.cpp:506] Successfully updated the registry in 2.580992ms
I0306 14:16:42.788939 27160 log.cpp:596] Attempting to truncate the log to 3
I0306 14:16:42.789057 27161 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4
I0306 14:16:42.789134 27165 slave.cpp:4133] Received ping from slave-observer(383)@10.99.136.60:39312
I0306 14:16:42.789139 27166 master.cpp:5184] Registered agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I0306 14:16:42.789218 27160 hierarchical.cpp:485] Added agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 (ip-10-99-136-60.ec2.internal) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: {})
I0306 14:16:42.789238 27165 slave.cpp:1115] Registered with master master@10.99.136.60:39312; given agent ID 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0
I0306 14:16:42.789253 27165 fetcher.cpp:90] Clearing fetcher cache
I0306 14:16:42.789345 27161 status_update_manager.cpp:184] Resuming sending status updates
I0306 14:16:42.789378 27162 replica.cpp:537] Replica received write request for position 4 from __req_res__(5549)@10.99.136.60:39312
I0306 14:16:42.789415 27165 slave.cpp:1138] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/slave.info'
I0306 14:16:42.789453 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.789490 27160 hierarchical.cpp:1309] Performed allocation for agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 in 236991ns
I0306 14:16:42.789577 27166 master.cpp:6519] Sending 1 offers to framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 (default) at scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312
I0306 14:16:42.789587 27165 slave.cpp:1175] Forwarding total oversubscribed resources {}
I0306 14:16:42.789656 27166 master.cpp:5581] Received update of agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal) with total oversubscribed resources {}
I0306 14:16:42.789749 27164 sched.cpp:917] Scheduler::resourceOffers took 55365ns
I0306 14:16:42.789755 27163 hierarchical.cpp:555] Agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 (ip-10-99-136-60.ec2.internal) updated with oversubscribed resources {} (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000])
I0306 14:16:42.789804 27163 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.789813 27163 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.789822 27163 hierarchical.cpp:1309] Performed allocation for agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 in 37911ns
I0306 14:16:42.790102 27162 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 695458ns
I0306 14:16:42.790120 27162 replica.cpp:708] Persisted action TRUNCATE at position 4
I0306 14:16:42.790268 27164 master.cpp:3554] Processing ACCEPT call for offers: [ 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-O0 ] on agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal) for framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 (default) at scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312
I0306 14:16:42.790312 27164 master.cpp:3159] Authorizing framework principal 'test-principal' to launch task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e
I0306 14:16:42.790345 27167 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0
I0306 14:16:42.790832 27166 master.cpp:8272] Adding task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.790886 27166 master.cpp:4192] Launching task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 (default) at scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.791117 27162 slave.cpp:1539] Got assigned task 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' for framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.791177 27162 slave.cpp:6093] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/framework.info'
I0306 14:16:42.791317 27162 slave.cpp:6104] Checkpointing framework pid 'scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312' to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/framework.pid'
I0306 14:16:42.791324 27167 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 960229ns
I0306 14:16:42.791365 27167 leveldb.cpp:399] Deleting ~2 keys from leveldb took 16774ns
I0306 14:16:42.791373 27167 replica.cpp:708] Persisted action TRUNCATE at position 4
I0306 14:16:42.791566 27162 slave.cpp:1701] Launching task 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' for framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.791780 27162 paths.cpp:536] Trying to chown '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/runs/b361a08d-36ea-42d9-bf47-8577986ae6d9' to user 'root'
I0306 14:16:42.797803 27162 slave.cpp:6572] Checkpointing ExecutorInfo to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/executor.info'
I0306 14:16:42.798091 27162 slave.cpp:6179] Launching executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/runs/b361a08d-36ea-42d9-bf47-8577986ae6d9'
I0306 14:16:42.798343 27163 containerizer.cpp:938] Starting container b361a08d-36ea-42d9-bf47-8577986ae6d9 for executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.798364 27162 slave.cpp:6595] Checkpointing TaskInfo to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/runs/b361a08d-36ea-42d9-bf47-8577986ae6d9/tasks/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/task.info'
I0306 14:16:42.798552 27162 slave.cpp:1987] Queued task 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' for executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.798601 27162 slave.cpp:868] Successfully attached file '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/runs/b361a08d-36ea-42d9-bf47-8577986ae6d9'
I0306 14:16:42.798938 27164 cgroups.cpp:405] Creating cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9' for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.799762 27164 cgroups.cpp:468] Chown the cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9' to user 'root' for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.799823 27164 cgroups.cpp:405] Creating cgroup at '/sys/fs/cgroup/memory/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9' for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.800835 27164 memory.cpp:451] Started listening for OOM events for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.801225 27164 memory.cpp:562] Started listening on 'low' memory pressure events for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.801594 27164 memory.cpp:562] Started listening on 'medium' memory pressure events for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.801911 27164 memory.cpp:562] Started listening on 'critical' memory pressure events for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.801926 27164 cgroups.cpp:468] Chown the cgroup at '/sys/fs/cgroup/memory/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9' to user 'root' for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.802609 27162 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 1056MB for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.803257 27162 memory.cpp:251] Updated 'memory.limit_in_bytes' to 1056MB for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.803572 27162 cpu.cpp:101] Updated 'cpu.shares' to 2150 (cpus 2.1) for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.804222 27166 containerizer.cpp:1463] Launching 'mesos-containerizer' with flags '--command="{"arguments":["mesos-executor","--launcher_dir=\/home\/admin\/workspace\/mesos\/Mesos_CI-build\/FLAG\/SSL\/label\/mesos-ec2-debian-8\/mesos\/build\/src"],"shell":false,"value":"\/home\/admin\/workspace\/mesos\/Mesos_CI-build\/FLAG\/SSL\/label\/mesos-ec2-debian-8\/mesos\/build\/src\/mesos-executor"}" --help="false" --pipe_read="38" --pipe_write="39" --pre_exec_commands="[]" --runtime_directory="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/containers/b361a08d-36ea-42d9-bf47-8577986ae6d9" --unshare_namespace_mnt="false" --user="root" --working_directory="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/runs/b361a08d-36ea-42d9-bf47-8577986ae6d9"'
I0306 14:16:42.804344 27162 linux_launcher.cpp:421] Launching container b361a08d-36ea-42d9-bf47-8577986ae6d9 and cloning with namespaces
I0306 14:16:42.808388 27166 containerizer.cpp:1500] Checkpointing container's forked pid 5216 to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/runs/b361a08d-36ea-42d9-bf47-8577986ae6d9/pids/forked.pid'
I0306 14:16:42.882657 5230 executor.cpp:189] Version: 1.1.1
I0306 14:16:42.884796 27141 slave.cpp:787] Agent terminating
I0306 14:16:42.884974 27167 master.cpp:1258] Agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal) disconnected
I0306 14:16:42.885000 27167 master.cpp:2963] Disconnecting agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.885025 27167 master.cpp:2982] Deactivating agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.885092 27161 hierarchical.cpp:584] Agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 deactivated
I0306 14:16:42.886492 27141 containerizer.cpp:200] Using isolation: cgroups/cpu,cgroups/mem,filesystem/posix,network/cni
Received ERROR event
Error: Received unexpected '500 Internal Server Error' () for SUBSCRIBE
I0306 14:16:42.889385 27141 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0306 14:16:42.900674 27141 cluster.cpp:435] Creating default 'local' authorizer
I0306 14:16:42.901161 27164 slave.cpp:208] Mesos agent started on @10.99.136.60:39312
I0306 14:16:42.901180 27164 slave.cpp:209] Flags at startup: --acls="" --agent_subsystems="memory,cpuacct" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="true" --http_credentials="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" --launcher="linux" --launcher_dir="/home/admin/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-debian-8/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh"
I0306 14:16:42.901371 27164 slave.cpp:229] Moving agent process into its own cgroup for subsystem: memory
I0306 14:16:42.905586 27164 slave.cpp:229] Moving agent process into its own cgroup for subsystem: cpuacct
I0306 14:16:42.908455 27164 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/credential'
I0306 14:16:42.908514 27164 slave.cpp:346] Agent using credential for: test-principal
I0306 14:16:42.908526 27164 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_x0aU5L/http_credentials'
I0306 14:16:42.908582 27164 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
I0306 14:16:42.908629 27164 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
I0306 14:16:42.909207 27164 slave.cpp:533] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I0306 14:16:42.909242 27164 slave.cpp:541] Agent attributes: [ ]
I0306 14:16:42.909247 27164 slave.cpp:546] Agent hostname: ip-10-99-136-60.ec2.internal
I0306 14:16:42.909672 27164 state.cpp:57] Recovering state from '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta'
I0306 14:16:42.909704 27164 state.cpp:698] No committed checkpointed resources found at '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/resources/resources.info'
W0306 14:16:42.910161 27164 state.cpp:606] Failed to find status updates file '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/runs/b361a08d-36ea-42d9-bf47-8577986ae6d9/tasks/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/task.updates'
W0306 14:16:42.910282 27164 state.cpp:544] Failed to find executor libprocess pid/http marker file
I0306 14:16:42.910697 27162 fetcher.cpp:90] Clearing fetcher cache
I0306 14:16:42.910733 27162 slave.cpp:5369] Recovering framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.910754 27162 slave.cpp:6329] Recovering executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.911003 27166 status_update_manager.cpp:203] Recovering status update manager
I0306 14:16:42.911023 27166 status_update_manager.cpp:211] Recovering executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
W0306 14:16:42.911042 27166 status_update_manager.cpp:250] No updates found for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.911234 27166 containerizer.cpp:555] Recovering containerizer
I0306 14:16:42.911268 27166 containerizer.cpp:611] Recovering container b361a08d-36ea-42d9-bf47-8577986ae6d9 for executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.911942 27162 linux_launcher.cpp:291] Recovered container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.913151 27166 memory.cpp:451] Started listening for OOM events for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.913532 27166 memory.cpp:562] Started listening on 'low' memory pressure events for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.913898 27166 memory.cpp:562] Started listening on 'medium' memory pressure events for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.914278 27166 memory.cpp:562] Started listening on 'critical' memory pressure events for container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.915864 27163 provisioner.cpp:253] Provisioner recovery complete
I0306 14:16:42.916134 27165 slave.cpp:5231] Unable to reconnect to executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 because no pid or http checkpoint file was found
I0306 14:16:42.916437 27162 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.916451 27165 slave.cpp:4901] Framework 5e5662bf-b443-4741-9d11-e590d81a7445-0000 seems to have exited. Ignoring shutdown timeout for executor 'default'
I0306 14:16:42.916460 27162 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.916470 27165 slave.cpp:3574] Cleaning up un-reregistered executors
I0306 14:16:42.916474 27165 slave.cpp:3592] Killing un-reregistered executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.916479 27162 hierarchical.cpp:1286] Performed allocation for 1 agents in 68892ns
I0306 14:16:42.916528 27165 slave.cpp:5281] Finished recovery
I0306 14:16:42.916554 27162 containerizer.cpp:1961] Destroying container b361a08d-36ea-42d9-bf47-8577986ae6d9 in RUNNING state
I0306 14:16:42.916677 27162 linux_launcher.cpp:498] Asked to destroy container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.916856 27165 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.916991 27163 status_update_manager.cpp:177] Pausing sending status updates
I0306 14:16:42.916987 27160 slave.cpp:915] New master detected at master@10.99.136.60:39312
I0306 14:16:42.917021 27160 slave.cpp:974] Authenticating with master master@10.99.136.60:39312
I0306 14:16:42.917032 27160 slave.cpp:985] Using default CRAM-MD5 authenticatee
I0306 14:16:42.917043 27162 linux_launcher.cpp:541] Using freezer to destroy cgroup mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.917064 27160 slave.cpp:947] Detecting new master
I0306 14:16:42.917120 27165 authenticatee.cpp:121] Creating new client SASL connection
I0306 14:16:42.917191 27160 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.917623 27165 master.cpp:6690] Authenticating agent@10.99.136.60:39312
I0306 14:16:42.917695 27163 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(829)@10.99.136.60:39312
I0306 14:16:42.917806 27164 authenticator.cpp:98] Creating new server SASL connection
I0306 14:16:42.917816 27161 cgroups.cpp:2705] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.918229 27164 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0306 14:16:42.918253 27164 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0306 14:16:42.918321 27161 authenticator.cpp:204] Received SASL authentication start
I0306 14:16:42.918365 27161 authenticator.cpp:326] Authentication requires more steps
I0306 14:16:42.918423 27161 authenticatee.cpp:259] Received SASL authentication step
I0306 14:16:42.918527 27167 authenticator.cpp:232] Received SASL authentication step
I0306 14:16:42.918552 27167 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-99-136-60.ec2.internal' server FQDN: 'ip-10-99-136-60.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I0306 14:16:42.918558 27167 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0306 14:16:42.918565 27167 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0306 14:16:42.918571 27167 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-10-99-136-60.ec2.internal' server FQDN: 'ip-10-99-136-60.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I0306 14:16:42.918576 27167 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0306 14:16:42.918579 27167 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0306 14:16:42.918591 27167 authenticator.cpp:318] Authentication success
I0306 14:16:42.918653 27161 authenticatee.cpp:299] Authentication success
I0306 14:16:42.918676 27167 master.cpp:6720] Successfully authenticated principal 'test-principal' at agent@10.99.136.60:39312
I0306 14:16:42.918715 27166 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(829)@10.99.136.60:39312
I0306 14:16:42.918820 27161 slave.cpp:1069] Successfully authenticated with master master@10.99.136.60:39312
I0306 14:16:42.918956 27161 slave.cpp:1475] Will retry registration in 4.446129ms if necessary
I0306 14:16:42.919051 27164 master.cpp:5261] Re-registering agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.919158 27161 slave.cpp:1217] Re-registered with master master@10.99.136.60:39312
I0306 14:16:42.919167 27164 master.cpp:5520] Sending updated checkpointed resources {} to agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.919242 27161 slave.cpp:1253] Forwarding total oversubscribed resources {}
I0306 14:16:42.919179 27163 hierarchical.cpp:572] Agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 reactivated
I0306 14:16:42.919286 27163 status_update_manager.cpp:184] Resuming sending status updates
I0306 14:16:42.919340 27161 slave.cpp:2680] Updating framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 pid to scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312
I0306 14:16:42.919368 27164 master.cpp:5581] Received update of agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal) with total oversubscribed resources {}
I0306 14:16:42.919380 27161 slave.cpp:2696] Checkpointing framework pid 'scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312' to '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/framework.pid'
I0306 14:16:42.919463 27164 hierarchical.cpp:555] Agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 (ip-10-99-136-60.ec2.internal) updated with oversubscribed resources {} (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000])
I0306 14:16:42.919525 27164 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.919548 27164 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.919548 27161 slave.cpp:2743] Ignoring new checkpointed resources identical to the current version: {}
I0306 14:16:42.919564 27163 status_update_manager.cpp:184] Resuming sending status updates
I0306 14:16:42.919564 27164 hierarchical.cpp:1309] Performed allocation for agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 in 62355ns
I0306 14:16:42.919626 27165 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9 after 0ns
I0306 14:16:42.920775 27162 cgroups.cpp:2723] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.921849 27164 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9 after 0ns
I0306 14:16:42.922220 27160 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.922238 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.922255 27160 hierarchical.cpp:1286] Performed allocation for 1 agents in 55324ns
I0306 14:16:42.922456 27164 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.922473 27164 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.922489 27164 hierarchical.cpp:1286] Performed allocation for 1 agents in 54274ns
I0306 14:16:42.922690 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.922711 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.922726 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 55870ns
I0306 14:16:42.922844 27165 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.922874 27166 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.922888 27166 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.922901 27166 hierarchical.cpp:1286] Performed allocation for 1 agents in 46961ns
I0306 14:16:42.922947 27162 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.923118 27160 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.923194 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.923218 27160 hierarchical.cpp:1286] Performed allocation for 1 agents in 111963ns
I0306 14:16:42.923343 27161 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.923370 27165 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.923383 27165 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.923394 27165 hierarchical.cpp:1286] Performed allocation for 1 agents in 44908ns
I0306 14:16:42.923444 27164 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.923624 27162 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.923727 27165 containerizer.cpp:2324] Container b361a08d-36ea-42d9-bf47-8577986ae6d9 has exited
I0306 14:16:42.923796 27160 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.923655 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.923849 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.923868 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 236634ns
I0306 14:16:42.924031 27164 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.924046 27164 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.924054 27164 hierarchical.cpp:1286] Performed allocation for 1 agents in 40460ns
I0306 14:16:42.924161 27160 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.924188 27165 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.924201 27165 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.924213 27165 hierarchical.cpp:1286] Performed allocation for 1 agents in 44301ns
I0306 14:16:42.924255 27165 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.924408 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.924427 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.924443 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 52736ns
I0306 14:16:42.924757 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.924772 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.924782 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 41502ns
I0306 14:16:42.924906 27163 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.924945 27162 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.924957 27162 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.924962 27163 slave.cpp:4133] Received ping from slave-observer(383)@10.99.136.60:39312
I0306 14:16:42.924971 27162 hierarchical.cpp:1286] Performed allocation for 1 agents in 46363ns
I0306 14:16:42.925036 27163 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.925179 27165 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.925195 27165 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.925210 27165 hierarchical.cpp:1286] Performed allocation for 1 agents in 58202ns
I0306 14:16:42.925354 27164 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.925375 27164 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.925391 27164 hierarchical.cpp:1286] Performed allocation for 1 agents in 55304ns
I0306 14:16:42.925490 27161 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.925547 27163 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.925592 27163 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.925592 27167 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.925616 27163 hierarchical.cpp:1286] Performed allocation for 1 agents in 106275ns
I0306 14:16:42.925745 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.925760 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.925776 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 44062ns
I0306 14:16:42.925904 27162 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.925918 27162 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.925926 27162 hierarchical.cpp:1286] Performed allocation for 1 agents in 40830ns
I0306 14:16:42.926066 27164 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.926086 27164 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.926097 27164 hierarchical.cpp:1286] Performed allocation for 1 agents in 51797ns
I0306 14:16:42.926192 27160 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.926236 27167 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.926252 27167 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.926267 27167 hierarchical.cpp:1286] Performed allocation for 1 agents in 51957ns
I0306 14:16:42.926291 27167 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.926432 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.926450 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.926465 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 46629ns
I0306 14:16:42.926601 27162 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.926637 27160 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.926646 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.926657 27160 hierarchical.cpp:1286] Performed allocation for 1 agents in 35428ns
I0306 14:16:42.926677 27162 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.926831 27163 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.926874 27166 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.926929 27166 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.926903 27163 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.926946 27166 hierarchical.cpp:1286] Performed allocation for 1 agents in 94259ns
I0306 14:16:42.927084 27164 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.927103 27164 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.927119 27164 hierarchical.cpp:1286] Performed allocation for 1 agents in 49893ns
I0306 14:16:42.927251 27165 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.927286 27167 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.927301 27167 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.927316 27167 hierarchical.cpp:1286] Performed allocation for 1 agents in 50270ns
I0306 14:16:42.927336 27160 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.927491 27165 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.927511 27165 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.927520 27165 hierarchical.cpp:1286] Performed allocation for 1 agents in 50750ns
I0306 14:16:42.927647 27163 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.927664 27163 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.927678 27163 hierarchical.cpp:1286] Performed allocation for 1 agents in 45266ns
I0306 14:16:42.927934 27162 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.927961 27162 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.927937 27167 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.927979 27162 hierarchical.cpp:1286] Performed allocation for 1 agents in 65882ns
I0306 14:16:42.928023 27167 slave.cpp:4133] Received ping from slave-observer(383)@10.99.136.60:39312
I0306 14:16:42.928103 27167 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.928246 27160 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.928267 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.928280 27160 hierarchical.cpp:1286] Performed allocation for 1 agents in 54376ns
I0306 14:16:42.928411 27165 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.928426 27165 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.928434 27165 hierarchical.cpp:1286] Performed allocation for 1 agents in 43730ns
I0306 14:16:42.928539 27166 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.928576 27162 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.928591 27162 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.928604 27162 hierarchical.cpp:1286] Performed allocation for 1 agents in 48152ns
I0306 14:16:42.928694 27165 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.928839 27164 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.928858 27164 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.928872 27164 hierarchical.cpp:1286] Performed allocation for 1 agents in 50952ns
I0306 14:16:42.928998 27167 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.929009 27167 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.929018 27167 hierarchical.cpp:1286] Performed allocation for 1 agents in 33791ns
I0306 14:16:42.929154 27166 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.929172 27166 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.929188 27166 hierarchical.cpp:1286] Performed allocation for 1 agents in 51924ns
I0306 14:16:42.929311 27160 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.929340 27162 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.929354 27162 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.929364 27162 hierarchical.cpp:1286] Performed allocation for 1 agents in 43296ns
I0306 14:16:42.929417 27160 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.929549 27163 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.929563 27163 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.929572 27163 hierarchical.cpp:1286] Performed allocation for 1 agents in 38074ns
I0306 14:16:42.929697 27166 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.929738 27164 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.929754 27164 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.929765 27164 hierarchical.cpp:1286] Performed allocation for 1 agents in 47560ns
I0306 14:16:42.929780 27163 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.929913 27164 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.929944 27160 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.929957 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.929966 27160 hierarchical.cpp:1286] Performed allocation for 1 agents in 42547ns
I0306 14:16:42.930016 27160 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.930173 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.930191 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.930204 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 44178ns
I0306 14:16:42.930338 27167 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.930377 27160 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.930392 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.930397 27167 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.930408 27160 hierarchical.cpp:1286] Performed allocation for 1 agents in 48185ns
I0306 14:16:42.930656 27167 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.930671 27167 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.930680 27167 hierarchical.cpp:1286] Performed allocation for 1 agents in 43040ns
I0306 14:16:42.930851 27166 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.930871 27166 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.930886 27166 hierarchical.cpp:1286] Performed allocation for 1 agents in 52324ns
I0306 14:16:42.931027 27164 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.931066 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.931095 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.931113 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 68991ns
I0306 14:16:42.931128 27164 slave.cpp:4133] Received ping from slave-observer(383)@10.99.136.60:39312
I0306 14:16:42.931185 27164 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.931349 27167 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.931370 27167 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.931385 27167 hierarchical.cpp:1286] Performed allocation for 1 agents in 58973ns
I0306 14:16:42.931501 27165 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.931515 27165 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.931529 27165 hierarchical.cpp:1286] Performed allocation for 1 agents in 40378ns
I0306 14:16:42.931694 27160 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.931749 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.931706 27161 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.931763 27160 hierarchical.cpp:1286] Performed allocation for 1 agents in 87957ns
I0306 14:16:42.931843 27166 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.932010 27163 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.932029 27163 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.932042 27163 hierarchical.cpp:1286] Performed allocation for 1 agents in 52063ns
I0306 14:16:42.932169 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.932183 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.932193 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 38778ns
I0306 14:16:42.932325 27162 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.932345 27162 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.932355 27162 hierarchical.cpp:1286] Performed allocation for 1 agents in 53651ns
I0306 14:16:42.932517 27161 slave.cpp:4952] Framework 1f27afd5-dbe9-4501-8638-930384d328cb-0000 seems to have exited. Ignoring registration timeout for executor 'f1a2d998-cf45-4e52-ae5a-0a718030bcd1'
I0306 14:16:42.932518 27163 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.932545 27163 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.932552 27161 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.932557 27163 hierarchical.cpp:1286] Performed allocation for 1 agents in 56536ns
I0306 14:16:42.932616 27161 slave.cpp:5044] Current disk usage 12.67%. Max allowed age: 5.413236434229086days
I0306 14:16:42.932669 27161 slave.cpp:5044] Current disk usage 12.67%. Max allowed age: 5.413236434229086days
I0306 14:16:42.932734 27161 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.932919 27160 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.932934 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.932950 27160 hierarchical.cpp:1286] Performed allocation for 1 agents in 49138ns
I0306 14:16:42.933076 27167 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.933106 27163 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.933118 27163 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.933132 27163 hierarchical.cpp:1286] Performed allocation for 1 agents in 43824ns
I0306 14:16:42.933187 27167 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.933356 27162 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.933362 27164 slave.cpp:4952] Framework 5e5662bf-b443-4741-9d11-e590d81a7445-0000 seems to have exited. Ignoring registration timeout for executor 'default'
I0306 14:16:42.933369 27162 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.933390 27162 hierarchical.cpp:1286] Performed allocation for 1 agents in 51252ns
I0306 14:16:42.933398 27164 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.933423 27164 slave.cpp:5044] Current disk usage 12.67%. Max allowed age: 5.413236434229086days
I0306 14:16:42.933506 27164 slave.cpp:5044] Current disk usage 12.67%. Max allowed age: 5.413236434229086days
I0306 14:16:42.933552 27164 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.933714 27167 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.933729 27167 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.933738 27167 hierarchical.cpp:1286] Performed allocation for 1 agents in 40735ns
I0306 14:16:42.933840 27163 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.933859 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.933868 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.933876 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 31707ns
I0306 14:16:42.933943 27165 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.934116 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.934130 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.934139 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 40401ns
I0306 14:16:42.934249 27164 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.934270 27164 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.934284 27164 hierarchical.cpp:1286] Performed allocation for 1 agents in 51093ns
I0306 14:16:42.934437 27161 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.934458 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.934473 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 54263ns
I0306 14:16:42.934504 27163 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.934567 27163 slave.cpp:5044] Current disk usage 12.67%. Max allowed age: 5.413236434229086days
I0306 14:16:42.934650 27163 slave.cpp:4133] Received ping from slave-observer(383)@10.99.136.60:39312
I0306 14:16:42.934713 27163 slave.cpp:5044] Current disk usage 12.67%. Max allowed age: 5.413236434229086days
I0306 14:16:42.934784 27163 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
I0306 14:16:42.934926 27165 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.934942 27165 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.934955 27165 hierarchical.cpp:1286] Performed allocation for 1 agents in 43003ns
I0306 14:16:42.935075 27160 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.935091 27160 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.935107 27160 hierarchical.cpp:1286] Performed allocation for 1 agents in 47681ns
I0306 14:16:42.935232 27161 slave.cpp:5455] Querying resource estimator for oversubscribable resources
I0306 14:16:42.935288 27162 hierarchical.cpp:1694] No allocations performed
I0306 14:16:42.935310 27162 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.935334 27162 hierarchical.cpp:1286] Performed allocation for 1 agents in 71197ns
I0306 14:16:42.935432 27163 slave.cpp:5469] Received oversubscribable resources {} from the resource estimator
E0306 14:16:42.935477 27163 slave.cpp:4520] Termination of executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 failed: Failed to kill all processes in the container: Timed out after 1mins
I0306 14:16:42.935518 27163 slave.cpp:3634] Handling status update TASK_LOST (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 from @0.0.0.0:0
W0306 14:16:42.935824 27164 containerizer.cpp:1882] Skipping status for container b361a08d-36ea-42d9-bf47-8577986ae6d9 because: Container does not exist
W0306 14:16:42.936007 27165 containerizer.cpp:1778] Ignoring update for currently being destroyed container b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.936137 27163 status_update_manager.cpp:323] Received status update TASK_LOST (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.936166 27163 status_update_manager.cpp:500] Creating StatusUpdate stream for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.936347 27163 status_update_manager.cpp:832] Checkpointing UPDATE for status update TASK_LOST (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.936436 27163 status_update_manager.cpp:377] Forwarding update TASK_LOST (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 to the agent
I0306 14:16:42.936522 27166 slave.cpp:4051] Forwarding the update TASK_LOST (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 to master@10.99.136.60:39312
I0306 14:16:42.936620 27166 slave.cpp:3945] Status update manager successfully handled status update TASK_LOST (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.936684 27163 master.cpp:5717] Status update TASK_LOST (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 from agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.936710 27163 master.cpp:5779] Forwarding status update TASK_LOST (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.936755 27163 master.cpp:7650] Updating the state of task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 (latest state: TASK_LOST, status update state: TASK_LOST)
I0306 14:16:42.936857 27161 sched.cpp:1025] Scheduler::statusUpdate took 52728ns
I0306 14:16:42.936962 27162 master.cpp:4829] Processing ACKNOWLEDGE call 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9 for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 (default) at scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312 on agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0
I0306 14:16:42.936969 27165 hierarchical.cpp:1018] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: {}) on agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 from framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.936987 27162 master.cpp:7746] Removing task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 on agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.937103 27162 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.937151 27162 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_LOST (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.937186 27162 status_update_manager.cpp:531] Cleaning up status update stream for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.937261 27162 slave.cpp:2958] Status update manager successfully handled status update acknowledgement (UUID: 07a4c503-f870-4ef3-8bfc-8eea55e3a7a9) for task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.937278 27162 slave.cpp:6551] Completing task aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e
I0306 14:16:42.937286 27162 slave.cpp:4646] Cleaning up executor 'aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' of framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.937424 27160 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/runs/b361a08d-36ea-42d9-bf47-8577986ae6d9' for gc 6.99998955919111days in the future
I0306 14:16:42.937469 27160 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' for gc 6.99998955919111days in the future
I0306 14:16:42.937491 27162 slave.cpp:4734] Cleaning up framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.937505 27160 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e/runs/b361a08d-36ea-42d9-bf47-8577986ae6d9' for gc 6.99998955919111days in the future
I0306 14:16:42.937522 27160 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000/executors/aa3a2a9c-d7cf-4c1d-b713-c00f1fd47e0e' for gc 6.99998955919111days in the future
I0306 14:16:42.937566 27167 status_update_manager.cpp:285] Closing status update streams for framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.937579 27160 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000' for gc 6.99998955919111days in the future
I0306 14:16:42.937623 27160 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredHTTPExecutor_QiqrPh/meta/slaves/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0/frameworks/81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000' for gc 6.99998955919111days in the future
I0306 14:16:42.937916 27161 hierarchical.cpp:1789] No inverse offers to send out!
I0306 14:16:42.937939 27161 hierarchical.cpp:1286] Performed allocation for 1 agents in 168654ns
I0306 14:16:42.938025 27162 master.cpp:6519] Sending 1 offers to framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 (default) at scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312
I0306 14:16:42.938172 27162 sched.cpp:917] Scheduler::resourceOffers took 27548ns
I0306 14:16:42.938426 27141 sched.cpp:1995] Asked to stop the driver
I0306 14:16:42.938519 27165 sched.cpp:1187] Stopping framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.938618 27161 master.cpp:7093] Processing TEARDOWN call for framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 (default) at scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312
I0306 14:16:42.938639 27161 master.cpp:7105] Removing framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 (default) at scheduler-ed3e8c72-b155-4c48-a20f-32b345bf6aa3@10.99.136.60:39312
I0306 14:16:42.938699 27165 hierarchical.cpp:386] Deactivated framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.938731 27163 slave.cpp:2511] Asked to shut down framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000 by master@10.99.136.60:39312
I0306 14:16:42.938750 27163 slave.cpp:2526] Cannot shut down unknown framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.938876 27165 hierarchical.cpp:1018] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: {}) on agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 from framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
I0306 14:16:42.938977 27165 hierarchical.cpp:337] Removed framework 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-0000
../../src/tests/cluster.cpp:565: Failure
Value of: containers.get().empty()
Actual: false
Expected: true
Failed to destroy containers: { b361a08d-36ea-42d9-bf47-8577986ae6d9 }
I0306 14:16:42.939290 27141 slave.cpp:787] Agent terminating
I0306 14:16:42.939400 27162 master.cpp:1258] Agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal) disconnected
I0306 14:16:42.939419 27162 master.cpp:2963] Disconnecting agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.939436 27162 master.cpp:2982] Deactivating agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 at agent@10.99.136.60:39312 (ip-10-99-136-60.ec2.internal)
I0306 14:16:42.939481 27162 hierarchical.cpp:584] Agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0 deactivated
I0306 14:16:42.994210 27141 master.cpp:1097] Master terminating
I0306 14:16:42.994364 27165 hierarchical.cpp:517] Removed agent 81fb2ed1-6c17-4dfb-a44f-160cfde9741e-S0
I0306 14:16:42.997756 27161 cgroups.cpp:2705] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:42.998745 27167 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9 after 956928ns
I0306 14:16:42.999704 27166 cgroups.cpp:2723] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9
I0306 14:16:43.000638 27161 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5/b361a08d-36ea-42d9-bf47-8577986ae6d9 after 907008ns
I0306 14:16:43.002099 27163 cgroups.cpp:2705] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5
I0306 14:16:43.104796 27166 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5 after 102.662912ms
I0306 14:16:43.105820 27162 cgroups.cpp:2723] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5
I0306 14:16:43.106741 27166 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_d548d98c-48dc-480b-84d1-1affc077ccf5 after 889856ns
[ FAILED ] SlaveRecoveryTest/0.RecoverUnregisteredHTTPExecutor, where TypeParam = mesos::internal::slave::MesosContainerizer (484 ms)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)