You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Alexander Rukletsov (JIRA)" <ji...@apache.org> on 2017/05/15 19:00:05 UTC
[jira] [Commented] (MESOS-7506) Multiple tests leave orphan
containers.
[ https://issues.apache.org/jira/browse/MESOS-7506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011122#comment-16011122 ]
Alexander Rukletsov commented on MESOS-7506:
--------------------------------------------
{noformat}
[ RUN ] SlaveRecoveryTest/0.KillTaskUnregisteredExecutor
I0512 23:25:58.069623 8155 cluster.cpp:162] Creating default 'local' authorizer
I0512 23:25:58.070519 8181 master.cpp:436] Master 7bb1bfc2-26a3-45fe-9635-079d48114602 (ip-172-16-10-226.ec2.internal) started on 172.16.10.226:36307
I0512 23:25:58.070578 8181 master.cpp:438] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/pY5HWb/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="100secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/pY5HWb/master" --zk_session_timeout="10secs"
I0512 23:25:58.070660 8181 master.cpp:488] Master only allowing authenticated frameworks to register
I0512 23:25:58.070667 8181 master.cpp:502] Master only allowing authenticated agents to register
I0512 23:25:58.070668 8181 master.cpp:515] Master only allowing authenticated HTTP frameworks to register
I0512 23:25:58.070672 8181 credentials.hpp:37] Loading credentials for authentication from '/tmp/pY5HWb/credentials'
I0512 23:25:58.070729 8181 master.cpp:560] Using default 'crammd5' authenticator
I0512 23:25:58.070765 8181 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readonly'
I0512 23:25:58.070791 8181 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readwrite'
I0512 23:25:58.070837 8181 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-scheduler'
I0512 23:25:58.070910 8181 master.cpp:640] Authorization enabled
I0512 23:25:58.070967 8180 whitelist_watcher.cpp:77] No whitelist given
I0512 23:25:58.070974 8176 hierarchical.cpp:158] Initialized hierarchical allocator process
I0512 23:25:58.071454 8181 master.cpp:2161] Elected as the leading master!
I0512 23:25:58.071467 8181 master.cpp:1700] Recovering from registrar
I0512 23:25:58.071528 8177 registrar.cpp:345] Recovering registrar
I0512 23:25:58.071755 8177 registrar.cpp:389] Successfully fetched the registry (0B) in 203008ns
I0512 23:25:58.071785 8177 registrar.cpp:493] Applied 1 operations in 3836ns; attempting to update the registry
I0512 23:25:58.071967 8178 registrar.cpp:550] Successfully updated the registry in 168960ns
I0512 23:25:58.072003 8178 registrar.cpp:422] Successfully recovered registrar
I0512 23:25:58.072139 8178 master.cpp:1799] Recovered 0 agents from the registry (172B); allowing 10mins for agents to re-register
I0512 23:25:58.072144 8180 hierarchical.cpp:185] Skipping recovery of hierarchical allocator: nothing to recover
I0512 23:25:58.072893 8155 containerizer.cpp:221] Using isolation: cgroups/cpu,cgroups/mem,filesystem/posix,network/cni
I0512 23:25:58.076190 8155 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0512 23:25:58.076443 8155 provisioner.cpp:249] Using default backend 'copy'
I0512 23:25:58.087144 8155 cluster.cpp:448] Creating default 'local' authorizer
I0512 23:25:58.087446 8178 slave.cpp:225] Mesos agent started on (467)@172.16.10.226:36307
I0512 23:25:58.087462 8178 slave.cpp:226] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_secret_key="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/executor_secret_key" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" --launcher="linux" --launcher_dir="/home/admin/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-debian-8/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z"
I0512 23:25:58.087604 8178 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/credential'
I0512 23:25:58.087662 8178 slave.cpp:258] Agent using credential for: test-principal
I0512 23:25:58.087682 8178 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/http_credentials'
I0512 23:25:58.087792 8178 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor'
I0512 23:25:58.087836 8178 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor'
I0512 23:25:58.087937 8178 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
I0512 23:25:58.087978 8178 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly'
I0512 23:25:58.088044 8178 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
I0512 23:25:58.088081 8178 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite'
I0512 23:25:58.088357 8155 sched.cpp:232] Version: 1.4.0
I0512 23:25:58.088500 8177 sched.cpp:336] New master detected at master@172.16.10.226:36307
I0512 23:25:58.088531 8177 sched.cpp:407] Authenticating with master master@172.16.10.226:36307
I0512 23:25:58.088536 8177 sched.cpp:414] Using default CRAM-MD5 authenticatee
I0512 23:25:58.088567 8178 slave.cpp:529] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I0512 23:25:58.088608 8178 slave.cpp:537] Agent attributes: [ ]
I0512 23:25:58.088613 8178 slave.cpp:542] Agent hostname: ip-172-16-10-226.ec2.internal
I0512 23:25:58.088631 8175 authenticatee.cpp:121] Creating new client SASL connection
I0512 23:25:58.088661 8174 status_update_manager.cpp:177] Pausing sending status updates
I0512 23:25:58.088920 8178 state.cpp:62] Recovering state from '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta'
I0512 23:25:58.089066 8175 master.cpp:7475] Authenticating scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.089069 8181 status_update_manager.cpp:203] Recovering status update manager
I0512 23:25:58.089117 8181 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(973)@172.16.10.226:36307
I0512 23:25:58.089200 8181 authenticator.cpp:98] Creating new server SASL connection
I0512 23:25:58.089227 8178 containerizer.cpp:608] Recovering containerizer
I0512 23:25:58.089541 8181 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0512 23:25:58.089565 8181 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0512 23:25:58.089632 8181 authenticator.cpp:204] Received SASL authentication start
I0512 23:25:58.089666 8181 authenticator.cpp:326] Authentication requires more steps
I0512 23:25:58.089748 8181 authenticatee.cpp:259] Received SASL authentication step
I0512 23:25:58.089824 8181 authenticator.cpp:232] Received SASL authentication step
I0512 23:25:58.089850 8181 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-226.ec2.internal' server FQDN: 'ip-172-16-10-226.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I0512 23:25:58.089869 8181 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0512 23:25:58.089880 8181 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0512 23:25:58.089890 8181 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-226.ec2.internal' server FQDN: 'ip-172-16-10-226.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I0512 23:25:58.089918 8181 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0512 23:25:58.089946 8181 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0512 23:25:58.089957 8181 authenticator.cpp:318] Authentication success
I0512 23:25:58.090020 8178 authenticatee.cpp:299] Authentication success
I0512 23:25:58.090070 8175 master.cpp:7505] Successfully authenticated principal 'test-principal' at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.090091 8177 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(973)@172.16.10.226:36307
I0512 23:25:58.090134 8175 sched.cpp:513] Successfully authenticated with master master@172.16.10.226:36307
I0512 23:25:58.090154 8175 sched.cpp:836] Sending SUBSCRIBE call to master@172.16.10.226:36307
I0512 23:25:58.090198 8175 sched.cpp:869] Will retry registration in 1.892273409secs if necessary
I0512 23:25:58.090250 8180 master.cpp:2813] Received SUBSCRIBE call for framework 'default' at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.090276 8180 master.cpp:2197] Authorizing framework principal 'test-principal' to receive offers for roles '{ * }'
I0512 23:25:58.090384 8180 master.cpp:2890] Subscribing framework default with checkpointing enabled and capabilities [ ]
I0512 23:25:58.090518 8175 hierarchical.cpp:273] Added framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.090555 8176 sched.cpp:759] Framework registered with 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.090562 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.090574 8176 sched.cpp:773] Scheduler::registered took 6990ns
I0512 23:25:58.090575 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.090587 8175 hierarchical.cpp:1434] Performed allocation for 0 agents in 31430ns
I0512 23:25:58.091061 8181 provisioner.cpp:410] Provisioner recovery complete
I0512 23:25:58.091192 8176 slave.cpp:5974] Finished recovery
I0512 23:25:58.091353 8176 slave.cpp:6156] Querying resource estimator for oversubscribable resources
I0512 23:25:58.091423 8179 slave.cpp:922] New master detected at master@172.16.10.226:36307
I0512 23:25:58.091434 8175 status_update_manager.cpp:177] Pausing sending status updates
I0512 23:25:58.091497 8179 slave.cpp:957] Detecting new master
I0512 23:25:58.091543 8179 slave.cpp:6170] Received oversubscribable resources {} from the resource estimator
I0512 23:25:58.098682 8178 slave.cpp:984] Authenticating with master master@172.16.10.226:36307
I0512 23:25:58.098708 8178 slave.cpp:995] Using default CRAM-MD5 authenticatee
I0512 23:25:58.098760 8178 authenticatee.cpp:121] Creating new client SASL connection
I0512 23:25:58.099088 8178 master.cpp:7475] Authenticating slave(467)@172.16.10.226:36307
I0512 23:25:58.099146 8181 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(974)@172.16.10.226:36307
I0512 23:25:58.099211 8181 authenticator.cpp:98] Creating new server SASL connection
I0512 23:25:58.099520 8181 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0512 23:25:58.099536 8181 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0512 23:25:58.099556 8181 authenticator.cpp:204] Received SASL authentication start
I0512 23:25:58.099577 8181 authenticator.cpp:326] Authentication requires more steps
I0512 23:25:58.099612 8181 authenticatee.cpp:259] Received SASL authentication step
I0512 23:25:58.099683 8179 authenticator.cpp:232] Received SASL authentication step
I0512 23:25:58.099706 8179 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-226.ec2.internal' server FQDN: 'ip-172-16-10-226.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I0512 23:25:58.099712 8179 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0512 23:25:58.099719 8179 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0512 23:25:58.099723 8179 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-226.ec2.internal' server FQDN: 'ip-172-16-10-226.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I0512 23:25:58.099727 8179 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0512 23:25:58.099730 8179 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0512 23:25:58.099736 8179 authenticator.cpp:318] Authentication success
I0512 23:25:58.099769 8179 authenticatee.cpp:299] Authentication success
I0512 23:25:58.099803 8181 master.cpp:7505] Successfully authenticated principal 'test-principal' at slave(467)@172.16.10.226:36307
I0512 23:25:58.099843 8176 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(974)@172.16.10.226:36307
I0512 23:25:58.099907 8179 slave.cpp:1079] Successfully authenticated with master master@172.16.10.226:36307
I0512 23:25:58.099973 8179 slave.cpp:1507] Will retry registration in 456749ns if necessary
I0512 23:25:58.100003 8175 master.cpp:5429] Received register agent message from slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.100028 8175 master.cpp:3659] Authorizing agent with principal 'test-principal'
I0512 23:25:58.100167 8177 master.cpp:5564] Registering agent at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal) with id 7bb1bfc2-26a3-45fe-9635-079d48114602-S0
I0512 23:25:58.100265 8179 registrar.cpp:493] Applied 1 operations in 11129ns; attempting to update the registry
I0512 23:25:58.100477 8176 registrar.cpp:550] Successfully updated the registry in 188160ns
I0512 23:25:58.100659 8174 slave.cpp:4749] Received ping from slave-observer(462)@172.16.10.226:36307
I0512 23:25:58.100654 8178 master.cpp:5639] Registered agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I0512 23:25:58.100705 8174 slave.cpp:1125] Registered with master master@172.16.10.226:36307; given agent ID 7bb1bfc2-26a3-45fe-9635-079d48114602-S0
I0512 23:25:58.100720 8174 fetcher.cpp:94] Clearing fetcher cache
I0512 23:25:58.100731 8176 hierarchical.cpp:525] Added agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 (ip-172-16-10-226.ec2.internal) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: {})
I0512 23:25:58.100803 8175 status_update_manager.cpp:184] Resuming sending status updates
I0512 23:25:58.100910 8174 slave.cpp:1153] Checkpointing SlaveInfo to '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/slave.info'
I0512 23:25:58.100941 8176 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.100957 8176 hierarchical.cpp:1434] Performed allocation for 1 agents in 189873ns
I0512 23:25:58.101070 8174 slave.cpp:1191] Forwarding total oversubscribed resources {}
I0512 23:25:58.101071 8180 master.cpp:7305] Sending 1 offers to framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.101181 8180 master.cpp:6324] Received update of agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal) with total oversubscribed resources {}
I0512 23:25:58.101230 8175 sched.cpp:933] Scheduler::resourceOffers took 37673ns
I0512 23:25:58.101680 8175 master.cpp:3875] Processing ACCEPT call for offers: [ 7bb1bfc2-26a3-45fe-9635-079d48114602-O0 ] on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal) for framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.101712 8175 master.cpp:3426] Authorizing framework principal 'test-principal' to launch task ddb637d7-7789-402c-92bc-4932eca15b80
I0512 23:25:58.102190 8181 master.cpp:9326] Adding task ddb637d7-7789-402c-92bc-4932eca15b80 with resources cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.102267 8181 master.cpp:4531] Launching task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307 with resources cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.102461 8177 slave.cpp:1617] Got assigned task 'ddb637d7-7789-402c-92bc-4932eca15b80' for framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.102538 8177 slave.cpp:6843] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/framework.info'
I0512 23:25:58.102710 8177 slave.cpp:6854] Checkpointing framework pid 'scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307' to '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/framework.pid'
I0512 23:25:58.102713 8174 hierarchical.cpp:850] Updated allocation of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 from cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] to cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000]
I0512 23:25:58.102960 8177 slave.cpp:1898] Authorizing task 'ddb637d7-7789-402c-92bc-4932eca15b80' for framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.102987 8177 slave.cpp:6593] Authorizing framework principal 'test-principal' to launch task ddb637d7-7789-402c-92bc-4932eca15b80
I0512 23:25:58.103233 8179 slave.cpp:2085] Launching task 'ddb637d7-7789-402c-92bc-4932eca15b80' for framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.103478 8179 paths.cpp:573] Trying to chown '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80/runs/da3e8aa8-98e7-4e72-a8fd-5d0bae960014' to user 'root'
I0512 23:25:58.103608 8179 slave.cpp:7296] Checkpointing ExecutorInfo to '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80/executor.info'
I0512 23:25:58.103868 8179 slave.cpp:6933] Launching executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 with resources cpus(*)(allocated: *):0.1; mem(*)(allocated: *):32 in work directory '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80/runs/da3e8aa8-98e7-4e72-a8fd-5d0bae960014'
I0512 23:25:58.104085 8179 slave.cpp:7324] Checkpointing TaskInfo to '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80/runs/da3e8aa8-98e7-4e72-a8fd-5d0bae960014/tasks/ddb637d7-7789-402c-92bc-4932eca15b80/task.info'
I0512 23:25:58.104256 8179 slave.cpp:2314] Queued task 'ddb637d7-7789-402c-92bc-4932eca15b80' for executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.104290 8179 slave.cpp:875] Successfully attached file '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80/runs/da3e8aa8-98e7-4e72-a8fd-5d0bae960014'
I0512 23:25:58.104465 8181 containerizer.cpp:1021] Starting container da3e8aa8-98e7-4e72-a8fd-5d0bae960014 for executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.104888 8176 cgroups.cpp:410] Creating cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014' for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.120307 8176 cgroups.cpp:473] Chown the cgroup at '/sys/fs/cgroup/cpu,cpuacct/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014' to user 'root' for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.120358 8176 cgroups.cpp:410] Creating cgroup at '/sys/fs/cgroup/memory/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014' for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.121373 8176 memory.cpp:479] Started listening for OOM events for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.121731 8176 memory.cpp:590] Started listening on 'low' memory pressure events for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.122074 8176 memory.cpp:590] Started listening on 'medium' memory pressure events for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.122426 8176 memory.cpp:590] Started listening on 'critical' memory pressure events for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.122447 8176 cgroups.cpp:473] Chown the cgroup at '/sys/fs/cgroup/memory/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014' to user 'root' for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.123072 8179 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 1056MB for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.123639 8179 memory.cpp:228] Updated 'memory.limit_in_bytes' to 1056MB for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.123968 8179 cpu.cpp:101] Updated 'cpu.shares' to 2150 (cpus 2.1) for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.124692 8176 containerizer.cpp:1641] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"command":{"arguments":["mesos-executor","--launcher_dir=\/home\/admin\/workspace\/mesos\/Mesos_CI-build\/FLAG\/SSL\/label\/mesos-ec2-debian-8\/mesos\/build\/src"],"shell":false,"value":"\/home\/admin\/workspace\/mesos\/Mesos_CI-build\/FLAG\/SSL\/label\/mesos-ec2-debian-8\/mesos\/build\/src\/mesos-executor"},"environment":{"variables":[{"name":"LIBPROCESS_PORT","type":"VALUE","value":"0"},{"name":"MESOS_AGENT_ENDPOINT","type":"VALUE","value":"172.16.10.226:36307"},{"name":"MESOS_CHECKPOINT","type":"VALUE","value":"1"},{"name":"MESOS_DIRECTORY","type":"VALUE","value":"\/tmp\/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z\/slaves\/7bb1bfc2-26a3-45fe-9635-079d48114602-S0\/frameworks\/7bb1bfc2-26a3-45fe-9635-079d48114602-0000\/executors\/ddb637d7-7789-402c-92bc-4932eca15b80\/runs\/da3e8aa8-98e7-4e72-a8fd-5d0bae960014"},{"name":"MESOS_EXECUTOR_AUTHENTICATION_TOKEN","type":"VALUE","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjaWQiOiJkYTNlOGFhOC05OGU3LTRlNzItYThmZC01ZDBiYWU5NjAwMTQiLCJlaWQiOiJkZGI2MzdkNy03Nzg5LTQwMmMtOTJiYy00OTMyZWNhMTViODAiLCJmaWQiOiI3YmIxYmZjMi0yNmEzLTQ1ZmUtOTYzNS0wNzlkNDgxMTQ2MDItMDAwMCJ9.Uf2hqOYYBXdVG8gQicK0f16AVtPli7Ff8haN1gQgAlY"},{"name":"MESOS_EXECUTOR_ID","type":"VALUE","value":"ddb637d7-7789-402c-92bc-4932eca15b80"},{"name":"MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD","type":"VALUE","value":"5secs"},{"name":"MESOS_FRAMEWORK_ID","type":"VALUE","value":"7bb1bfc2-26a3-45fe-9635-079d48114602-0000"},{"name":"MESOS_HTTP_COMMAND_EXECUTOR","type":"VALUE","value":"0"},{"name":"MESOS_RECOVERY_TIMEOUT","type":"VALUE","value":"15mins"},{"name":"MESOS_SLAVE_ID","type":"VALUE","value":"7bb1bfc2-26a3-45fe-9635-079d48114602-S0"},{"name":"MESOS_SLAVE_PID","type":"VALUE","value":"slave(467)@172.16.10.226:36307"},{"name":"MESOS_SUBSCRIPTION_BACKOFF_MAX","type":"VALUE","value":"2secs"},{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/tmp\/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z\/slaves\/7bb1bfc2-26a3-45fe-9635-079d48114602-S0\/frameworks\/7bb1bfc2-26a3-45fe-9635-079d48114602-0000\/executors\/ddb637d7-7789-402c-92bc-4932eca15b80\/runs\/da3e8aa8-98e7-4e72-a8fd-5d0bae960014"}]},"user":"root","working_directory":"\/tmp\/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z\/slaves\/7bb1bfc2-26a3-45fe-9635-079d48114602-S0\/frameworks\/7bb1bfc2-26a3-45fe-9635-079d48114602-0000\/executors\/ddb637d7-7789-402c-92bc-4932eca15b80\/runs\/da3e8aa8-98e7-4e72-a8fd-5d0bae960014"}" --pipe_read="35" --pipe_write="36" --runtime_directory="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/containers/da3e8aa8-98e7-4e72-a8fd-5d0bae960014" --unshare_namespace_mnt="false"'
I0512 23:25:58.124944 8179 linux_launcher.cpp:429] Launching container da3e8aa8-98e7-4e72-a8fd-5d0bae960014 and cloning with namespaces
I0512 23:25:58.128991 8176 containerizer.cpp:1740] Checkpointing container's forked pid 21141 to '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80/runs/da3e8aa8-98e7-4e72-a8fd-5d0bae960014/pids/forked.pid'
I0512 23:25:58.131134 8179 fetcher.cpp:353] Starting to fetch URIs for container: da3e8aa8-98e7-4e72-a8fd-5d0bae960014, directory: /tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80/runs/da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.195901 21175 exec.cpp:162] Version: 1.4.0
I0512 23:25:58.199352 8178 master.cpp:5013] Processing KILL call for task 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.199395 8178 master.cpp:5086] Telling agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal) to kill task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.199467 8178 slave.cpp:2831] Asked to kill task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
W0512 23:25:58.199489 8178 slave.cpp:2919] Transitioning the state of task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 to TASK_KILLED because the executor is not registered
I0512 23:25:58.199537 8178 slave.cpp:4219] Handling status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 from @0.0.0.0:0
I0512 23:25:58.200623 8174 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 32MB for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.201207 8174 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1) for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.201594 8179 status_update_manager.cpp:323] Received status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.201617 8179 status_update_manager.cpp:500] Creating StatusUpdate stream for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.201812 8179 status_update_manager.cpp:834] Checkpointing UPDATE for status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.201915 8179 status_update_manager.cpp:377] Forwarding update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 to the agent
I0512 23:25:58.202008 8177 slave.cpp:4659] Forwarding the update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 to master@172.16.10.226:36307
I0512 23:25:58.202136 8177 slave.cpp:4553] Status update manager successfully handled status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.202164 8176 master.cpp:6469] Status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 from agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.202183 8176 master.cpp:6537] Forwarding status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.202260 8176 master.cpp:8568] Updating the state of task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
I0512 23:25:58.202428 8175 sched.cpp:1041] Scheduler::statusUpdate took 49854ns
I0512 23:25:58.202551 8179 hierarchical.cpp:1114] Recovered cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: {}) on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 from framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.202603 8155 slave.cpp:794] Agent terminating
I0512 23:25:58.202621 8176 master.cpp:5194] Processing ACKNOWLEDGE call 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5 for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307 on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0
I0512 23:25:58.202646 8176 master.cpp:8662] Removing task ddb637d7-7789-402c-92bc-4932eca15b80 with resources cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.202738 8176 master.cpp:1313] Agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal) disconnected
I0512 23:25:58.202754 8176 master.cpp:3197] Disconnecting agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.202775 8176 master.cpp:3216] Deactivating agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.202826 8180 hierarchical.cpp:653] Agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 deactivated
I0512 23:25:58.202837 8155 containerizer.cpp:221] Using isolation: cgroups/cpu,cgroups/mem,filesystem/posix,network/cni
I0512 23:25:58.205909 8155 linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0512 23:25:58.206176 8155 provisioner.cpp:249] Using default backend 'copy'
I0512 23:25:58.217450 8155 cluster.cpp:448] Creating default 'local' authorizer
I0512 23:25:58.217885 8181 slave.cpp:225] Mesos agent started on (468)@172.16.10.226:36307
I0512 23:25:58.217905 8181 slave.cpp:226] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_secret_key="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/executor_secret_key" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" --launcher="linux" --launcher_dir="/home/admin/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-debian-8/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z"
I0512 23:25:58.218153 8181 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/credential'
I0512 23:25:58.218297 8181 slave.cpp:258] Agent using credential for: test-principal
I0512 23:25:58.218313 8181 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_vZZubO/http_credentials'
I0512 23:25:58.218407 8181 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor'
I0512 23:25:58.218441 8181 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor'
I0512 23:25:58.218489 8181 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
I0512 23:25:58.218564 8181 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly'
I0512 23:25:58.218623 8181 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
I0512 23:25:58.218684 8181 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite'
I0512 23:25:58.219270 8181 slave.cpp:529] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I0512 23:25:58.219300 8181 slave.cpp:537] Agent attributes: [ ]
I0512 23:25:58.219305 8181 slave.cpp:542] Agent hostname: ip-172-16-10-226.ec2.internal
I0512 23:25:58.219377 8175 status_update_manager.cpp:177] Pausing sending status updates
I0512 23:25:58.219593 8175 state.cpp:62] Recovering state from '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta'
I0512 23:25:58.219624 8175 state.cpp:710] No committed checkpointed resources found at '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/resources/resources.info'
W0512 23:25:58.220283 8175 state.cpp:552] Failed to find 'libprocess.pid' or 'http.marker' for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014 of executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.220674 8177 fetcher.cpp:94] Clearing fetcher cache
I0512 23:25:58.220701 8177 slave.cpp:6065] Recovering framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.220715 8177 slave.cpp:7041] Recovering executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.220958 8175 status_update_manager.cpp:203] Recovering status update manager
I0512 23:25:58.220975 8175 status_update_manager.cpp:211] Recovering executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.221006 8175 status_update_manager.cpp:500] Creating StatusUpdate stream for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.221135 8175 status_update_manager.cpp:810] Replaying status update stream for task ddb637d7-7789-402c-92bc-4932eca15b80
I0512 23:25:58.221364 8175 containerizer.cpp:608] Recovering containerizer
I0512 23:25:58.221410 8175 containerizer.cpp:664] Recovering container da3e8aa8-98e7-4e72-a8fd-5d0bae960014 for executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.222126 8181 linux_launcher.cpp:291] Recovered container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.223320 8174 memory.cpp:479] Started listening for OOM events for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.223683 8174 memory.cpp:590] Started listening on 'low' memory pressure events for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.224064 8174 memory.cpp:590] Started listening on 'medium' memory pressure events for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.224414 8174 memory.cpp:590] Started listening on 'critical' memory pressure events for container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.225769 8174 provisioner.cpp:410] Provisioner recovery complete
I0512 23:25:58.226100 8176 slave.cpp:5924] Unable to reconnect to executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 because no pid or http checkpoint file was found
I0512 23:25:58.226421 8181 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.226447 8181 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.226464 8181 hierarchical.cpp:1434] Performed allocation for 1 agents in 64253ns
I0512 23:25:58.226510 8180 slave.cpp:4149] Cleaning up un-reregistered executors
I0512 23:25:58.226526 8180 slave.cpp:4167] Killing un-reregistered executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.226616 8180 slave.cpp:5974] Finished recovery
I0512 23:25:58.226814 8180 slave.cpp:6156] Querying resource estimator for oversubscribable resources
I0512 23:25:58.226924 8174 containerizer.cpp:2218] Destroying container da3e8aa8-98e7-4e72-a8fd-5d0bae960014 in RUNNING state
I0512 23:25:58.226953 8179 slave.cpp:922] New master detected at master@172.16.10.226:36307
I0512 23:25:58.226959 8178 status_update_manager.cpp:177] Pausing sending status updates
I0512 23:25:58.226974 8179 slave.cpp:957] Detecting new master
I0512 23:25:58.227042 8176 linux_launcher.cpp:505] Asked to destroy container da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.227067 8179 slave.cpp:6170] Received oversubscribable resources {} from the resource estimator
I0512 23:25:58.227432 8176 linux_launcher.cpp:548] Using freezer to destroy cgroup mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.228140 8174 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.229092 8176 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014 after 0ns
I0512 23:25:58.230167 8176 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.231338 8179 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014 after 0ns
I0512 23:25:58.231673 8179 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.231691 8179 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.231698 8179 hierarchical.cpp:1434] Performed allocation for 1 agents in 39937ns
I0512 23:25:58.231623 8176 slave.cpp:984] Authenticating with master master@172.16.10.226:36307
I0512 23:25:58.231724 8176 slave.cpp:995] Using default CRAM-MD5 authenticatee
I0512 23:25:58.231801 8180 authenticatee.cpp:121] Creating new client SASL connection
I0512 23:25:58.232172 8180 master.cpp:7475] Authenticating slave(468)@172.16.10.226:36307
I0512 23:25:58.232228 8179 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(975)@172.16.10.226:36307
I0512 23:25:58.232409 8179 authenticator.cpp:98] Creating new server SASL connection
I0512 23:25:58.232723 8179 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0512 23:25:58.232741 8179 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0512 23:25:58.232779 8179 authenticator.cpp:204] Received SASL authentication start
I0512 23:25:58.232825 8179 authenticator.cpp:326] Authentication requires more steps
I0512 23:25:58.232900 8179 authenticatee.cpp:259] Received SASL authentication step
I0512 23:25:58.232991 8181 authenticator.cpp:232] Received SASL authentication step
I0512 23:25:58.233017 8181 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-226.ec2.internal' server FQDN: 'ip-172-16-10-226.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I0512 23:25:58.233023 8181 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0512 23:25:58.233031 8181 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0512 23:25:58.233042 8181 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-226.ec2.internal' server FQDN: 'ip-172-16-10-226.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I0512 23:25:58.233057 8181 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0512 23:25:58.233065 8181 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0512 23:25:58.233080 8181 authenticator.cpp:318] Authentication success
I0512 23:25:58.233144 8174 authenticatee.cpp:299] Authentication success
I0512 23:25:58.233155 8179 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(975)@172.16.10.226:36307
I0512 23:25:58.233173 8181 master.cpp:7505] Successfully authenticated principal 'test-principal' at slave(468)@172.16.10.226:36307
I0512 23:25:58.233289 8178 slave.cpp:1079] Successfully authenticated with master master@172.16.10.226:36307
I0512 23:25:58.233378 8178 slave.cpp:1507] Will retry registration in 7.091261ms if necessary
I0512 23:25:58.233469 8174 master.cpp:5708] Received re-register agent message from agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(468)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.233526 8174 master.cpp:3659] Authorizing agent with principal 'test-principal'
I0512 23:25:58.233703 8178 master.cpp:5824] Re-registering agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(467)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.233804 8181 slave.cpp:1233] Re-registered with master master@172.16.10.226:36307
I0512 23:25:58.233831 8176 hierarchical.cpp:641] Agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 reactivated
I0512 23:25:58.233837 8178 master.cpp:6233] Sending updated checkpointed resources {} to agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(468)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.233850 8181 slave.cpp:1270] Forwarding total oversubscribed resources {}
I0512 23:25:58.233870 8177 status_update_manager.cpp:184] Resuming sending status updates
W0512 23:25:58.233891 8177 status_update_manager.cpp:191] Resending status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.233907 8177 status_update_manager.cpp:377] Forwarding update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 to the agent
I0512 23:25:58.233929 8176 master.cpp:6324] Received update of agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(468)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal) with total oversubscribed resources {}
I0512 23:25:58.233935 8181 slave.cpp:3232] Updating info for framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 with pid updated to scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.233970 8181 slave.cpp:6843] Checkpointing FrameworkInfo to '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/framework.info'
I0512 23:25:58.234148 8181 slave.cpp:6854] Checkpointing framework pid 'scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307' to '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/framework.pid'
I0512 23:25:58.234308 8181 slave.cpp:3293] Ignoring new checkpointed resources identical to the current version: {}
I0512 23:25:58.234318 8176 status_update_manager.cpp:184] Resuming sending status updates
W0512 23:25:58.234330 8176 status_update_manager.cpp:191] Resending status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.234346 8176 status_update_manager.cpp:377] Forwarding update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 to the agent
I0512 23:25:58.234344 8181 slave.cpp:4659] Forwarding the update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 to master@172.16.10.226:36307
I0512 23:25:58.234417 8181 slave.cpp:4659] Forwarding the update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 to master@172.16.10.226:36307
I0512 23:25:58.234468 8177 master.cpp:6469] Status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 from agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(468)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.234500 8177 master.cpp:6537] Forwarding status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
W0512 23:25:58.234557 8177 master.cpp:6505] Could not lookup task for status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 from agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(468)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.234611 8176 sched.cpp:1041] Scheduler::statusUpdate took 6392ns
I0512 23:25:58.234635 8177 master.cpp:6469] Status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 from agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(468)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.234668 8177 master.cpp:6537] Forwarding status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
W0512 23:25:58.234730 8177 master.cpp:6505] Could not lookup task for status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 from agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 at slave(468)@172.16.10.226:36307 (ip-172-16-10-226.ec2.internal)
I0512 23:25:58.234783 8178 sched.cpp:1041] Scheduler::statusUpdate took 6200ns
I0512 23:25:58.234802 8177 master.cpp:5194] Processing ACKNOWLEDGE call 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5 for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307 on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0
I0512 23:25:58.234859 8177 master.cpp:5194] Processing ACKNOWLEDGE call 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5 for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307 on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0
I0512 23:25:58.234943 8179 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.234989 8179 status_update_manager.cpp:834] Checkpointing ACK for status update TASK_KILLED (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.235018 8179 status_update_manager.cpp:531] Cleaning up status update stream for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.235134 8179 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.235154 8180 slave.cpp:3508] Status update manager successfully handled status update acknowledgement (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.235173 8180 slave.cpp:7277] Completing task ddb637d7-7789-402c-92bc-4932eca15b80
E0512 23:25:58.235195 8180 slave.cpp:3501] Failed to handle status update acknowledgement (UUID: 8bc7c820-634f-41e3-92d1-5d0e87f9dbc5) for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000: Cannot find the status update stream for task ddb637d7-7789-402c-92bc-4932eca15b80 of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.235505 8180 containerizer.cpp:2624] Container da3e8aa8-98e7-4e72-a8fd-5d0bae960014 has exited
I0512 23:25:58.235548 8174 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.235571 8174 hierarchical.cpp:1434] Performed allocation for 1 agents in 197793ns
I0512 23:25:58.235673 8180 master.cpp:7305] Sending 1 offers to framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.235843 8180 sched.cpp:933] Scheduler::resourceOffers took 11870ns
I0512 23:25:58.236008 8179 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.236026 8179 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.236032 8179 hierarchical.cpp:1434] Performed allocation for 1 agents in 39133ns
I0512 23:25:58.236165 8181 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.236181 8181 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.236187 8181 hierarchical.cpp:1434] Performed allocation for 1 agents in 37612ns
I0512 23:25:58.236348 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.236366 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.236373 8175 hierarchical.cpp:1434] Performed allocation for 1 agents in 56761ns
I0512 23:25:58.236565 8174 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.236582 8174 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.236589 8174 hierarchical.cpp:1434] Performed allocation for 1 agents in 39055ns
I0512 23:25:58.236737 8177 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.236752 8177 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.236759 8177 hierarchical.cpp:1434] Performed allocation for 1 agents in 39500ns
I0512 23:25:58.236901 8176 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.236919 8176 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.236932 8176 hierarchical.cpp:1434] Performed allocation for 1 agents in 44929ns
I0512 23:25:58.237177 8180 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.237197 8180 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.237207 8180 hierarchical.cpp:1434] Performed allocation for 1 agents in 49564ns
I0512 23:25:58.237354 8178 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.237377 8178 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.237390 8178 hierarchical.cpp:1434] Performed allocation for 1 agents in 51327ns
I0512 23:25:58.237568 8177 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.237584 8177 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.237591 8177 hierarchical.cpp:1434] Performed allocation for 1 agents in 38292ns
I0512 23:25:58.237725 8176 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.237746 8176 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.237758 8176 hierarchical.cpp:1434] Performed allocation for 1 agents in 45785ns
I0512 23:25:58.237915 8181 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.237936 8181 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.237947 8181 hierarchical.cpp:1434] Performed allocation for 1 agents in 44732ns
I0512 23:25:58.238070 8174 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.238085 8174 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.238092 8174 hierarchical.cpp:1434] Performed allocation for 1 agents in 34438ns
I0512 23:25:58.238199 8179 slave.cpp:6156] Querying resource estimator for oversubscribable resources
I0512 23:25:58.238315 8181 slave.cpp:6170] Received oversubscribable resources {} from the resource estimator
I0512 23:25:58.238381 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.238391 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.238397 8175 hierarchical.cpp:1434] Performed allocation for 1 agents in 29426ns
I0512 23:25:58.238528 8177 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.238550 8177 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.238562 8177 hierarchical.cpp:1434] Performed allocation for 1 agents in 52271ns
I0512 23:25:58.238696 8179 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.238716 8179 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.238729 8179 hierarchical.cpp:1434] Performed allocation for 1 agents in 45108ns
I0512 23:25:58.238863 8176 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.238879 8176 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.238886 8176 hierarchical.cpp:1434] Performed allocation for 1 agents in 35748ns
I0512 23:25:58.239022 8178 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.239043 8178 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.239053 8178 hierarchical.cpp:1434] Performed allocation for 1 agents in 48341ns
I0512 23:25:58.239198 8180 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.239217 8180 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.239228 8180 hierarchical.cpp:1434] Performed allocation for 1 agents in 44883ns
I0512 23:25:58.239365 8181 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.239385 8181 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.239398 8181 hierarchical.cpp:1434] Performed allocation for 1 agents in 45107ns
I0512 23:25:58.239545 8177 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.239562 8177 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.239567 8177 hierarchical.cpp:1434] Performed allocation for 1 agents in 35154ns
I0512 23:25:58.239691 8176 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.239706 8176 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.239713 8176 hierarchical.cpp:1434] Performed allocation for 1 agents in 37361ns
I0512 23:25:58.239852 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.239873 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.239886 8175 hierarchical.cpp:1434] Performed allocation for 1 agents in 45941ns
I0512 23:25:58.240015 8178 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.240034 8178 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.240046 8178 hierarchical.cpp:1434] Performed allocation for 1 agents in 42804ns
I0512 23:25:58.240200 8179 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.240221 8179 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.240232 8179 hierarchical.cpp:1434] Performed allocation for 1 agents in 52885ns
I0512 23:25:58.240396 8181 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.240411 8181 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.240418 8181 hierarchical.cpp:1434] Performed allocation for 1 agents in 36493ns
I0512 23:25:58.240559 8180 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.240578 8180 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.240587 8180 hierarchical.cpp:1434] Performed allocation for 1 agents in 43673ns
I0512 23:25:58.240732 8176 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.240751 8176 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.240763 8176 hierarchical.cpp:1434] Performed allocation for 1 agents in 44338ns
I0512 23:25:58.240886 8174 slave.cpp:6156] Querying resource estimator for oversubscribable resources
I0512 23:25:58.240917 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.240928 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.240934 8175 hierarchical.cpp:1434] Performed allocation for 1 agents in 36543ns
I0512 23:25:58.240957 8178 slave.cpp:6170] Received oversubscribable resources {} from the resource estimator
I0512 23:25:58.241106 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.241127 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.241137 8175 hierarchical.cpp:1434] Performed allocation for 1 agents in 41792ns
I0512 23:25:58.241271 8174 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.241286 8174 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.241292 8174 hierarchical.cpp:1434] Performed allocation for 1 agents in 34408ns
I0512 23:25:58.241431 8180 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.241451 8180 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.241461 8180 hierarchical.cpp:1434] Performed allocation for 1 agents in 43553ns
I0512 23:25:58.241580 8179 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.241600 8179 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.241612 8179 hierarchical.cpp:1434] Performed allocation for 1 agents in 45021ns
I0512 23:25:58.241739 8178 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.241760 8178 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.241772 8178 hierarchical.cpp:1434] Performed allocation for 1 agents in 43973ns
I0512 23:25:58.241924 8181 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.241940 8181 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.241947 8181 hierarchical.cpp:1434] Performed allocation for 1 agents in 40165ns
I0512 23:25:58.242095 8177 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.242115 8177 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.242128 8177 hierarchical.cpp:1434] Performed allocation for 1 agents in 48008ns
I0512 23:25:58.242266 8176 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.242286 8176 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.242298 8176 hierarchical.cpp:1434] Performed allocation for 1 agents in 47201ns
I0512 23:25:58.242430 8174 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.242447 8174 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.242460 8174 hierarchical.cpp:1434] Performed allocation for 1 agents in 40841ns
I0512 23:25:58.242612 8180 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.242631 8180 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.242641 8180 hierarchical.cpp:1434] Performed allocation for 1 agents in 44606ns
I0512 23:25:58.242774 8178 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.242795 8178 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.242805 8178 hierarchical.cpp:1434] Performed allocation for 1 agents in 49376ns
I0512 23:25:58.242957 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.242974 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.242980 8175 hierarchical.cpp:1434] Performed allocation for 1 agents in 35891ns
I0512 23:25:58.243134 8179 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.243150 8179 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.243157 8179 hierarchical.cpp:1434] Performed allocation for 1 agents in 40887ns
I0512 23:25:58.243299 8177 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.243314 8177 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.243325 8177 hierarchical.cpp:1434] Performed allocation for 1 agents in 39437ns
I0512 23:25:58.243458 8174 slave.cpp:6156] Querying resource estimator for oversubscribable resources
I0512 23:25:58.243482 8181 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.243494 8181 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.243505 8174 slave.cpp:6170] Received oversubscribable resources {} from the resource estimator
I0512 23:25:58.243506 8181 hierarchical.cpp:1434] Performed allocation for 1 agents in 36457ns
I0512 23:25:58.243669 8178 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.243686 8178 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.243695 8178 hierarchical.cpp:1434] Performed allocation for 1 agents in 41822ns
I0512 23:25:58.243850 8174 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.243868 8174 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.243881 8174 hierarchical.cpp:1434] Performed allocation for 1 agents in 42883ns
I0512 23:25:58.244037 8176 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.244053 8176 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.244060 8176 hierarchical.cpp:1434] Performed allocation for 1 agents in 41045ns
I0512 23:25:58.244179 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.244195 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.244207 8175 hierarchical.cpp:1434] Performed allocation for 1 agents in 41963ns
I0512 23:25:58.244366 8177 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.244385 8177 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.244398 8177 hierarchical.cpp:1434] Performed allocation for 1 agents in 43423ns
I0512 23:25:58.244554 8181 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.244575 8181 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.244590 8181 hierarchical.cpp:1434] Performed allocation for 1 agents in 49603ns
I0512 23:25:58.244735 8180 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.244755 8180 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.244765 8180 hierarchical.cpp:1434] Performed allocation for 1 agents in 49815ns
I0512 23:25:58.244902 8178 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.244920 8178 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.244928 8178 hierarchical.cpp:1434] Performed allocation for 1 agents in 43116ns
I0512 23:25:58.245059 8179 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.245079 8179 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.245090 8179 hierarchical.cpp:1434] Performed allocation for 1 agents in 54162ns
I0512 23:25:58.245208 8174 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.245227 8174 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.245240 8174 hierarchical.cpp:1434] Performed allocation for 1 agents in 45419ns
I0512 23:25:58.245368 8176 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.245386 8176 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.245398 8176 hierarchical.cpp:1434] Performed allocation for 1 agents in 43933ns
I0512 23:25:58.245532 8181 slave.cpp:5670] Current disk usage 33.89%. Max allowed age: 3.927900674497697days
I0512 23:25:58.245587 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.245607 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.245620 8175 hierarchical.cpp:1434] Performed allocation for 1 agents in 47985ns
I0512 23:25:58.245789 8180 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.245807 8180 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.245820 8180 hierarchical.cpp:1434] Performed allocation for 1 agents in 45833ns
I0512 23:25:58.245967 8181 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.245983 8181 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.245990 8181 hierarchical.cpp:1434] Performed allocation for 1 agents in 43393ns
I0512 23:25:58.246115 8176 slave.cpp:6156] Querying resource estimator for oversubscribable resources
I0512 23:25:58.246158 8175 hierarchical.cpp:1850] No allocations performed
I0512 23:25:58.246206 8175 hierarchical.cpp:1940] No inverse offers to send out!
I0512 23:25:58.246220 8175 hierarchical.cpp:1434] Performed allocation for 1 agents in 73107ns
E0512 23:25:58.246383 8176 slave.cpp:5150] Termination of executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 failed: Failed to kill all processes in the container: Timed out after 1mins
I0512 23:25:58.246407 8176 slave.cpp:5272] Cleaning up executor 'ddb637d7-7789-402c-92bc-4932eca15b80' of framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.246558 8174 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80/runs/da3e8aa8-98e7-4e72-a8fd-5d0bae960014' for gc 6.99999746608889days in the future
I0512 23:25:58.246604 8174 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80' for gc 6.99999746608889days in the future
I0512 23:25:58.246610 8176 slave.cpp:5360] Cleaning up framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.246629 8174 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80/runs/da3e8aa8-98e7-4e72-a8fd-5d0bae960014' for gc 6.99999746608889days in the future
I0512 23:25:58.246680 8181 status_update_manager.cpp:285] Closing status update streams for framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.246701 8174 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000/executors/ddb637d7-7789-402c-92bc-4932eca15b80' for gc 6.99999746608889days in the future
I0512 23:25:58.246724 8176 slave.cpp:6170] Received oversubscribable resources {} from the resource estimator
I0512 23:25:58.246742 8174 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000' for gc 6.99999746608889days in the future
I0512 23:25:58.246783 8174 gc.cpp:55] Scheduling '/tmp/SlaveRecoveryTest_0_KillTaskUnregisteredExecutor_Kek63Z/meta/slaves/7bb1bfc2-26a3-45fe-9635-079d48114602-S0/frameworks/7bb1bfc2-26a3-45fe-9635-079d48114602-0000' for gc 6.99999746608889days in the future
I0512 23:25:58.246933 8155 sched.cpp:2021] Asked to stop the driver
I0512 23:25:58.246994 8175 sched.cpp:1203] Stopping framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.247097 8181 master.cpp:7988] Processing TEARDOWN call for framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.247122 8181 master.cpp:8000] Removing framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.247133 8181 master.cpp:3160] Deactivating framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 (default) at scheduler-ec3707f3-9836-4c3c-a62c-bdb5fa3b2f69@172.16.10.226:36307
I0512 23:25:58.247174 8176 hierarchical.cpp:374] Deactivated framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.247409 8174 hierarchical.cpp:1114] Recovered cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: {}) on agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0 from framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.247509 8174 hierarchical.cpp:325] Removed framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
I0512 23:25:58.247508 8176 slave.cpp:3061] Asked to shut down framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000 by master@172.16.10.226:36307
I0512 23:25:58.247545 8176 slave.cpp:3076] Cannot shut down unknown framework 7bb1bfc2-26a3-45fe-9635-079d48114602-0000
../../src/tests/cluster.cpp:580: Failure
Value of: containers->empty()
Actual: false
Expected: true
Failed to destroy containers: { da3e8aa8-98e7-4e72-a8fd-5d0bae960014 }
I0512 23:25:58.248136 8155 slave.cpp:794] Agent terminating
I0512 23:25:58.250381 8155 master.cpp:1155] Master terminating
I0512 23:25:58.250588 8177 hierarchical.cpp:558] Removed agent 7bb1bfc2-26a3-45fe-9635-079d48114602-S0
I0512 23:25:58.257766 8176 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.258714 8178 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014 after 928us
I0512 23:25:58.259631 8178 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014
I0512 23:25:58.260583 8181 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff/da3e8aa8-98e7-4e72-a8fd-5d0bae960014 after 926976ns
I0512 23:25:58.262421 8180 cgroups.cpp:2692] Freezing cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff
I0512 23:25:58.364909 8180 cgroups.cpp:1405] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff after 102.461952ms
I0512 23:25:58.365866 8174 cgroups.cpp:2710] Thawing cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff
I0512 23:25:58.366747 8174 cgroups.cpp:1434] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos_test_705a6b72-83e4-43be-8775-c184d89bf3ff after 855040ns
[ FAILED ] SlaveRecoveryTest/0.KillTaskUnregisteredExecutor, where TypeParam = mesos::internal::slave::MesosContainerizer (320 ms)
{noformat}
> Multiple tests leave orphan containers.
> ---------------------------------------
>
> Key: MESOS-7506
> URL: https://issues.apache.org/jira/browse/MESOS-7506
> Project: Mesos
> Issue Type: Bug
> Components: containerization
> Reporter: Alexander Rukletsov
> Labels: containerizer, flaky-test, mesosphere
>
> I've observed a number of flaky tests that leave orphan containers upon cleanup. A typical log looks like this:
> {noformat}
> ../../src/tests/cluster.cpp:580: Failure
> Value of: containers->empty()
> Actual: false
> Expected: true
> Failed to destroy containers: { da3e8aa8-98e7-4e72-a8fd-5d0bae960014 }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)