You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Jie Yu (JIRA)" <ji...@apache.org> on 2018/08/10 22:05:00 UTC
[jira] [Created] (MESOS-9150) libprocess deadlock
Jie Yu created MESOS-9150:
-----------------------------
Summary: libprocess deadlock
Key: MESOS-9150
URL: https://issues.apache.org/jira/browse/MESOS-9150
Project: Mesos
Issue Type: Bug
Components: libprocs
Affects Versions: 1.7.0
Reporter: Jie Yu
Observed this in CI
{code}
[ RUN ] SlaveTest.KillTaskBetweenRunTaskParts
I0810 17:38:56.969700 18618 cluster.cpp:173] Creating default 'local' authorizer
I0810 17:38:56.970886 18642 master.cpp:413] Master b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8 (ip-172-16-10-30.ec2.internal) started on 172.16.10.30:42270
I0810 17:38:56.970909 18642 master.cpp:416] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="hierarchical" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/BSktU6/credentials" --filter_gpu_resources="true" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --memory_profiling="false" --min_allocatable_resources="cpus:0.01|mem:32" --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="100secs" --registry_strict="false" --require_agent_domain="false" --role_sorter="drf" --root_submissions="true" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/BSktU6/master" --zk_session_timeout="10secs"
I0810 17:38:56.971032 18642 master.cpp:465] Master only allowing authenticated frameworks to register
I0810 17:38:56.971042 18642 master.cpp:471] Master only allowing authenticated agents to register
I0810 17:38:56.971050 18642 master.cpp:477] Master only allowing authenticated HTTP frameworks to register
I0810 17:38:56.971055 18642 credentials.hpp:37] Loading credentials for authentication from '/tmp/BSktU6/credentials'
I0810 17:38:56.971112 18642 master.cpp:521] Using default 'crammd5' authenticator
I0810 17:38:56.971185 18642 http.cpp:977] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readonly'
I0810 17:38:56.971247 18642 http.cpp:977] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readwrite'
I0810 17:38:56.971313 18642 http.cpp:977] Creating default 'basic' HTTP authenticator for realm 'mesos-master-scheduler'
I0810 17:38:56.971355 18642 master.cpp:602] Authorization enabled
I0810 17:38:56.971472 18641 whitelist_watcher.cpp:77] No whitelist given
I0810 17:38:56.971596 18638 hierarchical.cpp:182] Initialized hierarchical allocator process
I0810 17:38:56.972218 18640 master.cpp:2083] Elected as the leading master!
I0810 17:38:56.972239 18640 master.cpp:1638] Recovering from registrar
I0810 17:38:56.972280 18640 registrar.cpp:339] Recovering registrar
I0810 17:38:56.972399 18642 registrar.cpp:383] Successfully fetched the registry (0B) in 97792ns
I0810 17:38:56.972452 18642 registrar.cpp:487] Applied 1 operations in 7042ns; attempting to update the registry
I0810 17:38:56.972585 18638 registrar.cpp:544] Successfully updated the registry in 113920ns
I0810 17:38:56.972620 18638 registrar.cpp:416] Successfully recovered registrar
I0810 17:38:56.972754 18638 master.cpp:1752] Recovered 0 agents from the registry (172B); allowing 10mins for agents to reregister
I0810 17:38:56.972770 18639 hierarchical.cpp:220] Skipping recovery of hierarchical allocator: nothing to recover
W0810 17:38:56.974472 18618 process.cpp:2810] Attempted to spawn already running process files@172.16.10.30:42270
I0810 17:38:56.974625 18618 cluster.cpp:479] Creating default 'local' authorizer
I0810 17:38:56.975348 18638 slave.cpp:268] Mesos agent started on (618)@172.16.10.30:42270
W0810 17:38:56.975487 18618 process.cpp:2810] Attempted to spawn already running process version@172.16.10.30:42270
I0810 17:38:56.975544 18638 slave.cpp:269] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/SlaveTest_KillTaskBetweenRunTaskParts_xkhHPH/store/appc" --authenticate_http_readonly="true" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/SlaveTest_KillTaskBetweenRunTaskParts_xkhHPH/credential" --default_role="*" --disallow_sharing_agent_pid_namespace="false" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/SlaveTest_KillTaskBetweenRunTaskParts_xkhHPH/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_reregistration_timeout="2secs" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/SlaveTest_KillTaskBetweenRunTaskParts_xkhHPH/fetch" --fetcher_cache_size="2GB" --fetcher_stall_timeout="1mins" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/SlaveTest_KillTaskBetweenRunTaskParts_xkhHPH/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="linux" --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/Plain/label/mesos-ec2-ubuntu-16.04/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --memory_profiling="false" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --reconfiguration_policy="equal" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/SlaveTest_KillTaskBetweenRunTaskParts_xkhHPH" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/SlaveTest_KillTaskBetweenRunTaskParts_VHd5mU" --zk_session_timeout="10secs"
I0810 17:38:56.975776 18638 credentials.hpp:86] Loading credential for authentication from '/tmp/SlaveTest_KillTaskBetweenRunTaskParts_xkhHPH/credential'
I0810 17:38:56.975829 18638 slave.cpp:301] Agent using credential for: test-principal
I0810 17:38:56.975844 18638 credentials.hpp:37] Loading credentials for authentication from '/tmp/SlaveTest_KillTaskBetweenRunTaskParts_xkhHPH/http_credentials'
I0810 17:38:56.975898 18638 http.cpp:977] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
I0810 17:38:56.975982 18638 disk_profile_adaptor.cpp:80] Creating default disk profile adaptor module
I0810 17:38:56.976478 18638 slave.cpp:616] Agent resources: [{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}]
I0810 17:38:56.976645 18618 sched.cpp:232] Version: 1.7.0
I0810 17:38:56.976825 18642 sched.cpp:336] New master detected at master@172.16.10.30:42270
I0810 17:38:56.976943 18642 sched.cpp:396] Authenticating with master master@172.16.10.30:42270
I0810 17:38:56.976971 18642 sched.cpp:403] Using default CRAM-MD5 authenticatee
I0810 17:38:56.977041 18644 authenticatee.cpp:121] Creating new client SASL connection
I0810 17:38:56.976716 18638 slave.cpp:624] Agent attributes: [ ]
I0810 17:38:56.977221 18638 slave.cpp:633] Agent hostname: ip-172-16-10-30.ec2.internal
I0810 17:38:56.977403 18644 task_status_update_manager.cpp:181] Pausing sending task status updates
I0810 17:38:56.977530 18643 master.cpp:9604] Authenticating scheduler-c70288a9-3620-4d8d-a723-9683ea36a20e@172.16.10.30:42270
I0810 17:38:56.977725 18643 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(1204)@172.16.10.30:42270
I0810 17:38:56.977962 18643 authenticator.cpp:98] Creating new server SASL connection
I0810 17:38:56.977843 18638 state.cpp:66] Recovering state from '/tmp/SlaveTest_KillTaskBetweenRunTaskParts_VHd5mU/meta'
I0810 17:38:56.978197 18644 slave.cpp:6906] Finished recovering checkpointed state from '/tmp/SlaveTest_KillTaskBetweenRunTaskParts_VHd5mU/meta', beginning agent recovery
I0810 17:38:56.978255 18644 task_status_update_manager.cpp:207] Recovering task status update manager
I0810 17:38:56.978416 18644 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0810 17:38:56.978435 18644 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0810 17:38:56.978468 18644 authenticator.cpp:204] Received SASL authentication start
I0810 17:38:56.978498 18644 authenticator.cpp:326] Authentication requires more steps
I0810 17:38:56.978538 18644 authenticatee.cpp:259] Received SASL authentication step
I0810 17:38:56.978595 18644 authenticator.cpp:232] Received SASL authentication step
I0810 17:38:56.978617 18644 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-30.ec2.internal' server FQDN: 'ip-172-16-10-30.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I0810 17:38:56.978627 18644 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0810 17:38:56.978636 18644 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0810 17:38:56.978646 18644 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-30.ec2.internal' server FQDN: 'ip-172-16-10-30.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I0810 17:38:56.978652 18644 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0810 17:38:56.978658 18644 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0810 17:38:56.978670 18644 authenticator.cpp:318] Authentication success
I0810 17:38:56.978724 18644 authenticatee.cpp:299] Authentication success
I0810 17:38:56.978744 18641 master.cpp:9627] Successfully authenticated principal 'test-principal' at scheduler-c70288a9-3620-4d8d-a723-9683ea36a20e@172.16.10.30:42270
I0810 17:38:56.978772 18644 sched.cpp:501] Successfully authenticated with master master@172.16.10.30:42270
I0810 17:38:56.978781 18644 sched.cpp:822] Sending SUBSCRIBE call to master@172.16.10.30:42270
I0810 17:38:56.978821 18644 sched.cpp:855] Will retry registration in 948.509945ms if necessary
I0810 17:38:56.978862 18641 master.cpp:2854] Received SUBSCRIBE call for framework 'default' at scheduler-c70288a9-3620-4d8d-a723-9683ea36a20e@172.16.10.30:42270
I0810 17:38:56.978884 18641 master.cpp:2155] Authorizing framework principal 'test-principal' to receive offers for roles '{ * }'
I0810 17:38:56.978976 18640 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(1204)@172.16.10.30:42270
I0810 17:38:56.978977 18639 master.cpp:2935] Subscribing framework default with checkpointing disabled and capabilities [ MULTI_ROLE, RESERVATION_REFINEMENT ]
I0810 17:38:56.979029 18641 composing.cpp:339] Finished recovering all containerizers
I0810 17:38:56.979091 18644 slave.cpp:7135] Recovering executors
I0810 17:38:56.979120 18644 slave.cpp:7288] Finished recovery
I0810 17:38:56.979753 18639 master.cpp:9826] Adding framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 (default) at scheduler-c70288a9-3620-4d8d-a723-9683ea36a20e@172.16.10.30:42270 with roles { } suppressed
I0810 17:38:56.979945 18639 hierarchical.cpp:306] Added framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.980010 18644 slave.cpp:1255] New master detected at master@172.16.10.30:42270
I0810 17:38:56.980026 18641 task_status_update_manager.cpp:181] Pausing sending task status updates
I0810 17:38:56.980062 18644 slave.cpp:1310] Detecting new master
I0810 17:38:56.980080 18640 sched.cpp:749] Framework registered with b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.980132 18639 hierarchical.cpp:1564] Performed allocation for 0 agents in 7182ns
I0810 17:38:56.980142 18640 sched.cpp:763] Scheduler::registered took 9726ns
I0810 17:38:56.983361 18641 slave.cpp:1337] Authenticating with master master@172.16.10.30:42270
I0810 17:38:56.983389 18641 slave.cpp:1346] Using default CRAM-MD5 authenticatee
I0810 17:38:56.983443 18641 authenticatee.cpp:121] Creating new client SASL connection
I0810 17:38:56.983798 18641 master.cpp:9604] Authenticating slave(618)@172.16.10.30:42270
I0810 17:38:56.983844 18641 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(1205)@172.16.10.30:42270
I0810 17:38:56.983916 18641 authenticator.cpp:98] Creating new server SASL connection
I0810 17:38:56.984282 18638 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0810 17:38:56.984305 18638 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0810 17:38:56.984338 18638 authenticator.cpp:204] Received SASL authentication start
I0810 17:38:56.984374 18638 authenticator.cpp:326] Authentication requires more steps
I0810 17:38:56.984412 18638 authenticatee.cpp:259] Received SASL authentication step
I0810 17:38:56.984457 18644 authenticator.cpp:232] Received SASL authentication step
I0810 17:38:56.984475 18644 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-30.ec2.internal' server FQDN: 'ip-172-16-10-30.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I0810 17:38:56.984484 18644 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0810 17:38:56.984494 18644 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0810 17:38:56.984503 18644 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-30.ec2.internal' server FQDN: 'ip-172-16-10-30.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I0810 17:38:56.984510 18644 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0810 17:38:56.984516 18644 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0810 17:38:56.984529 18644 authenticator.cpp:318] Authentication success
I0810 17:38:56.984573 18644 master.cpp:9627] Successfully authenticated principal 'test-principal' at slave(618)@172.16.10.30:42270
I0810 17:38:56.984583 18639 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(1205)@172.16.10.30:42270
I0810 17:38:56.984745 18638 authenticatee.cpp:299] Authentication success
I0810 17:38:56.987234 18638 slave.cpp:1429] Successfully authenticated with master master@172.16.10.30:42270
I0810 17:38:56.987325 18638 slave.cpp:1874] Will retry registration in 9.018152ms if necessary
I0810 17:38:56.987404 18638 master.cpp:6562] Received register agent message from slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal)
I0810 17:38:56.987476 18638 master.cpp:3921] Authorizing agent providing resources 'cpus:2; mem:1024; disk:1024; ports:[31000-32000]' with principal 'test-principal'
I0810 17:38:56.987634 18643 master.cpp:6629] Authorized registration of agent at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal)
I0810 17:38:56.987674 18643 master.cpp:6744] Registering agent at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal) with id b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0
I0810 17:38:56.987793 18643 registrar.cpp:487] Applied 1 operations in 47973ns; attempting to update the registry
I0810 17:38:56.987923 18640 registrar.cpp:544] Successfully updated the registry in 103936ns
I0810 17:38:56.987977 18640 master.cpp:6792] Admitted agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal)
I0810 17:38:56.988152 18640 master.cpp:6837] Registered agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal) with cpus:2; mem:1024; disk:1024; ports:[31000-32000]
I0810 17:38:56.988209 18640 slave.cpp:1476] Registered with master master@172.16.10.30:42270; given agent ID b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0
I0810 17:38:56.988221 18642 hierarchical.cpp:601] Added agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 (ip-172-16-10-30.ec2.internal) with cpus:2; mem:1024; disk:1024; ports:[31000-32000] (allocated: {})
I0810 17:38:56.988345 18640 slave.cpp:1496] Checkpointing SlaveInfo to '/tmp/SlaveTest_KillTaskBetweenRunTaskParts_VHd5mU/meta/slaves/b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0/slave.info'
I0810 17:38:56.988456 18642 hierarchical.cpp:1564] Performed allocation for 1 agents in 180586ns
I0810 17:38:56.988484 18642 task_status_update_manager.cpp:188] Resuming sending task status updates
I0810 17:38:56.988602 18642 master.cpp:9415] Sending offers [ b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-O0 ] to framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 (default) at scheduler-c70288a9-3620-4d8d-a723-9683ea36a20e@172.16.10.30:42270
I0810 17:38:56.988822 18640 slave.cpp:1545] Forwarding agent update {"operations":{},"resource_version_uuid":{"value":"YBzi4TmVSO6EzBRyGVtKJA=="},"slave_id":{"value":"b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0"},"update_oversubscribed_resources":false}
I0810 17:38:56.988849 18642 sched.cpp:919] Scheduler::resourceOffers took 30033ns
I0810 17:38:56.988987 18640 master.cpp:7896] Ignoring update on agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal) as it reports no changes
I0810 17:38:56.989372 18637 master.cpp:11406] Removing offer b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-O0
I0810 17:38:56.989480 18637 master.cpp:4424] Processing ACCEPT call for offers: [ b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-O0 ] on agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal) for framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 (default) at scheduler-c70288a9-3620-4d8d-a723-9683ea36a20e@172.16.10.30:42270
I0810 17:38:56.989529 18637 master.cpp:3498] Authorizing framework principal 'test-principal' to launch task 1
W0810 17:38:56.989959 18638 validation.cpp:1444] Executor 'default' for task '1' uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0810 17:38:56.989985 18638 validation.cpp:1456] Executor 'default' for task '1' uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0810 17:38:56.990083 18638 master.cpp:12126] Adding task 1 with resources cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; ports(allocated: *):[31000-32000] on agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal)
I0810 17:38:56.990186 18638 master.cpp:5396] Launching task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 (default) at scheduler-c70288a9-3620-4d8d-a723-9683ea36a20e@172.16.10.30:42270 with resources [{"allocation_info":{"role":"*"},"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}] on agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal) on new executor
I0810 17:38:56.990435 18638 slave.cpp:2011] Got assigned task '1' for framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.990896 18642 master.cpp:5994] Processing KILL call for task '1' of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 (default) at scheduler-c70288a9-3620-4d8d-a723-9683ea36a20e@172.16.10.30:42270
I0810 17:38:56.990926 18642 master.cpp:6072] Telling agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal) to kill task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 (default) at scheduler-c70288a9-3620-4d8d-a723-9683ea36a20e@172.16.10.30:42270
I0810 17:38:56.990983 18642 slave.cpp:3612] Asked to kill task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
W0810 17:38:56.991000 18642 slave.cpp:3655] Killing task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 before it was launched
I0810 17:38:56.991044 18642 slave.cpp:5266] Handling status update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 from @0.0.0.0:0
I0810 17:38:56.991109 18642 slave.cpp:6513] Cleaning up framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
E0810 17:38:56.991185 18642 slave.cpp:7456] Failed to find the mtime of '/tmp/SlaveTest_KillTaskBetweenRunTaskParts_VHd5mU/slaves/b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0/frameworks/b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000': Failed to stat '/tmp/SlaveTest_KillTaskBetweenRunTaskParts_VHd5mU/slaves/b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0/frameworks/b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000': No such file or directory
W0810 17:38:56.991228 18642 slave.cpp:5381] Could not find the executor for status update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.991283 18642 task_status_update_manager.cpp:289] Closing task status update streams for framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.991304 18642 task_status_update_manager.cpp:328] Received task status update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.991318 18642 task_status_update_manager.cpp:507] Creating StatusUpdate stream for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.991453 18642 task_status_update_manager.cpp:383] Forwarding task status update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 to the agent
I0810 17:38:56.991674 18642 task_status_update_manager.cpp:328] Received task status update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
W0810 17:38:56.991700 18642 task_status_update_manager.cpp:746] Ignoring duplicate task status update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.991735 18642 slave.cpp:5758] Forwarding the update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 to master@172.16.10.30:42270
I0810 17:38:56.991906 18642 slave.cpp:5651] Task status update manager successfully handled status update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
W0810 17:38:56.991955 18642 slave.cpp:2324] Ignoring running task '1' because the framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 does not exist
I0810 17:38:56.992188 18643 master.cpp:8332] Status update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 from agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal)
I0810 17:38:56.992238 18643 master.cpp:8389] Forwarding status update TASK_KILLED (Status UUID: a6f1a596-3dc2-4ab0-b09a-fc5a707162d3) for task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.992347 18643 master.cpp:10876] Updating the state of task 1 of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
I0810 17:38:56.992421 18639 sched.cpp:1027] Scheduler::statusUpdate took 24069ns
I0810 17:38:56.992547 18637 hierarchical.cpp:1236] Recovered cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; ports(allocated: *):[31000-32000] (total: cpus:2; mem:1024; disk:1024; ports:[31000-32000], allocated: {}) on agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 from framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.992580 18618 sched.cpp:2013] Asked to stop the driver
I0810 17:38:56.992610 18637 sched.cpp:1189] Stopping framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
I0810 17:38:56.993120 18643 master.cpp:1093] Master terminating
I0810 17:38:56.993249 18640 hierarchical.cpp:637] Removed agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0
I0810 17:38:56.993352 18643 master.cpp:10974] Removing task 1 with resources cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; ports(allocated: *):[31000-32000] of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 on agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal)
I0810 17:38:56.993432 18643 master.cpp:11005] Removing executor 'default' with resources [] of framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000 on agent b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-S0 at slave(618)@172.16.10.30:42270 (ip-172-16-10-30.ec2.internal)
I0810 17:38:56.993587 18638 hierarchical.cpp:359] Removed framework b91ffdf5-b472-4fcf-8004-4c5a0cf3e2f8-0000
E0810 17:38:56.993912 18642 process.cpp:3949]
**** DEADLOCK DETECTED! ****
You are waiting on process slave(618)@172.16.10.30:42270 that it is currently executing.
[ OK ] SlaveTest.KillTaskBetweenRunTaskParts (26 ms)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)