You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Tan N. Le (Jira)" <ji...@apache.org> on 2021/10/29 14:45:00 UTC
[jira] [Created] (MESOS-10233) containers were not cleaned up
properly and left running.
Tan N. Le created MESOS-10233:
---------------------------------
Summary: containers were not cleaned up properly and left running.
Key: MESOS-10233
URL: https://issues.apache.org/jira/browse/MESOS-10233
Project: Mesos
Issue Type: Task
Environment: aurora-scheduler 0.25.0
mesos 1.11.0
executor plugin: DCE [https://github.com/paypal/dce-go] based on mesos-go v0.002
Reporter: Tan N. Le
We observe that tasks were in STARTING and mesos tried to killed and cleaned them up dueo to OOM.
however, cgroup freezer files are not there and it assumes the containers are being cleaned.
the containers left running but the tasks reported lost in aurora/mesos.
================
aurora logs
I1026 05:16:55.886 [TaskEventBatchWorker, StateMachine] -b0b685ca-3ded-4304-a591-9241d06d7728 state machine transition INIT -> PENDING
I1026 05:16:55.984 [TaskGroupBatchWorker, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 state machine transition PENDING -> ASSIGNED
I1026 05:16:55.984 [TaskGroupBatchWorker, TaskAssignerImpl] Offer on agent gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com (id a76961ab-bba0-46e5-ae7b-b234057b7a33-S307) is being assigned task for mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728.
I1026 05:16:57.402 [Thread-1715969, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 in state TASK_STARTING from SOURCE_EXECUTOR
I1026 05:16:57.402 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 state machine transition ASSIGNED -> STARTING
W1026 05:18:03.376 [Thread-1717148, MesosCallbackHandler$MesosCallbackHandlerImpl] Lost executor compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 on slave a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 with status -1
I1026 05:18:03.377 [Thread-1717149, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 in state TASK_FAILED from SOURCE_AGENT with REASON_EXECUTOR_TERMINATED: Abnormal executor termination: Failed to kill all processes in the container: Timed out after 1mins
I1026 05:18:03.390 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 state machine transition STARTING -> FAILED
I1026 05:18:03.390 [TaskStatusHandlerImpl, StateManagerImpl] Task being rescheduled: mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
I1026 05:21:08.948 [Thread-1720928, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 in state TASK_RUNNING from SOURCE_EXECUTOR
I1026 05:21:08.949 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 state machine transition FAILED -> RUNNING (not allowed)
I1026 05:21:08.950 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 state machine transition FAILED -> LOST (not allowed)
=================
mesos-master logs
I1026 05:16:55.991168 29973 master.cpp:3873] Adding executor 'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' with resources cpus(allocated: aurora):0.1; mem(allocated: aurora):256 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:16:55.991324 29973 master.cpp:3899] Adding task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 with resources cpus(allocated: aurora):0.9; disk(allocated: aurora):100; mem(allocated: aurora):4096; ports(allocated: aurora):[10020-10020, 10076-10076, 10137-10137, 10139-10139, 10150-10150] of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:16:56.090255 29973 master.cpp:5035] Launching task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 with resources [\{"allocation_info":{"role":"aurora"},"name":"cpus","scalar":\{"value":0.9},"type":"SCALAR"},\{"allocation_info":{"role":"aurora"},"name":"disk","scalar":\{"value":100.0},"type":"SCALAR"},\{"allocation_info":{"role":"aurora"},"name":"mem","scalar":\{"value":4096.0},"type":"SCALAR"},\{"allocation_info":{"role":"aurora"},"name":"ports","ranges":\{"range":[{"begin":10020,"end":10020},\{"begin":10076,"end":10076},\{"begin":10137,"end":10137},\{"begin":10139,"end":10139},\{"begin":10150,"end":10150}]},"type":"RANGES"}] on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) on new executor
I1026 05:16:57.401727 29979 master.cpp:8160] Status update TASK_STARTING (Status UUID: 9d316a48-3656-11ec-8d10-42010ab432d2) for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:16:57.401800 29979 master.cpp:8217] Forwarding status update TASK_STARTING (Status UUID: 9d316a48-3656-11ec-8d10-42010ab432d2) for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000
I1026 05:16:57.401979 29979 master.cpp:11195] Updating the state of task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (latest state: TASK_STARTING, status update state: TASK_STARTING)
I1026 05:16:57.409750 29990 master.cpp:5870] Processing ACKNOWLEDGE call for status 9d316a48-3656-11ec-8d10-42010ab432d2 for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307
I1026 05:18:03.375757 29982 master.cpp:8497] Executor 'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com): wait status -1
I1026 05:18:03.375849 29982 master.cpp:11335] Removing executor 'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' with resources cpus(allocated: aurora):0.1; mem(allocated: aurora):256 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:18:03.376816 29990 master.cpp:8160] Status update TASK_FAILED (Status UUID: 7fbe4d73-2eed-4713-8774-8299f3df9736) for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:18:03.376888 29990 master.cpp:8217] Forwarding status update TASK_FAILED (Status UUID: 7fbe4d73-2eed-4713-8774-8299f3df9736) for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000
I1026 05:18:03.377179 29990 master.cpp:11195] Updating the state of task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (latest state: TASK_FAILED, status update state: TASK_FAILED)
I1026 05:18:03.505921 29974 master.cpp:5870] Processing ACKNOWLEDGE call for status 7fbe4d73-2eed-4713-8774-8299f3df9736 for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307
I1026 05:18:03.505990 29974 master.cpp:11295] Removing task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 with resources cpus(allocated: aurora):0.9; disk(allocated: aurora):100; mem(allocated: aurora):4096; ports(allocated: aurora):[10020-10020, 10076-10076, 10137-10137, 10139-10139, 10150-10150] of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:21:08.947870 29968 master.cpp:8160] Status update TASK_RUNNING (Status UUID: 332069b9-3657-11ec-8d10-42010ab432d2) for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:21:08.947954 29968 master.cpp:8217] Forwarding status update TASK_RUNNING (Status UUID: 332069b9-3657-11ec-8d10-42010ab432d2) for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000
W1026 05:21:08.948144 29968 master.cpp:8190] Could not lookup task for status update TASK_RUNNING (Status UUID: 332069b9-3657-11ec-8d10-42010ab432d2) for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051 (gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:21:08.949649 29980 master.cpp:5696] Processing KILL call for task 'mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083
W1026 05:21:08.949703 29980 master.cpp:5703] Cannot kill task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 because it is unknown; performing reconciliation
I1026 05:21:08.949753 29980 master.cpp:9140] Sending explicit reconciliation state TASK_LOST for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083
I1026 05:21:08.949921 29980 master.cpp:5870] Processing ACKNOWLEDGE call for status 332069b9-3657-11ec-8d10-42010ab432d2 for task mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at scheduler-bf829a38-5c60-46cb-82dc-9c7fc7be7130@10.180.52.175:8083 on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307
==================
mesos-slave logs
E1026 05:17:03.372925 849 memory.cpp:665] Failed to read 'memory.limit_in_bytes': No such file or directory
E1026 05:17:03.373068 849 memory.cpp:674] Failed to read 'memory.max_usage_in_bytes': No such file or directory
E1026 05:17:03.373138 849 memory.cpp:686] Failed to read 'memory.stat': No such file or directory
W1026 05:17:03.373395 849 linux_launcher.cpp:600] Couldn't find freezer cgroup for container f9e69838-169e-4837-be5e-11ef4860bb01 so assuming partially destroyed
W1026 05:17:03.373773 848 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:03.375150 839 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:03.377418 841 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:03.381808 832 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:03.390095 829 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:03.406415 830 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:03.438736 843 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:03.503955 836 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:03.632373 846 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:03.888777 845 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:04.401167 826 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:05.425781 838 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:07.475064 827 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:11.571998 847 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:19.764526 847 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
W1026 05:17:36.149070 841 cgroups.cpp:294] Removal of cgroup /sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with EBUSY, will try again
E1026 05:18:03.375078 835 slave.cpp:7085] Termination of executor 'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728' of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 failed: Failed to kill all processes in the container: Timed out after 1mins
I1026 05:16:39.505 [qtp1387308322-1609115, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-36cae1b6-0b73-4caf-9820-04af86396a92 state machine transition RUNNING -> KILLING
I1026 05:16:52.192 [Thread-1715877, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task mstestenv-msmaster4int-g-mppnodeweb-a-6-36cae1b6-0b73-4caf-9820-04af86396a92 in state TASK_KILLED from SOURCE_EXECUTOR
I1026 05:16:52.192 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-36cae1b6-0b73-4caf-9820-04af86396a92 state machine transition KILLING -> KILLED
I1026 05:16:52.243 [TaskEventBatchWorker, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 state machine transition INIT -> PENDING
I1026 05:16:52.300 [TaskGroupBatchWorker, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 state machine transition PENDING -> ASSIGNED
I1026 05:16:52.300 [TaskGroupBatchWorker, TaskAssignerImpl] Offer on agent gpma773817.gpf-prod.us-central1.gcp.dev.paypalinc.com (id a76961ab-bba0-46e5-ae7b-b234057b7a33-S470) is being assigned task for mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649.
I1026 05:16:53.643 [Thread-1715899, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 in state TASK_STARTING from SOURCE_EXECUTOR
I1026 05:16:53.643 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 state machine transition ASSIGNED -> STARTING
W1026 05:18:02.991 [Thread-1717141, MesosCallbackHandler$MesosCallbackHandlerImpl] Lost executor compose-mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 on slave a76961ab-bba0-46e5-ae7b-b234057b7a33-S470 with status -1
I1026 05:18:02.991 [Thread-1717142, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 in state TASK_FAILED from SOURCE_AGENT with REASON_EXECUTOR_TERMINATED: Abnormal executor termination: Failed to kill all processes in the container: Timed out after 1mins
I1026 05:18:03.094 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 state machine transition STARTING -> FAILED
I1026 05:18:03.095 [TaskStatusHandlerImpl, StateManagerImpl] Task being rescheduled: mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649
I1026 05:18:03.095 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 state machine transition INIT -> PENDING
I1026 05:18:03.227 [TaskGroupBatchWorker, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 state machine transition PENDING -> ASSIGNED
I1026 05:18:03.227 [TaskGroupBatchWorker, TaskAssignerImpl] Offer on agent gpma778297.gpf-prod.us-central1.gcp.dev.paypalinc.com (id a76961ab-bba0-46e5-ae7b-b234057b7a33-S609) is being assigned task for mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889.
I1026 05:18:04.520 [Thread-1717191, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 in state TASK_STARTING from SOURCE_EXECUTOR
I1026 05:18:04.520 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 state machine transition ASSIGNED -> STARTING
I1026 05:20:52.246 [Thread-1720630, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 in state TASK_RUNNING from SOURCE_EXECUTOR
I1026 05:20:52.246 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 state machine transition FAILED -> RUNNING (not allowed)
I1026 05:20:52.248 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 state machine transition FAILED -> LOST (not allowed)
I1026 05:22:50.010 [Thread-1723190, MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 in state TASK_RUNNING from SOURCE_EXECUTOR
I1026 05:22:50.010 [TaskStatusHandlerImpl, StateMachine] mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 state machine transition STARTING -> RUNNING
--
This message was sent by Atlassian Jira
(v8.3.4#803005)