You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Alexander Rukletsov (JIRA)" <ji...@apache.org> on 2017/06/01 17:25:04 UTC
[jira] [Updated] (MESOS-7601) Some container launch failures are
mistakenly treated as errors.
[ https://issues.apache.org/jira/browse/MESOS-7601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Rukletsov updated MESOS-7601:
---------------------------------------
Story Points: 3
Labels: containerizer mesosphere (was: )
Description:
I've observed a case when a scheduler stops (i.e. calls TEARDOWN) while some of its tasks are being launched. While this is a valid behaviour, the agent prints an error and increased container launch errors metrics.
Below are log excerpts for such framework, {{6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092}}.
*Master log*
{noformat}
[centos@ip-172-31-6-200 ~]$ journalctl _PID=29716 --since "2 hours ago" --no-pager | grep "6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092"
Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.226218 29724 master.cpp:6072] Updating info for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.226356 29728 hierarchical.cpp:274] Added framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.226405 29728 hierarchical.cpp:379] Deactivated framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.228570 29728 hierarchical.cpp:343] Activated framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.246068 29721 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.247851 29721 master.cpp:7194] Sending 1 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.912937 29728 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509464 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:32:59 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:59.804184 29727 master.cpp:7105] Sending 2 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:32:59 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:59.804411 29727 master.cpp:7194] Sending 2 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.248924 29721 master.cpp:7105] Sending 2 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.249289 29721 master.cpp:7194] Sending 2 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.249724 29721 master.cpp:3851] Processing ACCEPT call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509469 ] on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122) for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.250141 29721 master.cpp:3851] Processing ACCEPT call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509470 ] on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051 (172.31.7.202) for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.252516 29721 master.cpp:4501] Launching task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.254794 29721 master.cpp:4501] Launching task 1 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051 (172.31.7.202)
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.255506 29725 hierarchical.cpp:855] Updated allocation of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 from ports(*)(allocated: *):[1025-2180, 2182-3887, 3889-5049, 5052-6999, 7002-7076, 7078-7079, 7082-7198, 7200-8079, 8082-8180, 8183-8608, 8610-8982, 8984-9002, 9006-9041, 9043-9159, 9161-10012, 10016-10666, 10669-11682, 11685-11872, 11874-21620, 21622-30643, 30646-32000]; cpus(*)(allocated: *):1.1; mem(*)(allocated: *):6456; disk(*)(allocated: *):46032; disk(*)(allocated: *)[MOUNT:/dcos/volume3]:47540; disk(*)(allocated: *)[MOUNT:/dcos/volume4]:47540 to ports(*)(allocated: *):[1025-2180, 2182-3887, 3889-5049, 5052-6999, 7002-7076, 7078-7079, 7082-7198, 7200-8079, 8082-8180, 8183-8608, 8610-8982, 8984-9002, 9006-9041, 9043-9159, 9161-10012, 10016-10666, 10669-11682, 11685-11872, 11874-21620, 21622-30643, 30646-32000]; cpus(*)(allocated: *):1.1; mem(*)(allocated: *):6456; disk(*)(allocated: *):46032; disk(*)(allocated: *)[MOUNT:/dcos/volume3]:47540; disk(*)(allocated: *)[MOUNT:/dcos/volume4]:47540
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.258015 29725 hierarchical.cpp:855] Updated allocation of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 from ports(*)(allocated: *):[1027-2180, 2182-3887, 3889-5049, 5052-6875, 6877-8079, 8082-8180, 8182-9299, 9301-9543, 9545-14041, 14043-15028, 15030-24844, 24846-32000]; disk(*)(allocated: *)[MOUNT:/dcos/volume0]:51042; disk(*)(allocated: *):43530; cpus(*)(allocated: *):1.08; mem(*)(allocated: *):9357 to ports(*)(allocated: *):[1027-2180, 2182-3887, 3889-5049, 5052-6875, 6877-8079, 8082-8180, 8182-9299, 9301-9543, 9545-14041, 14043-15028, 15030-24844, 24846-32000]; disk(*)(allocated: *)[MOUNT:/dcos/volume0]:51042; disk(*)(allocated: *):43530; cpus(*)(allocated: *):1.08; mem(*)(allocated: *):9357
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.322147 29724 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509473 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.322619 29724 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509474 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:03.113775 29722 master.cpp:6269] Status update TASK_RUNNING (UUID: 646de179-526f-48e4-8fe9-4deda3a09179) for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 from agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:03.113813 29722 master.cpp:6337] Forwarding status update TASK_RUNNING (UUID: 646de179-526f-48e4-8fe9-4deda3a09179) for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:03.117269 29722 master.cpp:8368] Updating the state of task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:03.216639 29723 master.cpp:5163] Processing ACKNOWLEDGE call 646de179-526f-48e4-8fe9-4deda3a09179 for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35
Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:05.410168 29722 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:05.410367 29722 master.cpp:7194] Sending 1 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:05.413863 29722 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509489 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:06 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:06.643015 29723 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:06 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:06.645283 29722 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509492 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:08 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:08.385871 29728 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:08 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:08.388234 29726 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509495 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:11 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:11.465273 29723 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:11 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:11.467978 29725 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509499 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:12 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:12.481941 29726 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:12 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:12.484498 29721 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509500 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.552039 29724 master.cpp:6269] Status update TASK_FINISHED (UUID: f49ba849-90cc-4110-b897-0d5d16a17588) for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 from agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.552119 29724 master.cpp:6337] Forwarding status update TASK_FINISHED (UUID: f49ba849-90cc-4110-b897-0d5d16a17588) for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.553474 29724 master.cpp:8368] Updating the state of task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (latest state: TASK_FINISHED, status update state: TASK_FINISHED)
Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.556002 29724 master.cpp:5163] Processing ACKNOWLEDGE call f49ba849-90cc-4110-b897-0d5d16a17588 for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35
Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.556046 29724 master.cpp:8462] Removing task 0 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.556318 29727 master.cpp:4911] Processing REVIVE call for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.556380 29727 hierarchical.cpp:1260] Revived offers for roles { * } of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.607833 29724 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.611508 29728 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509503 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:17 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:17.590775 29725 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:17 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:17.592618 29723 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509504 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:18 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:18.153723 29725 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:18 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:18.155370 29723 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509505 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:22 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:22.695742 29722 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:22 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:22.697412 29724 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509512 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:23 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:23.714365 29722 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:23 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:23.716039 29726 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509514 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:24.728322 29727 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:24.728845 29727 master.cpp:7194] Sending 1 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:24.729948 29728 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509515 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.295658 29723 master.cpp:7788] Processing TEARDOWN call for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.295702 29723 master.cpp:7800] Removing framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.295724 29723 master.cpp:3160] Deactivating framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.296236 29724 hierarchical.cpp:379] Deactivated framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.298550 29723 master.cpp:8368] Updating the state of task 1 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (latest state: TASK_KILLED, status update state: TASK_KILLED)
Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.298677 29723 master.cpp:8462] Removing task 1 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051 (172.31.7.202)
Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.298863 29726 hierarchical.cpp:326] Removed framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.299028 29723 master.cpp:7118] Master ignoring inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 because the framework has terminated or is inactive
{noformat}
*Agent log*
{noformat}
[centos@ip-172-31-7-202 ~]$ journalctl _PID=12073 --since "1 hour ago" --no-pager | grep -C 10 "failed to start:"
Jun 01 11:33:28 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:28.785028 12081 http.cpp:1115] HTTP GET for /slave(1)/state from 172.31.7.202:52855
Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:29.109624 12080 logfmt.cpp:177] dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:29.109526016+00:00 reason="Valid authorization token" uid="dcos_metrics_agent" object="/slave(1)/state" agent="dcos-metrics/1.1.0-64-g62702c3" authorizer="mesos-agent" action="GET" result=allow srcip=172.31.7.202 dstport=5051 srcport=49102
Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:29.109864 12081 http.cpp:1115] HTTP GET for /slave(1)/state from 172.31.7.202:49102 with User-Agent='dcos-metrics/1.1.0-64-g62702c3'
Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:29.347921 12084 logfmt.cpp:177] dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:29.347860992+00:00 reason="Valid authorization token" uid="dcos_metrics_agent" object="/slave(1)/containers" agent="dcos-metrics/1.1.0-64-g62702c3" authorizer="mesos-agent" action="GET" result=allow srcip=172.31.7.202 dstport=5051 srcport=49102
Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:29.348116 12077 http.cpp:1115] HTTP GET for /slave(1)/containers from 172.31.7.202:49102 with User-Agent='dcos-metrics/1.1.0-64-g62702c3'
Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: W0601 11:33:29.712091 12079 http.cpp:2160] Failed to get resource statistics for executor '"1"' of framework "6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092": Failed to run 'docker -H unix:///var/run/docker.sock inspect mesos-36a25adb-4ea2-49d3-a195-448cff1dc146-S2.5c1f53e4-fffe-4f3a-8847-c5a252a25d3b': exited with status 1; stderr='Error: No such object: mesos-36a25adb-4ea2-49d3-a195-448cff1dc146-S2.5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: '
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.298966 12083 slave.cpp:5548] Killing executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.299078 12083 docker.cpp:2123] Destroying container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.299108 12083 docker.cpp:2165] Destroying container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b in PULLING state
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: E0601 11:33:30.415652 12082 slave.cpp:5041] Container '5c1f53e4-fffe-4f3a-8847-c5a252a25d3b' for executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 failed to start: future discarded
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: E0601 11:33:30.415765 12082 slave.cpp:5148] Termination of executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 failed: unknown container
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.415794 12082 slave.cpp:5261] Cleaning up executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: W0601 11:33:30.415937 12077 composing.cpp:638] Attempted to destroy unknown container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.415966 12082 slave.cpp:5349] Cleaning up framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.415992 12077 gc.cpp:55] Scheduling '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092/executors/1/runs/5c1f53e4-fffe-4f3a-8847-c5a252a25d3b' for gc 1.99999518647111days in the future
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.416050 12082 status_update_manager.cpp:285] Closing status update streams for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.416061 12077 gc.cpp:55] Scheduling '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092/executors/1' for gc 1.99999518583407days in the future
Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.416138 12077 gc.cpp:55] Scheduling '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092' for gc 1.99999518486222days in the future
Jun 01 11:33:31 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:31.574026 12079 logfmt.cpp:177] dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:31.573729024+00:00 reason="Valid authorization token" uid="dcos_navstar_agent" object="/slave(1)/state" authorizer="mesos-agent" action="GET" result=allow srcip=172.31.7.202 dstport=5051 srcport=52855
Jun 01 11:33:31 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:31.574285 12079 http.cpp:1115] HTTP GET for /slave(1)/state from 172.31.7.202:52855
{noformat}
> Some container launch failures are mistakenly treated as errors.
> ----------------------------------------------------------------
>
> Key: MESOS-7601
> URL: https://issues.apache.org/jira/browse/MESOS-7601
> Project: Mesos
> Issue Type: Bug
> Components: containerization
> Affects Versions: 1.3.0
> Reporter: Alexander Rukletsov
> Labels: containerizer, mesosphere
>
> I've observed a case when a scheduler stops (i.e. calls TEARDOWN) while some of its tasks are being launched. While this is a valid behaviour, the agent prints an error and increased container launch errors metrics.
> Below are log excerpts for such framework, {{6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092}}.
> *Master log*
> {noformat}
> [centos@ip-172-31-6-200 ~]$ journalctl _PID=29716 --since "2 hours ago" --no-pager | grep "6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092"
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.226218 29724 master.cpp:6072] Updating info for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.226356 29728 hierarchical.cpp:274] Added framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.226405 29728 hierarchical.cpp:379] Deactivated framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.228570 29728 hierarchical.cpp:343] Activated framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.246068 29721 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.247851 29721 master.cpp:7194] Sending 1 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:58.912937 29728 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509464 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:59 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:59.804184 29727 master.cpp:7105] Sending 2 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:59 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:32:59.804411 29727 master.cpp:7194] Sending 2 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.248924 29721 master.cpp:7105] Sending 2 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.249289 29721 master.cpp:7194] Sending 2 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.249724 29721 master.cpp:3851] Processing ACCEPT call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509469 ] on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122) for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.250141 29721 master.cpp:3851] Processing ACCEPT call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509470 ] on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051 (172.31.7.202) for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.252516 29721 master.cpp:4501] Launching task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.254794 29721 master.cpp:4501] Launching task 1 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051 (172.31.7.202)
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.255506 29725 hierarchical.cpp:855] Updated allocation of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 from ports(*)(allocated: *):[1025-2180, 2182-3887, 3889-5049, 5052-6999, 7002-7076, 7078-7079, 7082-7198, 7200-8079, 8082-8180, 8183-8608, 8610-8982, 8984-9002, 9006-9041, 9043-9159, 9161-10012, 10016-10666, 10669-11682, 11685-11872, 11874-21620, 21622-30643, 30646-32000]; cpus(*)(allocated: *):1.1; mem(*)(allocated: *):6456; disk(*)(allocated: *):46032; disk(*)(allocated: *)[MOUNT:/dcos/volume3]:47540; disk(*)(allocated: *)[MOUNT:/dcos/volume4]:47540 to ports(*)(allocated: *):[1025-2180, 2182-3887, 3889-5049, 5052-6999, 7002-7076, 7078-7079, 7082-7198, 7200-8079, 8082-8180, 8183-8608, 8610-8982, 8984-9002, 9006-9041, 9043-9159, 9161-10012, 10016-10666, 10669-11682, 11685-11872, 11874-21620, 21622-30643, 30646-32000]; cpus(*)(allocated: *):1.1; mem(*)(allocated: *):6456; disk(*)(allocated: *):46032; disk(*)(allocated: *)[MOUNT:/dcos/volume3]:47540; disk(*)(allocated: *)[MOUNT:/dcos/volume4]:47540
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.258015 29725 hierarchical.cpp:855] Updated allocation of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 from ports(*)(allocated: *):[1027-2180, 2182-3887, 3889-5049, 5052-6875, 6877-8079, 8082-8180, 8182-9299, 9301-9543, 9545-14041, 14043-15028, 15030-24844, 24846-32000]; disk(*)(allocated: *)[MOUNT:/dcos/volume0]:51042; disk(*)(allocated: *):43530; cpus(*)(allocated: *):1.08; mem(*)(allocated: *):9357 to ports(*)(allocated: *):[1027-2180, 2182-3887, 3889-5049, 5052-6875, 6877-8079, 8082-8180, 8182-9299, 9301-9543, 9545-14041, 14043-15028, 15030-24844, 24846-32000]; disk(*)(allocated: *)[MOUNT:/dcos/volume0]:51042; disk(*)(allocated: *):43530; cpus(*)(allocated: *):1.08; mem(*)(allocated: *):9357
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.322147 29724 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509473 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:01.322619 29724 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509474 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:03.113775 29722 master.cpp:6269] Status update TASK_RUNNING (UUID: 646de179-526f-48e4-8fe9-4deda3a09179) for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 from agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:03.113813 29722 master.cpp:6337] Forwarding status update TASK_RUNNING (UUID: 646de179-526f-48e4-8fe9-4deda3a09179) for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:03.117269 29722 master.cpp:8368] Updating the state of task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:03.216639 29723 master.cpp:5163] Processing ACKNOWLEDGE call 646de179-526f-48e4-8fe9-4deda3a09179 for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35
> Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:05.410168 29722 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:05.410367 29722 master.cpp:7194] Sending 1 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:05.413863 29722 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509489 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:06 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:06.643015 29723 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:06 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:06.645283 29722 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509492 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:08 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:08.385871 29728 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:08 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:08.388234 29726 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509495 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:11 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:11.465273 29723 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:11 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:11.467978 29725 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509499 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:12 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:12.481941 29726 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:12 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:12.484498 29721 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509500 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.552039 29724 master.cpp:6269] Status update TASK_FINISHED (UUID: f49ba849-90cc-4110-b897-0d5d16a17588) for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 from agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.552119 29724 master.cpp:6337] Forwarding status update TASK_FINISHED (UUID: f49ba849-90cc-4110-b897-0d5d16a17588) for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.553474 29724 master.cpp:8368] Updating the state of task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (latest state: TASK_FINISHED, status update state: TASK_FINISHED)
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.556002 29724 master.cpp:5163] Processing ACKNOWLEDGE call f49ba849-90cc-4110-b897-0d5d16a17588 for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.556046 29724 master.cpp:8462] Removing task 0 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.556318 29727 master.cpp:4911] Processing REVIVE call for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.556380 29727 hierarchical.cpp:1260] Revived offers for roles { * } of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.607833 29724 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:16.611508 29728 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509503 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:17 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:17.590775 29725 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:17 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:17.592618 29723 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509504 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:18 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:18.153723 29725 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:18 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:18.155370 29723 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509505 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:22 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:22.695742 29722 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:22 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:22.697412 29724 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509512 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:23 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:23.714365 29722 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:23 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:23.716039 29726 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509514 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:24.728322 29727 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:24.728845 29727 master.cpp:7194] Sending 1 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:24.729948 29728 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509515 ] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.295658 29723 master.cpp:7788] Processing TEARDOWN call for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.295702 29723 master.cpp:7800] Removing framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.295724 29723 master.cpp:3160] Deactivating framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.296236 29724 hierarchical.cpp:379] Deactivated framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.298550 29723 master.cpp:8368] Updating the state of task 1 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (latest state: TASK_KILLED, status update state: TASK_KILLED)
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.298677 29723 master.cpp:8462] Removing task 1 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051 (172.31.7.202)
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.298863 29726 hierarchical.cpp:326] Removed framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601 11:33:25.299028 29723 master.cpp:7118] Master ignoring inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 because the framework has terminated or is inactive
> {noformat}
> *Agent log*
> {noformat}
> [centos@ip-172-31-7-202 ~]$ journalctl _PID=12073 --since "1 hour ago" --no-pager | grep -C 10 "failed to start:"
> Jun 01 11:33:28 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:28.785028 12081 http.cpp:1115] HTTP GET for /slave(1)/state from 172.31.7.202:52855
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:29.109624 12080 logfmt.cpp:177] dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:29.109526016+00:00 reason="Valid authorization token" uid="dcos_metrics_agent" object="/slave(1)/state" agent="dcos-metrics/1.1.0-64-g62702c3" authorizer="mesos-agent" action="GET" result=allow srcip=172.31.7.202 dstport=5051 srcport=49102
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:29.109864 12081 http.cpp:1115] HTTP GET for /slave(1)/state from 172.31.7.202:49102 with User-Agent='dcos-metrics/1.1.0-64-g62702c3'
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:29.347921 12084 logfmt.cpp:177] dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:29.347860992+00:00 reason="Valid authorization token" uid="dcos_metrics_agent" object="/slave(1)/containers" agent="dcos-metrics/1.1.0-64-g62702c3" authorizer="mesos-agent" action="GET" result=allow srcip=172.31.7.202 dstport=5051 srcport=49102
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:29.348116 12077 http.cpp:1115] HTTP GET for /slave(1)/containers from 172.31.7.202:49102 with User-Agent='dcos-metrics/1.1.0-64-g62702c3'
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: W0601 11:33:29.712091 12079 http.cpp:2160] Failed to get resource statistics for executor '"1"' of framework "6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092": Failed to run 'docker -H unix:///var/run/docker.sock inspect mesos-36a25adb-4ea2-49d3-a195-448cff1dc146-S2.5c1f53e4-fffe-4f3a-8847-c5a252a25d3b': exited with status 1; stderr='Error: No such object: mesos-36a25adb-4ea2-49d3-a195-448cff1dc146-S2.5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: '
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.298966 12083 slave.cpp:5548] Killing executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.299078 12083 docker.cpp:2123] Destroying container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.299108 12083 docker.cpp:2165] Destroying container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b in PULLING state
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: E0601 11:33:30.415652 12082 slave.cpp:5041] Container '5c1f53e4-fffe-4f3a-8847-c5a252a25d3b' for executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 failed to start: future discarded
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: E0601 11:33:30.415765 12082 slave.cpp:5148] Termination of executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 failed: unknown container
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.415794 12082 slave.cpp:5261] Cleaning up executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: W0601 11:33:30.415937 12077 composing.cpp:638] Attempted to destroy unknown container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.415966 12082 slave.cpp:5349] Cleaning up framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.415992 12077 gc.cpp:55] Scheduling '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092/executors/1/runs/5c1f53e4-fffe-4f3a-8847-c5a252a25d3b' for gc 1.99999518647111days in the future
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.416050 12082 status_update_manager.cpp:285] Closing status update streams for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.416061 12077 gc.cpp:55] Scheduling '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092/executors/1' for gc 1.99999518583407days in the future
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:30.416138 12077 gc.cpp:55] Scheduling '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092' for gc 1.99999518486222days in the future
> Jun 01 11:33:31 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:31.574026 12079 logfmt.cpp:177] dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:31.573729024+00:00 reason="Valid authorization token" uid="dcos_navstar_agent" object="/slave(1)/state" authorizer="mesos-agent" action="GET" result=allow srcip=172.31.7.202 dstport=5051 srcport=52855
> Jun 01 11:33:31 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601 11:33:31.574285 12079 http.cpp:1115] HTTP GET for /slave(1)/state from 172.31.7.202:52855
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)