You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "A. Dukhovniy (JIRA)" <ji...@apache.org> on 2017/10/04 16:46:00 UTC
[jira] [Updated] (MESOS-8051) Killing TASK_GROUP fail to kill some
tasks
[ https://issues.apache.org/jira/browse/MESOS-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
A. Dukhovniy updated MESOS-8051:
--------------------------------
Summary: Killing TASK_GROUP fail to kill some tasks (was: Killing TASK_GROUP fails to kill some tasks)
> Killing TASK_GROUP fail to kill some tasks
> ------------------------------------------
>
> Key: MESOS-8051
> URL: https://issues.apache.org/jira/browse/MESOS-8051
> Project: Mesos
> Issue Type: Bug
> Components: agent, executor
> Affects Versions: 1.4.0
> Reporter: A. Dukhovniy
> Priority: Critical
> Attachments: dcos-mesos-master.log.gz, dcos-mesos-slave.log.gz, screenshot-1.png
>
>
> When starting following pod definition via marathon:
> {code:java}
> {
> "id": "/simple-pod",
> "scaling": {
> "kind": "fixed",
> "instances": 3
> },
> "environment": {
> "PING": "PONG"
> },
> "containers": [
> {
> "name": "ct1",
> "resources": {
> "cpus": 0.1,
> "mem": 32
> },
> "image": {
> "kind": "MESOS",
> "id": "busybox"
> },
> "exec": {
> "command": {
> "shell": "while true; do echo the current time is $(date) > ./test-v1/clock; sleep 1; done"
> }
> },
> "volumeMounts": [
> {
> "name": "v1",
> "mountPath": "test-v1"
> }
> ]
> },
> {
> "name": "ct2",
> "resources": {
> "cpus": 0.1,
> "mem": 32
> },
> "exec": {
> "command": {
> "shell": "while true; do echo -n $PING ' '; cat ./etc/clock; sleep 1; done"
> }
> },
> "volumeMounts": [
> {
> "name": "v1",
> "mountPath": "etc"
> },
> {
> "name": "v2",
> "mountPath": "docker"
> }
> ]
> }
> ],
> "networks": [
> {
> "mode": "host"
> }
> ],
> "volumes": [
> {
> "name": "v1"
> },
> {
> "name": "v2",
> "host": "/var/lib/docker"
> }
> ]
> }
> {code}
> mesos will successfully kill all {{ct2}} containers but fail to kill all/some of the {{ct1}} containers. I've attached both master and agent logs. The interesting part starts after marathon issues 6 kills:
> {code:java}
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.209966 4746 master.cpp:5297] Processing KILL call for task 'simple-pod.instance-3c1098e5-a914-11e7-bcd5-e63c853d
> bf20.ct1' of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5.229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.210033 4746 master.cpp:5371] Telling agent bae11d5d-20c2-4d66-9ec3-773d1d717e58-S1 at slave(1)@10.0.1.207:5051 (
> 10.0.1.207) to kill task simple-pod.instance-3c1098e5-a914-11e7-bcd5-e63c853dbf20.ct1 of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5
> .229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.210471 4748 master.cpp:5297] Processing KILL call for task 'simple-pod.instance-3c1098e5-a914-11e7-bcd5-e63c853d
> bf20.ct2' of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5.229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.210518 4748 master.cpp:5371] Telling agent bae11d5d-20c2-4d66-9ec3-773d1d717e58-S1 at slave(1)@10.0.1.207:5051 (
> 10.0.1.207) to kill task simple-pod.instance-3c1098e5-a914-11e7-bcd5-e63c853dbf20.ct2 of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5
> .229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.210602 4748 master.cpp:5297] Processing KILL call for task 'simple-pod.instance-3c0ffca4-a914-11e7-bcd5-e63c853d
> bf20.ct1' of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5.229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.210639 4748 master.cpp:5371] Telling agent bae11d5d-20c2-4d66-9ec3-773d1d717e58-S1 at slave(1)@10.0.1.207:5051 (
> 10.0.1.207) to kill task simple-pod.instance-3c0ffca4-a914-11e7-bcd5-e63c853dbf20.ct1 of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5
> .229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.210932 4753 master.cpp:5297] Processing KILL call for task 'simple-pod.instance-3c0ffca4-a914-11e7-bcd5-e63c853d
> bf20.ct2' of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5.229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.210968 4753 master.cpp:5371] Telling agent bae11d5d-20c2-4d66-9ec3-773d1d717e58-S1 at slave(1)@10.0.1.207:5051 (
> 10.0.1.207) to kill task simple-pod.instance-3c0ffca4-a914-11e7-bcd5-e63c853dbf20.ct2 of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5
> .229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.211210 4747 master.cpp:5297] Processing KILL call for task 'simple-pod.instance-328cd633-a914-11e7-bcd5-e63c853d
> bf20.ct1' of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5.229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.211251 4747 master.cpp:5371] Telling agent bae11d5d-20c2-4d66-9ec3-773d1d717e58-S1 at slave(1)@10.0.1.207:5051 (
> 10.0.1.207) to kill task simple-pod.instance-328cd633-a914-11e7-bcd5-e63c853dbf20.ct1 of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5
> .229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.211474 4746 master.cpp:5297] Processing KILL call for task 'simple-pod.instance-328cd633-a914-11e7-bcd5-e63c853d
> bf20.ct2' of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5.229:15101
> Oct 04 14:58:25 ip-10-0-5-229.eu-central-1.compute.internal mesos-master[4708]: I1004 14:58:25.211514 4746 master.cpp:5371] Telling agent bae11d5d-20c2-4d66-9ec3-773d1d717e58-S1 at slave(1)@10.0.1.207:5051 (
> 10.0.1.207) to kill task simple-pod.instance-328cd633-a914-11e7-bcd5-e63c853dbf20.ct2 of framework bae11d5d-20c2-4d66-9ec3-773d1d717e58-0001 (marathon) at scheduler-c61c493c-728f-4bd9-be60-7373574749af@10.0.5
> .229:15101
> {code}
> All {{.ct1}} tasks fail eventually (~30s) where {{.ct2}} are successfully killed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)