You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Greg Mann (JIRA)" <ji...@apache.org> on 2016/11/30 23:42:58 UTC
[jira] [Commented] (MESOS-6656) Nested containers can become
unkillable
[ https://issues.apache.org/jira/browse/MESOS-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710228#comment-15710228 ]
Greg Mann commented on MESOS-6656:
----------------------------------
Note that after about 12 hours, it looks like the agent's connection with the executor has been closed, but the same pattern continues. Agent logs:
{code}
Nov 30 12:39:49 ip-10-190-112-199 mesos-agent[6397]: I1130 12:39:49.639824 6409 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 30 12:39:49 ip-10-190-112-199 mesos-agent[6397]: W1130 12:39:49.639853 6409 slave.hpp:857] Unable to send event to executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000 (via HTTP): connection closed
Nov 30 12:39:49 ip-10-190-112-199 mesos-agent[6397]: I1130 12:39:49.639876 6409 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 30 12:39:49 ip-10-190-112-199 mesos-agent[6397]: W1130 12:39:49.639906 6409 slave.hpp:857] Unable to send event to executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000 (via HTTP): connection closed
{code}
> Nested containers can become unkillable
> ---------------------------------------
>
> Key: MESOS-6656
> URL: https://issues.apache.org/jira/browse/MESOS-6656
> Project: Mesos
> Issue Type: Bug
> Components: agent, containerization
> Reporter: Greg Mann
> Labels: nested
>
> An incident occurred recently in a cluster running a build of Mesos based on commit {{757319357471227c0a1e906076eae8f9aa2fdbd6}} from master. A task group of five tasks was launched via Marathon. After the tasks were launched, one of the containers quickly exited and was successfully destroyed. A couple minutes later, the task group was killed manually via Marathon, and the agent can then be seen repeatedly attempting to kill the tasks for hours. No calls to {{WAIT_NESTED_CONTAINER}} are visible in the agent logs, and the executor logs do not indicate at any point that the nested containers were launched successfully.
> Agent logs:
> {code}
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.890911 6406 slave.cpp:1539] Got assigned task group containing tasks [ dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] for framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.892299 6406 gc.cpp:83] Unscheduling '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000' from gc
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.892379 6406 gc.cpp:83] Unscheduling '/var/lib/mesos/slave/meta/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000' from gc
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.893131 6405 slave.cpp:1701] Launching task group containing tasks [ dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] for framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.893435 6405 paths.cpp:536] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91' to user 'root'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.898026 6405 slave.cpp:6179] Launching executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000 with resources cpus(*):0.1; mem(*):32; disk(*):10; ports(*):[21421-21425] in work directory '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.898731 6407 docker.cpp:1000] Skipping non-docker container
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.899050 6407 containerizer.cpp:938] Starting container 8750c2a7-8bef-4a69-8ef2-b873f884bf91 for executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.899909 6405 slave.cpp:1987] Queued task group containing tasks [ dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] for executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.907033 6405 memory.cpp:451] Started listening for OOM events for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.908336 6405 memory.cpp:562] Started listening on 'low' memory pressure events for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.909638 6405 memory.cpp:562] Started listening on 'medium' memory pressure events for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.910923 6405 memory.cpp:562] Started listening on 'critical' memory pressure events for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.912464 6405 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 32MB for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.914640 6405 memory.cpp:251] Updated 'memory.limit_in_bytes' to 32MB for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.915769 6405 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.917944 6405 cpu.cpp:121] Updated 'cpu.cfs_period_us' to 100ms and 'cpu.cfs_quota_us' to 10ms (cpus 0.1) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918515 6404 isolator_module.cpp:55] Container prepare: container_id[value: "8750c2a7-8bef-4a69-8ef2-b873f884bf91"] container_config[directory: "/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91" user: "root" executor_info { executor_id { value: "instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001" } resources { name: "cpus" type: SCALAR scalar { value: 0.1 } } resources { name: "mem" type: SCALAR scalar { value: 32 } } resources { name: "disk" type: SCALAR scalar { value: 10 } } resources { name: "ports" type: RANGES ranges { range { begin: 21421 end: 21425 } } role: "*" } command { value: "/opt/mesosphere/packages/mesos--a00089c894f82b6455d1db7c4267a742458020b3/libexec/mesos/mesos-default-executor" user: "root" shell: false arguments: "mesos-default-executor" } framework_id { value: "ce4bd8be-1198-4819-81d4-9a8439439741-0000" } container { type: MESOS network_infos { labels { labels { key: "rcluster.location" value: "globalqa.las2.test0" } labels { key: "rcluster.group" value: "dat" } labels { key: "rcluster.application" value: "scout" } } name: "rcluster-cni" port_mappings { host_port: 21421 container_port: 8080 protocol: "tcp" } port_mappings { host_port: 21422 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21423 container_port: 8093 protocol: "tcp" } port_mappings { host_port: 21424 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21425 container_port: 8090 protocol: "tcp" } } } labels { } type: DEFAULT } command_info { value: "/opt/mesosphere/packages/mesos--a00089c894f82b6455d1db7c4267a742458020b3/libexec/mesos/mesos-default-executor" user: "root" shell: false arguments: "mesos-default-executor" } container_info { type: MESOS network_infos { labels { labels { key: "rcluster.location" value: "globalqa.las2.test0" } labels { key: "rcluster.group" value: "dat" } labels { key: "rc
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: luster.application" value: "scout" } } name: "rcluster-cni" port_mappings { host_port: 21421 container_port: 8080 protocol: "tcp" } port_mappings { host_port: 21422 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21423 container_port: 8093 protocol: "tcp" } port_mappings { host_port: 21424 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21425 container_port: 8090 protocol: "tcp" } } } resources { name: "cpus" type: SCALAR scalar { value: 0.1 } } resources { name: "mem" type: SCALAR scalar { value: 32 } } resources { name: "disk" type: SCALAR scalar { value: 10 } } resources { name: "ports" type: RANGES ranges { range { begin: 21421 end: 21425 } } role: "*" }]
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918647 6404 container_assigner.cpp:64] Registering and retrieving endpoint for container_id[value: "8750c2a7-8bef-4a69-8ef2-b873f884bf91"] executor_info[executor_id { value: "instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001" } resources { name: "cpus" type: SCALAR scalar { value: 0.1 } } resources { name: "mem" type: SCALAR scalar { value: 32 } } resources { name: "disk" type: SCALAR scalar { value: 10 } } resources { name: "ports" type: RANGES ranges { range { begin: 21421 end: 21425 } } role: "*" } command { value: "/opt/mesosphere/packages/mesos--a00089c894f82b6455d1db7c4267a742458020b3/libexec/mesos/mesos-default-executor" user: "root" shell: false arguments: "mesos-default-executor" } framework_id { value: "ce4bd8be-1198-4819-81d4-9a8439439741-0000" } container { type: MESOS network_infos { labels { labels { key: "rcluster.location" value: "globalqa.las2.test0" } labels { key: "rcluster.group" value: "dat" } labels { key: "rcluster.application" value: "scout" } } name: "rcluster-cni" port_mappings { host_port: 21421 container_port: 8080 protocol: "tcp" } port_mappings { host_port: 21422 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21423 container_port: 8093 protocol: "tcp" } port_mappings { host_port: 21424 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21425 container_port: 8090 protocol: "tcp" } } } labels { } type: DEFAULT].
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918715 6404 sync_util.hpp:39] Dispatching and waiting <=5s for ticket 431: register_and_update_cache
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918865 6488 container_reader_impl.cpp:34] Reader constructed for 198.51.100.1:0
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919052 6488 container_reader_impl.cpp:103] Reader listening on 198.51.100.1:51243
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919108 6488 container_assigner_strategy.cpp:156] New ephemeral-port reader for container[value: "8750c2a7-8bef-4a69-8ef2-b873f884bf91"] created at endpoint[198.51.100.1:51243].
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919144 6488 container_state_cache_impl.cpp:134] Writing container file[/var/run/mesos/isolators/com_mesosphere_MetricsIsolatorModule/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91] with endpoint[198.51.100.1:51243]
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919220 6488 sync_util.hpp:136] Result for ticket 431 complete, returning value.
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919275 6404 sync_util.hpp:83] Dispatch result obtained for ticket 431 after waiting <=5s: register_and_update_cache
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.924211 6404 systemd.cpp:96] Assigned child process '2701' to 'mesos_executors.slice'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.927724 6404 systemd.cpp:96] Assigned child process '2702' to 'mesos_executors.slice'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.929244 6403 linux_launcher.cpp:421] Launching container 8750c2a7-8bef-4a69-8ef2-b873f884bf91 and cloning with namespaces CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWNET
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.935976 6403 systemd.cpp:96] Assigned child process '2705' to 'mesos_executors.slice'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.943640 6409 containerizer.cpp:1489] Checkpointing container's forked pid 2705 to '/var/lib/mesos/slave/meta/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/pids/forked.pid'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.945247 6410 cni.cpp:814] Bind mounted '/proc/2705/ns/net' to '/run/mesos/isolators/network/cni/8750c2a7-8bef-4a69-8ef2-b873f884bf91/ns' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:16.961654 2701 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:16.963691 2702 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.250937 6410 cni.cpp:1230] Got assigned IPv4 address '10.47.11.252/32' from CNI network 'rcluster-cni' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.251171 6410 cni.cpp:939] DNS nameservers for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91 are:
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: nameserver 10.190.4.22
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: nameserver 10.190.4.23
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.942303 6404 http.cpp:277] HTTP POST for /slave(1)/api/v1/executor from 10.47.11.252:38416
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.942446 6404 slave.cpp:3022] Received Subscribe request for HTTP executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.942533 6404 slave.cpp:3085] Creating a marker file for HTTP based executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000 (via HTTP) at path '/var/lib/mesos/slave/meta/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/http.marker'
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.944046 6404 disk.cpp:207] Updating the disk resources for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91 to cpus(*):0.6; mem(*):192; disk(*):170; ports(*):[21421-21425]
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.945669 6407 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 192MB for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.947779 6407 memory.cpp:251] Updated 'memory.limit_in_bytes' to 192MB for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.948951 6407 cpu.cpp:101] Updated 'cpu.shares' to 614 (cpus 0.6) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.951117 6407 cpu.cpp:121] Updated 'cpu.cfs_period_us' to 100ms and 'cpu.cfs_quota_us' to 60ms (cpus 0.6) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.952219 6404 slave.cpp:2220] Sending queued task group task group containing tasks [ dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] to executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000 (via HTTP)
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.002063 6408 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.002223 6408 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003235 6409 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003327 6407 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003432 6409 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/063dbcd6-1a0a-4058-a615-7e6deaf17707' to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003465 6407 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.005939 6407 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.006057 6407 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.006265 6407 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.006384 6407 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.008982 6409 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.009096 6409 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/a3b19108-d56c-44ea-b20a-912d539dde7b' to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.013618 6409 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.013708 6409 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/02c64ae4-ccdd-415d-946a-3f665acbb668' to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.018230 6409 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.018399 6409 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c5c10cff-a6af-4e51-911e-61d04e5c1d47' to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.022992 6403 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.023133 6403 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.023598 6406 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.023705 6406 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c' to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.462172 6405 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/063dbcd6-1a0a-4058-a615-7e6deaf17707/backends/copy/rootfses/97f1a1df-1eec-46e3-8a93-8a2b80d48dd2' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.462401 6405 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/a3b19108-d56c-44ea-b20a-912d539dde7b/backends/copy/rootfses/b93ffe07-6f64-42e1-b8c4-77bb224fc9e0' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.465819 6410 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/02c64ae4-ccdd-415d-946a-3f665acbb668/backends/copy/rootfses/5d388325-8b4d-496f-b520-4a96584a5971' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.470854 6409 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c5c10cff-a6af-4e51-911e-61d04e5c1d47/backends/copy/rootfses/5b4d32da-7d82-45d2-ad28-85a48456a18b' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.471132 6409 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c/backends/copy/rootfses/244e36f5-ecca-444f-9bf6-05aa77cb7263' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.893782 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.164958 6405 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.171018 6405 systemd.cpp:96] Assigned child process '2880' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.176822 6405 systemd.cpp:96] Assigned child process '2881' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.179283 6405 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707 and cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.185796 6409 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.200559 6405 systemd.cpp:96] Assigned child process '2884' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.209010 2881 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.211642 2880 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.215493 6405 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.228160 6407 systemd.cpp:96] Assigned child process '2896' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.234012 6407 systemd.cpp:96] Assigned child process '2897' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.239496 6409 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47 and cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.242473 6405 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.243351 6405 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.254703 6409 systemd.cpp:96] Assigned child process '2900' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.268317 2896 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.274961 6405 systemd.cpp:96] Assigned child process '2902' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.279268 6405 systemd.cpp:96] Assigned child process '2911' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.284328 6405 systemd.cpp:96] Assigned child process '2912' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.286672 6404 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b and cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.299939 6405 systemd.cpp:96] Assigned child process '2914' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.304103 6405 systemd.cpp:96] Assigned child process '2917' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.307452 6405 systemd.cpp:96] Assigned child process '2918' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.309020 2897 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.311029 6404 systemd.cpp:96] Assigned child process '2916' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.331065 2911 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.333595 6405 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668 and cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.342017 2902 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.372247 6405 systemd.cpp:96] Assigned child process '2937' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.372742 2912 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.378247 2914 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.388391 2917 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.388517 6410 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c and cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.396700 2918 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.400269 6410 systemd.cpp:96] Assigned child process '2969' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.772186 6403 containerizer.cpp:2313] Container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c has exited
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.772228 6403 containerizer.cpp:1950] Destroying container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c in RUNNING state
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.772330 6403 linux_launcher.cpp:498] Asked to destroy container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.773825 6403 linux_launcher.cpp:541] Using freezer to destroy cgroup mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.776217 6407 cgroups.cpp:2705] Freezing cgroup /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.780053 6410 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c after 3.798016ms
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.783502 6407 cgroups.cpp:2723] Thawing cgroup /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.786717 6407 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c after 3.186176ms
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.791746 6404 provisioner.cpp:488] Destroying container rootfs at '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c/backends/copy/rootfses/244e36f5-ecca-444f-9bf6-05aa77cb7263' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.873241 6405 containerizer.cpp:2229] Checkpointing termination state to nested container's runtime directory '/var/run/mesos/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c/termination'
> Nov 29 04:04:21 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:21.153594 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:23 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:23.410384 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:25 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:25.667546 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:27 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:27.925597 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:30 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:30.198536 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:32 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:32.456645 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:34 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:34.716507 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:36.974459 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:39 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:39.229652 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:41 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:41.482465 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:43 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:43.736668 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:45 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:45.994588 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:45 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:45.999001 6488 metrics_tcp_sender.cpp:252] TCP Throughput (bytes): sent=0, dropped=0, failed=0, pending=0 (state CONNECT_PENDING)
> Nov 29 04:04:46 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:46.187460 6408 slave.cpp:5044] Current disk usage 10.89%. Max allowed age: 1.582110178819861days
> Nov 29 04:04:48 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:48.161314 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57362 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:48 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:48.250435 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.033789 6488 metrics_tcp_sender.cpp:127] Timed out when opening metrics connection to 127.0.0.1:8124. This is expected if no Metrics Collector is running on this agent. (state CONNECT_PENDING)
> Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.033946 6488 metrics_tcp_sender.cpp:112] Attempting to open connection to 127.0.0.1:8124
> Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.034059 6488 metrics_tcp_sender.cpp:140] Got error 'Connection refused'(system:111) when connecting to metrics service at 127.0.0.1:8124. This is expected if no Metrics Collector is running on this agent. (state CONNECT_IN_PROGRESS)
> Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.034081 6488 metrics_tcp_sender.cpp:94] Scheduling reconnect to 127.0.0.1:8124 in 60s...
> Nov 29 04:04:50 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:50.161448 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57426 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:50 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:50.503459 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:52 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:52.161912 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57486 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:52 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:52.765455 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:54 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:54.161412 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57546 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:55 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:55.022529 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:56 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:56.163820 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57606 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:57 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:57.279559 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:58 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:58.162578 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57666 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:59 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:59.538532 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:00 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:00.161715 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57726 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:01 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:01.801764 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:02 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:02.160838 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57790 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:04 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:04.060504 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:04 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:04.162878 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57908 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:06 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:06.161875 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57962 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:06 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:06.318395 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:08 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:08.160797 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58016 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:08 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:08.572476 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:10 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:10.162160 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58070 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:10 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:10.828313 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:12 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:12.161572 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58124 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:13 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:13.084312 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:14 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:14.164269 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58182 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:15 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:15.339344 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:16.165223 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58238 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:16.919145 6488 container_reader_impl.cpp:146] Throughput from container at port 51243 (bytes): received=0, throttled=0
> Nov 29 04:05:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:17.476977 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58298 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:17.594532 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:19 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:19.856494 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:22 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:22.112598 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:24 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:24.371556 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:26.628479 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:28 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:28.885504 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:31 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:31.143674 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:33 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:33.434650 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:35 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:35.694718 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:37 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:37.952450 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:40 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:40.209468 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:42 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:42.466473 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:44 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:44.724468 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:45 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:45.999135 6488 metrics_tcp_sender.cpp:252] TCP Throughput (bytes): sent=0, dropped=373, failed=0, pending=0 (state CONNECT_PENDING)
> Nov 29 04:05:46 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:46.188647 6408 slave.cpp:5044] Current disk usage 10.83%. Max allowed age: 1.583336138877014days
> Nov 29 04:05:46 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:46.986548 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034080 6488 metrics_tcp_sender.cpp:127] Timed out when opening metrics connection to 127.0.0.1:8124. This is expected if no Metrics Collector is running on this agent. (state CONNECT_PENDING)
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034231 6488 metrics_tcp_sender.cpp:112] Attempting to open connection to 127.0.0.1:8124
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034332 6488 metrics_tcp_sender.cpp:140] Got error 'Connection refused'(system:111) when connecting to metrics service at 127.0.0.1:8124. This is expected if no Metrics Collector is running on this agent. (state CONNECT_IN_PROGRESS)
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034358 6488 metrics_tcp_sender.cpp:94] Scheduling reconnect to 127.0.0.1:8124 in 60s...
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.243544 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:51 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:51.500578 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:53 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:53.755667 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:56 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:56.010568 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:58 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:58.269441 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:00 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:00.526623 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:02 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:02.784615 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:05 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:05.043455 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:07 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:07.299536 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:09 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:09.556404 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:11 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:11.813442 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:14 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:14.068482 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:16.325479 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:16.919404 6488 container_reader_impl.cpp:146] Throughput from container at port 51243 (bytes): received=0, throttled=0
> Nov 29 04:06:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:18.581571 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:20.841563 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:23 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:23.098562 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:25 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:25.358496 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886557 6406 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886762 6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886831 6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886884 6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886924 6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:27 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:27.615567 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:29 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:29.877619 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:32 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:32.135524 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:34 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:34.402644 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.664479 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903115 6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903259 6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903317 6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903436 6408 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903501 6408 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> {code}
> Executor log:
> {code}
> I1129 04:04:17.832391 2734 executor.cpp:189] Version: 1.1.0
> I1129 04:04:17.956600 2738 default_executor.cpp:123] Received SUBSCRIBED event
> I1129 04:04:17.956832 2738 default_executor.cpp:127] Subscribed executor on 10.190.112.199
> I1129 04:04:17.968544 2735 default_executor.cpp:123] Received LAUNCH_GROUP event
> I1129 04:06:26.899070 2735 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899101 2735 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5'
> W1129 04:06:26.899122 2735 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5' as it is no longer active
> I1129 04:06:26.899235 2742 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899251 2742 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1'
> W1129 04:06:26.899260 2742 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1' as it is no longer active
> I1129 04:06:26.899425 2741 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899443 2741 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3'
> W1129 04:06:26.899448 2741 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3' as it is no longer active
> I1129 04:06:26.899591 2740 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899616 2740 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4'
> W1129 04:06:26.899627 2740 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4' as it is no longer active
> I1129 04:06:26.899699 2735 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899713 2735 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2'
> W1129 04:06:26.899721 2735 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2' as it is no longer active
> I1129 04:06:36.915755 2740 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.915786 2740 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5'
> W1129 04:06:36.915796 2740 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5' as it is no longer active
> I1129 04:06:36.915915 2736 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.915931 2736 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1'
> W1129 04:06:36.915940 2736 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1' as it is no longer active
> I1129 04:06:36.915962 2736 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.915972 2736 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3'
> W1129 04:06:36.915979 2736 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3' as it is no longer active
> I1129 04:06:36.915997 2736 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.916007 2736 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4'
> W1129 04:06:36.916013 2736 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4' as it is no longer active
> I1129 04:06:36.916031 2736 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.916040 2736 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2'
> W1129 04:06:36.916048 2736 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2' as it is no longer active
> continuing for hours ...
> {code}
> Meanwhile, the tasks show up as {{STAGING}} in the Mesos web UI, and their sandboxes are empty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)