You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Chun-Hung Hsiao (JIRA)" <ji...@apache.org> on 2017/04/20 23:33:04 UTC
[jira] [Commented] (MESOS-6004) Tasks fail when provisioning
multiple containers with large docker images using copy backend
[ https://issues.apache.org/jira/browse/MESOS-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977790#comment-15977790 ]
Chun-Hung Hsiao commented on MESOS-6004:
----------------------------------------
I did some experiments as well as the following simple math: for a good SSD storage, we have a typical 500MB/s disk write speed. When running 5 tasks on the copy backend, each gets 100MB/s on average. It means that it can only handle images up to 6GB within 1 minute. For a HDD up to 160MB/s, the copy backend can handle images up to around 2GB before timeout.
I was thinking about using copy-on-write for the copy backend. Unfortunately, it seems that CoW is not supported by common filesystems. Any suggestion?
> Tasks fail when provisioning multiple containers with large docker images using copy backend
> ---------------------------------------------------------------------------------------------
>
> Key: MESOS-6004
> URL: https://issues.apache.org/jira/browse/MESOS-6004
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
> Affects Versions: 0.28.2, 1.0.0
> Environment: h4. Agent Platform
> - Ubuntu 16.04
> - AWS g2.x2large instance
> - Nvidia support enabled
> h4. Agent Configuration
> -{noformat}
> --containerizers=mesos,docker
> --docker_config=<docker auth json>
> --docker_store_dir=/mnt/mesos/store/docker
> --executor_registration_timeout=3mins
> --hostname=<aws public dns>
> --image_providers=docker
> --image_provisioner_backend=copy
> --isolation=filesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia
> --switch_user=false
> --work_dir=/mnt/mesos
> {noformat}
> h4. Framework
> - custom framework written in python
> - using unified containerizer with docker images
> h4. Test Setup
> * 1 master
> * 1 agent
> * 5 tasks scheduled at the same time:
> ** resources: cpus: 0.1, mem: 128
> ** command: `echo test`
> ** docker image: custom docker image, based on nvidia/cuda ~5gb
> ** the same docker image was for all tasks, already pulled.
> Reporter: Michael Thomas
> Assignee: Chun-Hung Hsiao
> Labels: containerizer, docker, performance
>
> When scheduling more than one task on the same agent, all tasks fail a as containers seem to be destroyed during provisioning.
> Specifically, the errors on the agent logs are:
> {noformat}
> E0808 15:53:09.691315 30996 slave.cpp:3976] Container 'eb20f642-bb90-4293-8eec-6f1576ccaeb1' for executor '3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0001 failed to start: Container is being destroyed during provisioning
> {noformat}
> and
> {noformat}
> I0808 15:52:32.510210 30999 slave.cpp:4539] Terminating executor ''2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0001' because it did not register within 3mins
> {noformat}
> As the default provisioning method {{copy}} is being used, I assume this is due to the provisioning of multiple containers taking too long and the agent will not wait. For large images, this method is simply not performant.
> The issue did not occur, when only one tasks was scheduled.
> Increasing the {{executor_registration_timeout}} parameter, seemed to help a bit as it allowed to schedule at least 2 tasks at the same time. But still fails with more (5 in this case)
> h4. Complete logs
> (with GLOG_v=1)
> {noformat}
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800375 3738 slave.cpp:198] Agent started on 1)@172.31.23.17:5051
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800403 3738 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos,docker" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_config="XXX" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/mnt/t" --docker_volume_checkp
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: oint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname="ec2-52-59-113-0.eu-central-1.compute.amazonaws.com" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_providers="docker" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="filesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://172.31.19.240:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --re
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: covery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="false" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/mesos"
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09 10:11:41,800:3738(0x7f71172a1700):ZOO_INFO@check_events@1728: initiated connection to server [172.31.19.240:2181]
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801007 3738 slave.cpp:519] Agent resources: gpus(*):1; cpus(*):8; mem(*):14014; disk(*):60257; ports(*):[31000-32000]
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801059 3738 slave.cpp:527] Agent attributes: [ ]
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801077 3738 slave.cpp:532] Agent hostname: ec2-52-59-113-0.eu-central-1.compute.amazonaws.com
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.802232 3772 state.cpp:57] Recovering state from '/mnt/mesos/meta'
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803539 3774 status_update_manager.cpp:200] Recovering status update manager
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803745 3772 containerizer.cpp:522] Recovering containerizer
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803788 3770 docker.cpp:775] Recovering Docker containers
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808550 3777 metadata_manager.cpp:251] Successfully loaded 2 Docker images
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808688 3771 provisioner.cpp:253] Provisioner recovery complete
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09 10:11:41,816:3738(0x7f71172a1700):ZOO_INFO@check_events@1775: session establishment complete on server [172.31.19.240:2181], sessionId=0x1566a66ab9b0010, negotiated timeout=10000
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817003 3774 group.cpp:349] Group process (group(1)@172.31.23.17:5051) connected to ZooKeeper
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817116 3774 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817147 3774 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.818857 3774 detector.cpp:152] Detected a new leader: (id='64')
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819005 3776 group.cpp:706] Trying to get '/mesos/json.info_0000000064' in ZooKeeper
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819855 3775 zookeeper.cpp:259] A new leading master (UPID=master@172.31.19.240:5050) is detected
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.899876 3777 slave.cpp:4782] Finished recovery
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900573 3777 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900687 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2' for gc 6.99998957623704days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900729 3777 slave.cpp:4815] Garbage collecting old agent c9852a23-bc07-422d-8d69-23c167a1924d-S0
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900779 3773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2' for gc 6.99998957520889days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900817 3777 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900825 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S0' for gc 6.99998957451259days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900899 3773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S0' for gc 6.99998957412148days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900933 3777 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900944 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1' for gc 6.99998957332148days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901010 3773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1' for gc 6.99998957291556days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901051 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3' for gc 6.99998957196741days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901101 3773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3' for gc 6.99998957156148days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901201 3775 status_update_manager.cpp:174] Pausing sending status updates
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901203 3777 slave.cpp:895] New master detected at master@172.31.19.240:5050
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901278 3777 slave.cpp:916] No credentials provided. Attempting to register without authentication
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901317 3777 slave.cpp:927] Detecting new master
> Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267201 3774 slave.cpp:1197] Re-registered with master master@172.31.19.240:5050
> Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 3771 status_update_manager.cpp:181] Resuming sending status updates
> Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 3774 slave.cpp:1233] Forwarding total oversubscribed resources
> Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267577 3774 slave.cpp:2526] Updated checkpointed resources from to
> Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:48.269104 3777 slave.cpp:2218] Asked to shut down framework c9852a23-bc07-422d-8d69-23c167a1924d-0002 by master@172.31.19.240:5050
> Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: W0809 10:11:48.269186 3777 slave.cpp:2233] Cannot shut down unknown framework c9852a23-bc07-422d-8d69-23c167a1924d-0002
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.969521 3772 slave.cpp:1495] Got assigned task 0 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970026 3772 slave.cpp:1614] Launching task 0 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970826 3772 slave.cpp:5674] Launching executor 0 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work directory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-4581-966a-734d24ae96f3'
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971051 3772 slave.cpp:1840] Queuing task '0' for executor '0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971212 3773 containerizer.cpp:781] Starting container '09a77a80-f904-4581-966a-734d24ae96f3' for executor '0' of framework 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.973953 3774 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec9bb10e-6726-4872-be2b-4a72dfc0fafc' for container 09a77a80-f904-4581-966a-734d24ae96f3
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978344 3774 slave.cpp:1495] Got assigned task 1 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978687 3774 slave.cpp:1614] Launching task 1 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978976 3774 slave.cpp:5674] Launching executor 1 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work directory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b68-851a-531ed101de5a'
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979173 3774 slave.cpp:1840] Queuing task '1' for executor '1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979387 3777 containerizer.cpp:781] Starting container '0cea1dc7-b523-4b68-851a-531ed101de5a' for executor '1' of framework 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.981374 3776 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3af89fe-c889-45fc-929b-f008b18e5d4e' for container 0cea1dc7-b523-4b68-851a-531ed101de5a
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983610 3776 slave.cpp:1495] Got assigned task 2 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983947 3776 slave.cpp:1614] Launching task 2 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984236 3776 slave.cpp:5674] Launching executor 2 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work directory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4b-8481-79ca320ddab7'
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984450 3776 slave.cpp:1840] Queuing task '2' for executor '2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984778 3777 containerizer.cpp:781] Starting container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' for executor '2' of framework 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.986330 3776 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/feaf70a4-df43-45c9-b6be-a654c9210a54' for container a4c972da-b12d-4b4b-8481-79ca320ddab7
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996104 3771 slave.cpp:1495] Got assigned task 3 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996462 3771 slave.cpp:1614] Launching task 3 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997241 3771 slave.cpp:5674] Launching executor 3 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work directory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468f-947a-8bf97d15a630'
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997406 3771 slave.cpp:1840] Queuing task '3' for executor '3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997640 3777 containerizer.cpp:781] Starting container 'd9363f0d-645b-468f-947a-8bf97d15a630' for executor '3' of framework 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.999593 3777 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/eeb1b066-94b8-4880-a696-090f830cf2d6' for container d9363f0d-645b-468f-947a-8bf97d15a630
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.003924 3775 slave.cpp:1495] Got assigned task 4 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.004278 3775 slave.cpp:1614] Launching task 4 for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005383 3775 slave.cpp:5674] Launching executor 4 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work directory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-4389-87ba-8696c9a523eb'
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005681 3775 slave.cpp:1840] Queuing task '4' for executor '4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005971 3772 containerizer.cpp:781] Starting container '58c398e2-38f9-4389-87ba-8696c9a523eb' for executor '4' of framework 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.007786 3777 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5' for container 58c398e2-38f9-4389-87ba-8696c9a523eb
> Aug 9 10:12:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:41.801560 3777 slave.cpp:4591] Current disk usage 43.28%. Max allowed age: 3.270091966898553days
> Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972057 3772 slave.cpp:4539] Terminating executor ''0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within 1mins
> Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972322 3775 containerizer.cpp:1622] Destroying container '09a77a80-f904-4581-966a-734d24ae96f3'
> Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.979907 3777 slave.cpp:4539] Terminating executor ''1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within 1mins
> Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.980058 3775 containerizer.cpp:1622] Destroying container '0cea1dc7-b523-4b68-851a-531ed101de5a'
> Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985003 3777 slave.cpp:4539] Terminating executor ''2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within 1mins
> Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985255 3774 containerizer.cpp:1622] Destroying container 'a4c972da-b12d-4b4b-8481-79ca320ddab7'
> Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998438 3777 slave.cpp:4539] Terminating executor ''3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within 1mins
> Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998641 3776 containerizer.cpp:1622] Destroying container 'd9363f0d-645b-468f-947a-8bf97d15a630'
> Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006058 3773 slave.cpp:4539] Terminating executor ''4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within 1mins
> Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006228 3773 containerizer.cpp:1622] Destroying container '58c398e2-38f9-4389-87ba-8696c9a523eb'
> Aug 9 10:13:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:41.802209 3775 slave.cpp:4591] Current disk usage 57.59%. Max allowed age: 2.269006805055764days
> Aug 9 10:14:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:14:41.803022 3776 slave.cpp:4591] Current disk usage 68.70%. Max allowed age: 1.491204118572176days
> Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:01.516207 3775 slave.cpp:3976] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' for executor '4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start: Container is being destroyed during provisioning
> Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:01.518277 3772 composing.cpp:541] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' is already destroyed
> Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:02.291764 3775 slave.cpp:3976] Container '09a77a80-f904-4581-966a-734d24ae96f3' for executor '0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start: Container is being destroyed during provisioning
> Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:02.291877 3775 composing.cpp:541] Container '09a77a80-f904-4581-966a-734d24ae96f3' is already destroyed
> Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.022406 3774 slave.cpp:3976] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' for executor '3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start: Container is being destroyed during provisioning
> Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.022595 3771 composing.cpp:541] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' is already destroyed
> Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.461519 3775 slave.cpp:3976] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' for executor '1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start: Container is being destroyed during provisioning
> Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.461643 3774 composing.cpp:541] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' is already destroyed
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.190901 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisioner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5' for container 58c398e2-38f9-4389-87ba-8696c9a523eb
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:04.190904 3777 slave.cpp:3976] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' for executor '2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start: Container is being destroyed during provisioning
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:04.191128 3770 composing.cpp:541] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' is already destroyed
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197221 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisioner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec9bb10e-6726-4872-be2b-4a72dfc0fafc' for container 09a77a80-f904-4581-966a-734d24ae96f3
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197410 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisioner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/eeb1b066-94b8-4880-a696-090f830cf2d6' for container d9363f0d-645b-468f-947a-8bf97d15a630
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197510 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisioner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3af89fe-c889-45fc-929b-f008b18e5d4e' for container 0cea1dc7-b523-4b68-851a-531ed101de5a
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197619 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisioner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/feaf70a4-df43-45c9-b6be-a654c9210a54' for container a4c972da-b12d-4b4b-8481-79ca320ddab7
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.390928 3776 slave.cpp:4082] Executor '3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.395370 3776 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402472 3776 slave.cpp:4082] Executor '2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402556 3776 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.406177 3772 containerizer.cpp:1451] Ignoring update for unknown container: d9363f0d-645b-468f-947a-8bf97d15a630
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.407325 3777 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.407598 3773 containerizer.cpp:1451] Ignoring update for unknown container: a4c972da-b12d-4b4b-8481-79ca320ddab7
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409890 3774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 to master@172.31.19.240:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409893 3777 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.414727 3774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 to master@172.31.19.240:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.418180 3777 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419615 3777 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419636 3774 slave.cpp:4193] Cleaning up executor '3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419924 3772 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468f-947a-8bf97d15a630' for gc 6.9999951410637days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420703 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3' for gc 6.99999513154074days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420722 3774 slave.cpp:4193] Cleaning up executor '2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420897 3774 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4b-8481-79ca320ddab7' for gc 6.99999512931852days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420940 3774 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2' for gc 6.99999512869333days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490155 3772 slave.cpp:4082] Executor '4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490289 3772 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490414 3772 slave.cpp:4082] Executor '0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490489 3772 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490687 3773 containerizer.cpp:1451] Ignoring update for unknown container: 58c398e2-38f9-4389-87ba-8696c9a523eb
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490774 3773 containerizer.cpp:1451] Ignoring update for unknown container: 09a77a80-f904-4581-966a-734d24ae96f3
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490867 3773 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 3773 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 3772 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 to master@172.31.19.240:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491303 3774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 to master@172.31.19.240:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494438 3775 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494572 3775 slave.cpp:4193] Cleaning up executor '4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494757 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-4389-87ba-8696c9a523eb' for gc 6.99999427436444days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494817 3775 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494937 3775 slave.cpp:4193] Cleaning up executor '0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494823 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4' for gc 6.99999427361185days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495107 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-4581-966a-734d24ae96f3' for gc 6.99999427055407days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495149 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0' for gc 6.99999427024296days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591338 3775 slave.cpp:4082] Executor '1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591469 3775 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.591841 3776 containerizer.cpp:1451] Ignoring update for unknown container: 0cea1dc7-b523-4b68-851a-531ed101de5a
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591974 3770 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.592157 3772 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 to master@172.31.19.240:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594518 3771 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594636 3771 slave.cpp:4193] Cleaning up executor '1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594792 3770 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b68-851a-531ed101de5a' for gc 6.99999311635556days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594820 3771 slave.cpp:4281] Cleaning up framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594861 3770 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1' for gc 6.99999311568296days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594897 3770 status_update_manager.cpp:282] Closing status update streams for framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)