You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Andrei Budnik (JIRA)" <ji...@apache.org> on 2018/10/11 14:45:00 UTC
[jira] [Commented] (MESOS-9268) Hitting agent's `/containers`
endpoint might backlog Docker containerizer process.
[ https://issues.apache.org/jira/browse/MESOS-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646526#comment-16646526 ]
Andrei Budnik commented on MESOS-9268:
--------------------------------------
After applying patches that fix related performance issues MESOS-9283 and MESOS-9279, the response time of `/containers` endpoint for 200 running Docker containers decreased to less than one second.
> Hitting agent's `/containers` endpoint might backlog Docker containerizer process.
> ----------------------------------------------------------------------------------
>
> Key: MESOS-9268
> URL: https://issues.apache.org/jira/browse/MESOS-9268
> Project: Mesos
> Issue Type: Bug
> Components: agent, containerization, docker
> Reporter: Andrei Budnik
> Priority: Major
>
> When the agent's `/containers` endpoint is hit, the agent calls `DockerContainerizerProcess::usage()` method for every running Docker container. If there are lots (hundreds) of running Docker containers and the system is under heavy load, then `DockerContainerizerProcess::usage()` method takes a lot of time to be processed. Hence, when the Docker containerizer's mailbox is full of `DockerContainerizerProcess::usage()` commands waiting to be processed, an attempt to launch a new Docker container will lead to putting `DockerContainerizer::launch()` into a long queue. This is an example from logs:
> {code:java}
> 2018-09-07 00:06:03: I0907 00:06:03.031744 65329 slave.cpp:2857] Launching container 320547f2-9ab9-4ade-bddb-bfd6f5ed0834 for executor 'example_test2519.ceeefb8a-b231-11e8-8d62-8e820bbfbfd1' of framework 51b197db-56ae-47dc-a1bd-3aaf2306bd1a-0001
> ...
> 2018-09-07 00:36:20: I0907 00:36:20.472048 65307 docker.cpp:1179] Starting container '320547f2-9ab9-4ade-bddb-bfd6f5ed0834' for task 'example_test2519.ceeefb8a-b231-11e8-8d62-8e820bbfbfd1' (and executor 'middleware_test2519.ceeefb8a-b231-11e8-8d62-8e820bbfbfd1') of framework 51b197db-56ae-47dc-a1bd-3aaf2306bd1a-0001
> {code}
> After disabling `/containers` endpoint, the issue with a backlogged Docker containerizer disappears.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)