[jira] [Comment Edited] (MESOS-6264) Investigate the high memory usage of the default executor.

["Joris Van Remoortere (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-6264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549111#comment-15549111 ] 

Joris Van Remoortere edited comment on MESOS-6264 at 10/5/16 3:45 PM:
----------------------------------------------------------------------

cc [~vinodkone][~jieyu]
The bulk of this comes from loading in {{libmesos.so}}
We do this because the autoconf build treats libmesos as a dynamic dependency.
Since we load libmesos dynamically, there is no chance for the linker to strip unused code. This means that all of the code in libmesos regardless of use gets loaded into resident memory.
In contrast the cmake build generates a static library for {{libmesos.a}}. This is then used to build the {{mesos-executor}} binary without a dynamic dependency on libmesos. The benefit of this approach is that the linker is able to strip out all unused code. In an optimized build this is {{~10MB}}.

Some approaches for the quick win are:
# Consider using the cmake build. This only needs to be modified slightly to strip symbols from the final executor binary {{-s}}.
# Modiy the autoconf build to build a {{libmesos.a}} so that we can statically link it in to the {{mesos-executor}} binary and allow the linker to strip unused code.

Regardless of the above approach, {{libmesos}} would still be by far the largest contributor of the {{RSS}}. This is for 2 reasons:
# Much of our code is structured such that the linker can't determine if it is unused. We would need to adjust our patterns such that the unused code analyzer can do a better job.
# Much of our code is {{inlined}} or written such that it can't be optimized. 2 examples are:
## https://github.com/apache/mesos/blob/9beb8eae6408249cdb3e2f16ba68b31a00d3452c/3rdparty/libprocess/include/process/mime.hpp#L35-L154
This code could be moved to a {{.cpp}} file and should be a {{static const std::unordered_map<string, string>}} that we {{insert(begin(), end())}} into {{types}}. This would reduce the size of libmesos by {{~20KB}}!
## https://github.com/apache/mesos/blob/9beb8eae6408249cdb3e2f16ba68b31a00d3452c/3rdparty/libprocess/include/process/http.hpp#L453-L517
This code and sibling {{struct Request}} have auto-generated {{inlined}} destructors. These are very expensive. Just declaring and then defining in the {{.cpp}} the default destructor can remove another {{~20KB}} each from libmesos. There are plenty of other opportunities like this scattered through the codebase. It's work to find them and the returns are small for each, but end up adding to much of the {{9MB}} left over.


was (Author: jvanremoortere):
cc [~vinodkone][~jieyu]
The bulk of this comes from loading in {{libmesos.so}}
We do this because the autoconf build treats libmesos as a dynamic dependency.
Since we load libmesos dynamically, there is no chance for the linker to strip unused code. This means that all of the code in libmesos regardless of use gets loaded into resident memory.
In contrast the cmake build generates a static library for {{libmesos.a}}. This is then used to build the {{mesos-executor}} binary without a dynamic dependency on libmesos. The benefit of this approach is that the linker is able to strip out all unused code. In an optimized build this is {{~10MB}}.

Some approaches for the quick win are:
# Consider using the cmake build. This only needs to be modified slightly to strip symbols from the final executor binary {{-s}}.
# Modiy the autoconf build to build a {{libmesos.a}} so that we can statically link it in to the {{mesos-executor}} binary and allow the linker to strip unused code.

Regardless of the above approach, {{libmesos}} would still be by far the largest contributor of the {{RSS}}. This is for 2 reasons:
# Much of our code is structured such that the linker can't determine if it is unused. We would need to adjust our patterns such that the unused code analyzer can do a better job.
# Much of our code is {{inlined}} or written such that it can't be optimized. 2 examples are:
## https://github.com/apache/mesos/blob/9beb8eae6408249cdb3e2f16ba68b31a00d3452c/3rdparty/libprocess/include/process/mime.hpp#L35-L154
This code could be moved to a {{.cpp}} file and should be a {{static const std::unordered_map<string, string>}} that we {{insert(begin(), end())}} into {{types}}. This would reduce the size of libmesos by {{~20KB}}!
## https://github.com/apache/mesos/blob/master/3rdparty/libprocess/include/process/http.hpp#L453-L517
This code and sibling {{struct Request}} have auto-generated {{inlined}} destructors. These are very expensive. Just declaring and then defining in the {{.cpp}} the default destructor can remove another {{~20KB}} each from libmesos. There are plenty of other opportunities like this scattered through the codebase. It's work to find them and the returns are small for each, but end up adding to much of the {{9MB}} left over.

> Investigate the high memory usage of the default executor.
> ----------------------------------------------------------
>
>                 Key: MESOS-6264
>                 URL: https://issues.apache.org/jira/browse/MESOS-6264
>             Project: Mesos
>          Issue Type: Bug
>            Reporter: Anand Mazumdar
>              Labels: mesosphere
>             Fix For: 1.1.0
>
>         Attachments: pmap_output_for_the_default_executor.txt
>
>
> It seems that a default executor with two sleep tasks is using ~32 mb on average and can sometimes lead to it being killed for some tests like {{SlaveRecoveryTest/0.ROOT_CGROUPS_ReconnectDefaultExecutor}} on our internal CI. Attached the {{pmap}} output for the default executor. Please note that the command executor memory usage is also pretty high (~26 mb).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)