[jira] [Resolved] (IMPALA-5658) Report details of process memory maps via metrics

["Tim Armstrong (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-5658.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0


IMPALA-5658: addtl. process/system-wide memory metrics

This is meant to help understand how the process is using memory, to
answer questions like:
* Is the Impala daemon using transparent huge pages?
* What is the system THP configuration?
* What is the RSS of the process?
* What is the virtual memory size of the process?

Most of these questions can be answered via other command line or
monitoring, except for the THP usage by the process, but adding
metrics helps consolidate the information in one place.

This commit adds a memory maintenance thread that periodically scrapes
information from the /proc and /sys filesystems and updates the
metrics.

The interfaces used are:
* /proc/<pid>/smaps, which is a documented interface to get detailed
  information about mapped memory:
  https://www.kernel.org/doc/Documentation/filesystems/proc.txt
* /sys/kernel/mm/ config values for transparent huge pages

Change-Id: I13873e305ba464d11dea0d7244a29ff4f332f1a9
Reviewed-on: http://gerrit.cloudera.org:8080/7472
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins

> Report details of process memory maps via metrics
> -------------------------------------------------
>
>                 Key: IMPALA-5658
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5658
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 2.10.0
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>              Labels: supportability
>             Fix For: Impala 2.10.0
>
>
> To help understand the details of process memory consumption, it would be great to have metrics that answered questions like:
> * How many anonymous huge pages are allocated?
> * How many memory maps is the process using?
> * What is the RSS of the process?
> * What is the virtual memory consumption of the process?
> It looks like /proc/<pid>/smaps is the documented interface for this: https://www.kernel.org/doc/Documentation/filesystems/proc.txt
> I'm thinking that maybe we could have the memory maintenance thread periodically scrape /proc/<pid>/smaps and update a set of metrics like:
> * Number of mappings
> * Total virtual address space
> * Total RSS
> * Total bytes of anon huge pages
> We should also report the transparent huge pages config under:
> /sys/kernel/mm/transparent_hugepage/ (or /sys/kernel/mm/redhat_transparent_hugepage if not present).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)