You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2026/06/09 18:07:00 UTC

[jira] [Resolved] (IMPALA-15022) Consider modifying our tcmalloc metrics to include malloc metadata

     [ https://issues.apache.org/jira/browse/IMPALA-15022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe McDonnell resolved IMPALA-15022.
------------------------------------
    Fix Version/s: Impala 5.0.0
         Assignee: Joe McDonnell
       Resolution: Fixed

> Consider modifying our tcmalloc metrics to include malloc metadata
> ------------------------------------------------------------------
>
>                 Key: IMPALA-15022
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15022
>             Project: IMPALA
>          Issue Type: Task
>          Components: Backend
>    Affects Versions: Impala 5.0.0
>            Reporter: Joe McDonnell
>            Assignee: Joe McDonnell
>            Priority: Major
>             Fix For: Impala 5.0.0
>
>
> Our existing gperftools tcmalloc metrics historically have not included the malloc metadata. Back when the tcmalloc metrics were added to Impala, gperftools did not expose metrics that would include the malloc metadata. Since then, gperftools has added new metrics that do include it, e.g. generic.total_physical_bytes. Malloc metadata can be a couple hundred MBs on busy systems, so this is not a major change, but it seems like a slight improvement to our metrics.
> One up side to making this change is that our metrics would line up exactly to the text output from tcmalloc in the /memz page.
> {noformat}
>     // ------------------------------------------------
>     // MALLOC:      145138800 (  138.4 MiB) Bytes in use by application (#1)
>     // MALLOC: +      1056768 (    1.0 MiB) Bytes in page heap freelist (#2)
>     // MALLOC: +      1929472 (    1.8 MiB) Bytes in central cache freelist (#3)
>     // MALLOC: +      7655168 (    7.3 MiB) Bytes in transfer cache freelist (#4)
>     // MALLOC: +      6626192 (    6.3 MiB) Bytes in thread cache freelists (#5)
>     // MALLOC: +      6160448 (    5.9 MiB) Bytes in malloc metadata
>     // MALLOC:   ------------
>     // MALLOC: =    168566848 (  160.8 MiB) Actual memory used (physical + swap) (#6**)
>     // MALLOC: +     51183616 (   48.8 MiB) Bytes released to OS (aka unmapped) (#7)
>     // MALLOC:   ------------
>     // MALLOC: =    219750464 (  209.6 MiB) Virtual address space used (#8**)
>     // MALLOC:
>     // MALLOC:           4354              Spans in use
>     // MALLOC:            790              Thread heaps in use
>     // MALLOC:           8192              Tcmalloc page size{noformat}
> #1 = tcmalloc.bytes-in-use (generic.current_allocated_bytes)
> #2 = tcmalloc.pageheap-free-bytes (tcmalloc.pageheap_free_bytes)
> #3 = tcmalloc.central-cache-free-bytes (tcmalloc.central_cache_free_bytes)
> #4 = tcmalloc.transfer-cache-free-bytes (tcmalloc.transfer_cache_free_bytes)
> #5 = tcmalloc.thread-cache-free-bytes (tcmalloc.thread_cache_free_bytes)
> #6 = tcmalloc.physical-bytes-reserved = #8 - #7 ** Except Impala's metric is currently missing the metadata
> #7 = tcmalloc.pageheap-unmapped-bytes (tcmalloc.pageheap_unmapped_bytes)
> #8 = tcmalloc.total-bytes-reserved (generic.heap_size) ** Except Impala's metric is currently missing the metadata
> If we reworked it to include the metadata, then #6 and #8 would line up nicely with the memz output.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)