You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2026/10/08 23:12:00 UTC

[jira] [Commented] (IMPALA-15316) Support huge pages when tcmalloc's aggressive decommit is off

    [ https://issues.apache.org/jira/browse/IMPALA-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125291#comment-18125291 ] 

ASF subversion and git services commented on IMPALA-15316:
----------------------------------------------------------

Commit 5d6ca16dd9382d89c0aaefdf0f72c060ceaf0f96 in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5d6ca16dd ]

IMPALA-15316: Support huge pages with aggressive decommit off

Currently, running with tcmalloc_aggressive_memory_decommit=false
is incompatible with using madvise_huge_pages=true. Gperftools
TCMalloc can hold on to memory and potentially break up huge
pages. That can lead to deceptive memory usage metrics.

This changes the system allocator to use mmap for huge pages
when tcmalloc_aggressive_memory_decommit=false. This bypasses
malloc and avoids any issues with TCMalloc behavior. This does
not add a new startup flag. Instead, it is now possible to use
tcmalloc_aggressive_memory_decommit=false together with
madvise_huge_pages=true. Users can disable the mmap behavior
by using madvise_huge_pages=false.

This adds the buffer-pool.mmapped-bytes metric to track memory
allocated via mmap. This is included in the process memory
tracker.

This does not change the behavior when building with Google
TCMalloc, as it is already huge page aware.

Testing:
 - Ran tests with tcmalloc_aggressive_memory_decommit=false
   and madvise_huge_pages=true.

Change-Id: I13ac5f065104d5cf35efb5029ba3c4d482904096
Reviewed-on: http://gerrit.cloudera.org:8080/24761
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Support huge pages when tcmalloc's aggressive decommit is off
> -------------------------------------------------------------
>
>                 Key: IMPALA-15316
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15316
>             Project: IMPALA
>          Issue Type: Task
>          Components: Backend
>    Affects Versions: Impala 5.0.0
>            Reporter: Joe McDonnell
>            Assignee: Joe McDonnell
>            Priority: Critical
>
> In IMPALA-14900, we added support for turning off TCMalloc's aggressive decommit mode. This came at the cost of disabling madvise_huge_pages for that configuration. Without aggressive decommit, TCMalloc could break up huge pages and reuse smaller chunks. This could make the memory usage deceptive.
> We can support huge pages with aggressive decommit off by mmap them ourselves in the SystemAllocator. We would need to track this memory usage and count it toward the process limit.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org