[jira] [Assigned] (IMPALA-14900) Add support for running Impala without tcmalloc's aggressive decommit mode

["Joe McDonnell (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe McDonnell reassigned IMPALA-14900:
--------------------------------------

    Assignee: Joe McDonnell

> Add support for running Impala without tcmalloc's aggressive decommit mode
> --------------------------------------------------------------------------
>
>                 Key: IMPALA-14900
>                 URL: https://issues.apache.org/jira/browse/IMPALA-14900
>             Project: IMPALA
>          Issue Type: Task
>          Components: Backend
>    Affects Versions: Impala 5.0.0
>            Reporter: Joe McDonnell
>            Assignee: Joe McDonnell
>            Priority: Major
>
> Impala currently sets tcmalloc's aggressive decommit mode and asserts that it is set. This forces tcmalloc to return all free spans to the operating system. Impala does this for a couple reasons:
>  # Without aggressive decommit, tcmalloc doesn't release memory fast enough for Impala's memory access patterns. Impala would need to manually force tcmalloc to free memory with a background thread and integration with the process memory tracker's garbage collection. A long time ago, there were performance issues with the way this was implemented. See IMPALA-2800.
>  # When Impala uses huge pages, we don't want tcmalloc to reuse the huge page as smaller pieces. This can potentially cause inaccuracy in our memory use statistics. See this comment in system-allocator.cc: https://github.com/apache/impala/blob/cefeb760a886d28c05f98917cdfeac864728f270/be/src/runtime/bufferpool/system-allocator.cc#L152-L157 
> However, aggressive decommit increases the memory allocation / deallocation rate and can lead to extra interactions with the operating system and contention on tcmalloc locks. We have seen issues when Parquet files have large data pages. We have also seen issues with queries operating on nested data.
> It would be useful to have a way to run Impala with aggressive decommit disabled. This will require a background thread monitoring memory use and integration with the process mem tracker's garbage collection. One option is to allow aggressive decommit to be disabled when huge pages are disabled. Another option is to use mmap for huge pages to allow using huge pages with aggressive decommit disabled.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org