You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2024/10/24 22:25:00 UTC
[jira] [Commented] (IMPALA-13479) Patch gperftools to allow max_total_thread_cache_bytes to exceed 1GB
[ https://issues.apache.org/jira/browse/IMPALA-13479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892628#comment-17892628 ]
Quanlong Huang commented on IMPALA-13479:
-----------------------------------------
It's interesting that the tcmalloc repo itself doens't have such a hard limit:
https://github.com/google/tcmalloc/blob/1d27c087909ed8c24f522e2c3c73cc88e4010653/tcmalloc/thread_cache.cc#L409-L414
Maybe gperftools adds it for some reason but it's not suitable for Impala.
> Patch gperftools to allow max_total_thread_cache_bytes to exceed 1GB
> --------------------------------------------------------------------
>
> Key: IMPALA-13479
> URL: https://issues.apache.org/jira/browse/IMPALA-13479
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
> Affects Versions: Impala 4.5.0
> Reporter: Joe McDonnell
> Priority: Major
>
> gperftools limits max_total_thread_cache_bytes to 1GB here:
> [https://github.com/gperftools/gperftools/blob/gperftools-2.10/src/thread_cache.cc#L520-L523]
> {noformat}
> void ThreadCache::set_overall_thread_cache_size(size_t new_size) {
> // Clip the value to a reasonable range
> if (new_size < kMinThreadCacheSize) new_size = kMinThreadCacheSize;
> if (new_size > (1<<30)) new_size = (1<<30); // Limit to 1GB{noformat}
> I confirmed that setting --tcmalloc_max_total_thread_cache_bytes=2147483648 still results in a 1GB limit.
> Sometimes, we would want a higher limit for systems with a large amount of memory and CPUs. For example, some systems now have 1TB of memory and 96 CPUs. With high concurrency, there is high contention on tcmalloc locks on central data structures. Increasing the total thread cache size could avoid this, and a value higher than 1GB is still a small part of system memory.
> We can patch our toolchain gperftools to allow a higher value (and notify gperftools community).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org