[jira] [Commented] (KUDU-2836) Wrong memory used detection

["Adar Dembo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852817#comment-16852817 ] 

Adar Dembo commented on KUDU-2836:
----------------------------------

For more context, look at KUDU-1953, KUDU-1772, and https://gerrit.cloudera.org/c/1711. In another code review not in gerrit, Todd and I discussed the merits of using generic.current_allocated_bytes vs generic.heap_size. Here's our conversation:

Adar: It wasn't clear whether to look at tcmalloc's generic.heap_size or generic.current_allocated_bytes. The latter purports to include tcmalloc overhead and memory fragmentation, but experimentation revealed that its value never drops, presumably because as a "heap" size, it makes sense for it to measure virtual memory.

Todd: I don't think it's measuring virtual memory... I think we should probably set 'tcmalloc_heap_limit_mb' to the same as our own limit:
{noformat}
DEFINE_int64(tcmalloc_heap_limit_mb,
              EnvToInt("TCMALLOC_HEAP_LIMIT_MB", 0),
              "Limit total size of the process heap to the "
              "specified number of MiB. "
              "When we approach the limit the memory is released "
              "to the system more aggressively (more minor page faults). "
              "Zero means to allocate as long as system allows.");
{noformat}
I think tcmalloc might start releasing more aggressively at that point? You looked at RSS vs the metric?

Adar: Within tcmalloc, generic.heap_size maps to PageHeap::Stats.system_bytes, which grows with every heap allocation and never drops. True, it's not a measure of all virtual memory, but it's effectively the same as the virtual address space size of the tcmalloc heap. Anyway, we talked about this on HipChat and I'm summarizing our discussion here. It appears that when we allocate such that we've mapped tcmalloc_heap_limit_mb megabytes of data, tcmalloc will force the current allocation to release some pages. That is, it's a more aggressive way to release pages vs. our current method (MemTracker will release en masse every 128 MB of memory Released()). But, I'm not yet convinced that we need to be that aggressive. In tpch_real_world there was a very small difference between memory we track and memory tcmalloc tracks, so it's not like we're losing out on opportunities to release memory. My suggestion is that we keep the behavior as-is and potentially change our minds later.

> Wrong memory used detection
> ---------------------------
>
>                 Key: KUDU-2836
>                 URL: https://issues.apache.org/jira/browse/KUDU-2836
>             Project: Kudu
>          Issue Type: Improvement
>          Components: tserver
>            Reporter: Yingchun Lai
>            Assignee: Yingchun Lai
>            Priority: Critical
>
> One of my tserver, totally 128G memory, gflags: 
> {code:java}
> -memory_limit_hard_bytes=107374182475 (100G)  -memory_limit_soft_percentage=85 -memory_pressure_percentage=80{code}
> Memory used about 95%, "top" result like:
> {code:java}
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8359 work 20 0 0.326t 0.116t 81780 S 727.9 94.6 230228:10 kudu_tablet_ser
> {code}
> That is kudu_tablet_server process used about 116G memory.
> On mem-trackers page, I find the "Total consumption" value is about 65G, much lower than 116G.
> Then, I login to the server and read code to check any free memory MM operations are work correctly. Unfortunatly, the memory pressure detect function(process_memory::UnderMemoryPressure) doesn't report it's under pressure, because the tcmalloc function GetNumericProperty(const char* property, size_t* value) with parameter "generic.current_allocated_bytes" doesn't return the memory as the memory use reported by the OS.
> [https://gperftools.github.io/gperftools/tcmalloc.html]
> {quote}
> |{{generic.current_allocated_bytes}}|Number of bytes used by the application. This will not typically match the memory use reported by the OS, because it does not include TCMalloc overhead or memory fragmentation.|
> {quote}
> This situation may lead to OPs prefer to free memory could not be scheduled promptly, and the OS memory may consumed empty, and then kill tserver because of OOM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)