[kudu-CR] Only release memory to OS when overhead is unacceptable

["Todd Lipcon (Code Review)" <ge...@ip-10-146-233-104.ec2.internal>]

Hello Mike Percy, Adar Dembo,

I'd like you to do a code review.  Please visit

    http://gerrit.cloudera.org:8080/1711

to review the following change.

Change subject: Only release memory to OS when overhead is unacceptable
......................................................................

Only release memory to OS when overhead is unacceptable

Previously, we would call MallocExtension::ReleaseFreeMemory() after
every 128MB of memory released by MemTrackers. This had two issues:

1) it releases _all_ free memory from the tcmalloc heap, which causes
unnecessary minor page faults later

2) it holds the tcmalloc page heap lock, preventing other threads from
accessing central free lists[1]

Looking at a trace of an Impala query that hit OS cache but missed the
Kudu cache, I saw periodic blips where all scan RPCs would block for
20ms several time per second. After adding a trace span around the
tcmalloc call, it was clear that this was the culprit. This workload
triggers a lot of allocations and frees due to block cache churn.

To fix, this patch introduces a new flag for the acceptable ratio of
free memory compared to in-use memory on the heap, defaulting to 10%.
The MemTracker now only releases memory back if this threshold is
exceeded, and does so in small increments of 1MB at a time, such that
the tcmalloc-internal lock is never held for too long.

To test the effects of the patch, I looked at percentiles of the Scan
RPC on this workload before and after the patch:

Before: {'max': 14527, 'p999': 8895, 'p50': 3279, 'p95': 5759, 'p99': 6783}
After:  {'max': 14527, 'p999': 8447, 'p50': 3167, 'p95': 5471, 'p99': 6399}

It seems to have improved the tail latencies by about 5%. Other
workloads may improve more or less.

[1] https://github.com/gperftools/gperftools/issues/754

Change-Id: I184c9b7c64a57d4cd178fe7cfb2c548e323edecb
---
M src/kudu/util/mem_tracker.cc
1 file changed, 32 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/11/1711/1
-- 
To view, visit http://gerrit.cloudera.org:8080/1711
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I184c9b7c64a57d4cd178fe7cfb2c548e323edecb
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Mike Percy <mp...@cloudera.com>