kudu git commit: block manager: reduce file cache to 40% of total fds

[to...@apache.org]

Repository: kudu
Updated Branches:
  refs/heads/master 114056701 -> 1bf12cf92


block manager: reduce file cache to 40% of total fds

Some management applications (including Cloudera Manager) monitor the number
of open fds belonging to each process, and warn when that number exceeds 50%
of the process' configured limit. By claiming half of the limit, the block
manager's file cache is pretty much guaranteed to trigger this warning.

These monitoring thresholds are reconfigurable, but since our 50% limit was
chosen pretty much arbitrarily, we can also (arbitrarily) drop it to, say,
40% in order to avoid tripping the warning in the first place. The extra 10%
should be enough for the (low) number of fds used by open WAL segments.

Change-Id: I67427bd05245c98315dca4140629976f67e2c6ed
Reviewed-on: http://gerrit.cloudera.org:8080/6094
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
Tested-by: Kudu Jenkins
Reviewed-by: Todd Lipcon <to...@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/1bf12cf9
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/1bf12cf9
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/1bf12cf9

Branch: refs/heads/master
Commit: 1bf12cf929a6aaf2d50aed618f69b55dc8e822af
Parents: 1140567
Author: Adar Dembo <ad...@cloudera.com>
Authored: Tue Feb 21 08:23:14 2017 -0800
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue Feb 21 21:52:28 2017 +0000

----------------------------------------------------------------------
 src/kudu/fs/block_manager.cc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/1bf12cf9/src/kudu/fs/block_manager.cc
----------------------------------------------------------------------
diff --git a/src/kudu/fs/block_manager.cc b/src/kudu/fs/block_manager.cc
index 40f1cb0..856e1a7 100644
--- a/src/kudu/fs/block_manager.cc
+++ b/src/kudu/fs/block_manager.cc
@@ -38,7 +38,7 @@ TAG_FLAG(block_manager_lock_dirs, unsafe);
 
 DEFINE_int64(block_manager_max_open_files, -1,
              "Maximum number of open file descriptors to be used for data "
-             "blocks. If 0, there is no limit. If -1, Kudu will use half of "
+             "blocks. If 0, there is no limit. If -1, Kudu will use 40% of "
              "its resource limit as per getrlimit(). This is a soft limit.");
 TAG_FLAG(block_manager_max_open_files, advanced);
 TAG_FLAG(block_manager_max_open_files, evolving);
@@ -64,7 +64,7 @@ int64_t GetFileCacheCapacityForBlockManager(Env* env) {
 
   // See block_manager_max_open_files.
   if (FLAGS_block_manager_max_open_files == -1) {
-    return env->GetOpenFileLimit() / 2;
+    return (2 * env->GetOpenFileLimit()) / 5;
   }
   if (FLAGS_block_manager_max_open_files == 0) {
     return kint64max;