[GitHub] [incubator-pegasus] empiredan commented on a change in pull request #646: feat: support costing memtable and index and filter blocks to block c…

[GitBox <gi...@apache.org>]

empiredan commented on a change in pull request #646:
URL: https://github.com/apache/incubator-pegasus/pull/646#discussion_r530149162



##########
File path: src/server/pegasus_server_impl_init.cpp
##########
@@ -299,6 +300,130 @@ pegasus_server_impl::pegasus_server_impl(dsn::replication::replica *r)
         _db_opts.rate_limiter = _s_rate_limiter;
     }
 
+    bool enable_write_buffer_manager = dsn_config_get_value_bool(
+                "pegasus.server",
+                "rocksdb_enable_write_buffer_manager",
+                false,
+                "enable write buffer manager to limit total memory "
+                "used by memtables and block caches across multiple replicas");
+    ddebug("rocksdb_enable_write_buffer_manager = %d", enable_write_buffer_manager);
+    if (enable_write_buffer_manager) {
+        // If write buffer manager is enabled, all replicas(one DB instance for each
+        // replica) on this server will share the same write buffer manager object,
+        // thus the same block cache object. It's convenient to control the total memory
+        // of memtables and bloack caches used by this server
+        static std::once_flag flag;
+        std::call_once(flag, [&]() {
+            int64_t total_size_across_write_buffer = dsn_config_get_value_int64(
+                    "pegasus.server",
+                    "rocksdb_total_size_across_write_buffer",
+                    0,
+                    "total size limit used by memtables across multiple replicas");
+            ddebug("rocksdb_total_size_across_write_buffer = %" PRId64, total_size_across_write_buffer);
+            if (total_size_across_write_buffer < 0) {
+                total_size_across_write_buffer = 0;
+            }
+            _s_write_buffer_manager = std::make_shared<rocksdb::WriteBufferManager>(
+                    static_cast<size_t>(total_size_across_write_buffer), tbl_opts.block_cache);
+        });
+        _db_opts.write_buffer_manager = _s_write_buffer_manager;
+    }
+
+    int64_t max_open_files = dsn_config_get_value_int64(
+            "pegasus.server",
+            "rocksdb_max_open_files",
+            -1,
+            "number of open files that can be used by a replica(namely a DB instance)");
+    _db_opts.max_open_files = static_cast<int>(max_open_files);
+    ddebug("rocksdb_max_open_files = %d", _db_opts.max_open_files);

Review comment:
       That's right, keeping per-replica  (i.e. a rocksdb instance) size not very large is the best practice. However, our case is somewhat complex. We deploy the services on our customer's private cluster for security. Some customer's configuration, for example, say max-open-files per process, is small (e.g. 65536). A case that once we experienced is making import faster, we set bulk-load mode, but it was failed at last due to too-many-open-files.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org