[GitHub] [samza] li-afaris commented on a change in pull request #1500: [SAMZA-2655] Expose RocksDB maxOpenFiles setting as Samza parameter

[GitBox <gi...@apache.org>]

li-afaris commented on a change in pull request #1500:
URL: https://github.com/apache/samza/pull/1500#discussion_r631495076



##########
File path: samza-kv-rocksdb/src/main/java/org/apache/samza/storage/kv/RocksDbOptionsHelper.java
##########
@@ -109,6 +111,8 @@ public static Options options(Config storeConfig, int numTasksForContainer, File
     options.setMaxLogFileSize(storeConfig.getLong(ROCKSDB_MAX_LOG_FILE_SIZE_BYTES, 64 * 1024 * 1024L));
     options.setKeepLogFileNum(storeConfig.getLong(ROCKSDB_KEEP_LOG_FILE_NUM, 2));
     options.setDeleteObsoleteFilesPeriodMicros(storeConfig.getLong(ROCKSDB_DELETE_OBSOLETE_FILES_PERIOD_MICROS, 21600000000L));
+    options.setMaxOpenFiles(storeConfig.getInt(ROCKSDB_MAX_OPEN_FILES, -1));
+    options.setMaxOpenFiles(storeConfig.getInt(ROCKSDB_MAX_FILE_OPENING_THREADS, 16));

Review comment:
       This is not a typo. We want to limit the number of sst files which are opened to prevent exceeding ulimit values.
   
   Here's the java doc description for the method 
   ```
   Number of open files that can be used by the DB. You may need to increase this if your database has a large working set. Value -1 means files opened are always kept open. You can estimate number of files based on target_file_size_base and target_file_size_multiplier for level-based compaction. For universal-style compaction, you can usually set it to -1.
   ```
   https://javadoc.io/doc/org.rocksdb/rocksdbjni/6.6.4/org/rocksdb/Options.html#maxOpenFiles--
   
   Which matches this RocksDB option
   ```
     // Number of open files that can be used by the DB.  You may need to
     // increase this if your database has a large working set. Value -1 means
     // files opened are always kept open. You can estimate number of files based
     // on target_file_size_base and target_file_size_multiplier for level-based
     // compaction. For universal-style compaction, you can usually set it to -1.
   ```
   https://github.com/facebook/rocksdb/blob/master/include/rocksdb/options.h#L460




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org