[GitHub] [kafka] cadonna commented on a change in pull request #10587: KAFKA-8897: Upgrade RocksDB to 6.8.1

[GitBox <gi...@apache.org>]

cadonna commented on a change in pull request #10587:
URL: https://github.com/apache/kafka/pull/10587#discussion_r626469074



##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
##########
@@ -99,35 +101,7 @@ public Env getEnv() {
 
     @Override
     public Options prepareForBulkLoad() {
-        /* From https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ
-         *
-         * Q: What's the fastest way to load data into RocksDB?
-         *
-         * A: A fast way to direct insert data to the DB:
-         *
-         *  1. using single writer thread and insert in sorted order
-         *  2. batch hundreds of keys into one write batch
-         *  3. use vector memtable
-         *  4. make sure options.max_background_flushes is at least 4
-         *  5. before inserting the data,
-         *       disable automatic compaction,
-         *       set options.level0_file_num_compaction_trigger,
-         *           options.level0_slowdown_writes_trigger
-         *           and options.level0_stop_writes_trigger to very large.
-         *     After inserting all the data, issue a manual compaction.
-         *
-         * 3-5 will be automatically done if you call Options::PrepareForBulkLoad() to your option
-         */
-        // (1) not in our control
-        // (2) is done via bulk-loading API
-        // (3) skipping because, not done in actual PrepareForBulkLoad() code in https://github.com/facebook/rocksdb/blob/master/options/options.cc
-        //columnFamilyOptions.setMemTableConfig(new VectorMemTableConfig());
-        // (4-5) below:
-        dbOptions.setMaxBackgroundFlushes(4);
-        columnFamilyOptions.setDisableAutoCompactions(true);
-        columnFamilyOptions.setLevel0FileNumCompactionTrigger(1 << 30);
-        columnFamilyOptions.setLevel0SlowdownWritesTrigger(1 << 30);
-        columnFamilyOptions.setLevel0StopWritesTrigger(1 << 30);
+        super.prepareForBulkLoad();

Review comment:
       TBH, I do not understand why we set the options here instead of simply calling `super.prepareForBulkLoad()`. Also in the version that we currently use (5.18.4) the options are set like here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org