[jira] [Created] (FLINK-15068) Disable RocksDB's local LOG by default

["Nico Kruber (Jira)" <ji...@apache.org>]

Nico Kruber created FLINK-15068:
-----------------------------------

             Summary: Disable RocksDB's local LOG by default
                 Key: FLINK-15068
                 URL: https://issues.apache.org/jira/browse/FLINK-15068
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / State Backends
    Affects Versions: 1.9.1, 1.8.2, 1.7.2
            Reporter: Nico Kruber
            Assignee: Nico Kruber
             Fix For: 1.10.0


With Flink's default settings for RocksDB, it will write a log file (not the WAL, but pure logging statements) into the data folder. Besides periodic statistics, it will log compaction attempts, new memtable creations, flushes, etc.

A few things to note about this practice:
 # *this LOG file is growing over time with no limit (!)*
 # the default logging level is INFO
 # the statistics in there may help looking into performance and/or disk space problems (but maybe you should be looking and monitoring metrics instead)
 # this file is not useful for debugging errors since it will be deleted along with the local dir when the TM goes down

With a custom \{{OptionsFactory}}, the user can change the behaviour like the following:
{code:java}
    @Override
    public DBOptions createDBOptions(DBOptions currentOptions) {
        currentOptions = super.createDBOptions(currentOptions);

        currentOptions.setKeepLogFileNum(10);
        currentOptions.setInfoLogLevel(InfoLogLevel.WARN_LEVEL);
        currentOptions.setStatsDumpPeriodSec(0);
        currentOptions.setMaxLogFileSize(1024 * 1024); // 1 MB each

        return currentOptions;
    }{code}
However, the rotating logger does currently not work (it will not delete old log files - see [https://github.com/dataArtisans/frocksdb/pull/12]). Also, the user should not have to write his own {{OptionsFactory}} to get a sensible default.

To prevent this file from filling up the disk, I propose to change Flink's default RocksDB settings so that the LOG file is effectively disabled (nothing is written to it by default).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)