[jira] [Commented] (FLINK-15068) Disable RocksDB's local LOG by default

["Nico Kruber (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022957#comment-17022957 ] 

Nico Kruber commented on FLINK-15068:
-------------------------------------

Thanks for adding the release note, this is probably good to have.

> Disable RocksDB's local LOG by default
> --------------------------------------
>
>                 Key: FLINK-15068
>                 URL: https://issues.apache.org/jira/browse/FLINK-15068
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / State Backends
>    Affects Versions: 1.7.2, 1.8.2, 1.9.1
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.10.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> With Flink's default settings for RocksDB, it will write a log file (not the WAL, but pure logging statements) into the data folder. Besides periodic statistics, it will log compaction attempts, new memtable creations, flushes, etc.
> A few things to note about this practice:
>  # *this LOG file is growing over time with no limit (!)*
>  # the default logging level is INFO
>  # the statistics in there may help looking into performance and/or disk space problems (but maybe you should be looking and monitoring metrics instead)
>  # this file is not useful for debugging errors since it will be deleted along with the local dir when the TM goes down
> With a custom \{{OptionsFactory}}, the user can change the behaviour like the following:
> {code:java}
>     @Override
>     public DBOptions createDBOptions(DBOptions currentOptions) {
>         currentOptions = super.createDBOptions(currentOptions);
>         currentOptions.setKeepLogFileNum(10);
>         currentOptions.setInfoLogLevel(InfoLogLevel.WARN_LEVEL);
>         currentOptions.setStatsDumpPeriodSec(0);
>         currentOptions.setMaxLogFileSize(1024 * 1024); // 1 MB each
>         return currentOptions;
>     }{code}
> However, the rotating logger does currently not work (it will not delete old log files - see [https://github.com/dataArtisans/frocksdb/pull/12]). Also, the user should not have to write his own {{OptionsFactory}} to get a sensible default.
> To prevent this file from filling up the disk, I propose to change Flink's default RocksDB settings so that the LOG file is effectively disabled (nothing is written to it by default).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)