You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Stephan Ewen (Jira)" <ji...@apache.org> on 2019/12/07 15:38:00 UTC
[jira] [Closed] (FLINK-15068) Disable RocksDB's local LOG by
default
[ https://issues.apache.org/jira/browse/FLINK-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephan Ewen closed FLINK-15068.
--------------------------------
> Disable RocksDB's local LOG by default
> --------------------------------------
>
> Key: FLINK-15068
> URL: https://issues.apache.org/jira/browse/FLINK-15068
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.7.2, 1.8.2, 1.9.1
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> With Flink's default settings for RocksDB, it will write a log file (not the WAL, but pure logging statements) into the data folder. Besides periodic statistics, it will log compaction attempts, new memtable creations, flushes, etc.
> A few things to note about this practice:
> # *this LOG file is growing over time with no limit (!)*
> # the default logging level is INFO
> # the statistics in there may help looking into performance and/or disk space problems (but maybe you should be looking and monitoring metrics instead)
> # this file is not useful for debugging errors since it will be deleted along with the local dir when the TM goes down
> With a custom \{{OptionsFactory}}, the user can change the behaviour like the following:
> {code:java}
> @Override
> public DBOptions createDBOptions(DBOptions currentOptions) {
> currentOptions = super.createDBOptions(currentOptions);
> currentOptions.setKeepLogFileNum(10);
> currentOptions.setInfoLogLevel(InfoLogLevel.WARN_LEVEL);
> currentOptions.setStatsDumpPeriodSec(0);
> currentOptions.setMaxLogFileSize(1024 * 1024); // 1 MB each
> return currentOptions;
> }{code}
> However, the rotating logger does currently not work (it will not delete old log files - see [https://github.com/dataArtisans/frocksdb/pull/12]). Also, the user should not have to write his own {{OptionsFactory}} to get a sensible default.
> To prevent this file from filling up the disk, I propose to change Flink's default RocksDB settings so that the LOG file is effectively disabled (nothing is written to it by default).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)