You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2022/06/17 13:42:05 UTC
[GitHub] [incubator-pegasus] empiredan opened a new pull request, #1009: Restrict info log size
empiredan opened a new pull request, #1009:
URL: https://github.com/apache/incubator-pegasus/pull/1009
# 1. Motivation
This PR is to solve the problem of https://github.com/apache/incubator-pegasus/issues/994.
The replica server can run for a very long without any fault. Each instance of rocksdb will generate a lot of info logs that occupies considerable disk spaces. Thus each instance of rocksdb should clear out-dated logs themselves periodically.
# 2. Implementation
We can use 3 options provided by `rocksdb::DBOptions` to restrict the size of info logs:
- `max_log_file_size`
- `log_file_time_to_roll`
- `keep_log_file_num`
## 2.1 `max_log_file_size`
This option specifies the maximal size of the info log file.
Once the log file is larger than this option, a new info log file will be created.
If this option is set to 0, all logs will be written to one log file.");
## 2.2 `log_file_time_to_roll`
This options specifies time for the info log file to roll (in seconds).
If this option is specified with non-zero value, log file will be rolled if it has been active longer than this option.
Otherwise, if this options is set to 0, log file will never be rolled by life time.
## 2.3 `keep_log_file_num`
This options specifies the maximal numbers of info log files to be kept.
Once the number of info logs goes beyond this option, stale log files will be cleaned.
## 2.4 default values
In production environment, it has been observed that an instance of rocksdb about 20GB in total size which has run beyond 199 days, generated a big log file sized 96MB, with 492KB for each day.
Accordingly, default value for `rocksdb_log_file_time_to_roll` can be set to one day, and that for `rocksdb_keep_log_file_num` can be set to 32, which means log files for recent one month will be reserved.
On the other hand, the max size of a log file is also be restricted to 8MB. In practice, the size of logs over a day tends to be less than 1MB; however, once errors are reported very frequently, the log file will grow larger and go far beyond several hundreds of KB.
# 3. Configurations
The added configurations are listed below, which have been set to the default values:
```diff
+ rocksdb_max_log_file_size = 8388608
+ rocksdb_log_file_time_to_roll = 86400
+ rocksdb_keep_log_file_num = 32
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] levy5307 merged pull request #1009: feat: support to restrict the size of rocksdb info logs
Posted by GitBox <gi...@apache.org>.
levy5307 merged PR #1009:
URL: https://github.com/apache/incubator-pegasus/pull/1009
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 commented on a diff in pull request #1009: feat: support to restrict the size of rocksdb info logs
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on code in PR #1009:
URL: https://github.com/apache/incubator-pegasus/pull/1009#discussion_r900149636
##########
src/server/pegasus_server_impl_init.cpp:
##########
@@ -95,6 +95,41 @@ DSN_DEFINE_uint64(
"batch-get operation iterate count exceed this threshold will be logged, 0 means no check");
DSN_TAG_VARIABLE(rocksdb_abnormal_batch_get_count_threshold, FT_MUTABLE);
+// In production environment, it has been observed that an instance of rocksdb about 20GB in total
+// size which has run beyond 199 days, generated a big log file sized 96MB, with 492KB for each
+// day.
+//
+// Accordingly, default value for `rocksdb_log_file_time_to_roll` can be set to one day, and
+// that for `rocksdb_keep_log_file_num` can be set to 32, which means log files for recent one
+// month will be reserved.
+//
+// On the other hand, the max size of a log file is also be restricted to 8MB. In practice, the
+// size of logs over a day tends to be less than 1MB; however, once errors are reported very
+// frequently, the log file will grow larger and go far beyond several hundreds of KB.
+DSN_DEFINE_uint64("pegasus.server",
+ rocksdb_max_log_file_size,
+ 8 * 1024 * 1024,
+ "specify the maximal size of the info log file: once the log file is larger "
+ "than this option, a new info log file will be created; if this option is set "
+ "to 0, all logs will be written to one log file.");
+DSN_TAG_VARIABLE(rocksdb_max_log_file_size, FT_MUTABLE);
Review Comment:
it’s hard to say this config is mutable, since for the already running rocksdb instances, this config is immutable.
of course we can find some other way to update these configs, but that’s another matter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] empiredan commented on a diff in pull request #1009: feat: support to restrict the size of rocksdb info logs
Posted by GitBox <gi...@apache.org>.
empiredan commented on code in PR #1009:
URL: https://github.com/apache/incubator-pegasus/pull/1009#discussion_r900701289
##########
src/server/pegasus_server_impl_init.cpp:
##########
@@ -95,6 +95,41 @@ DSN_DEFINE_uint64(
"batch-get operation iterate count exceed this threshold will be logged, 0 means no check");
DSN_TAG_VARIABLE(rocksdb_abnormal_batch_get_count_threshold, FT_MUTABLE);
+// In production environment, it has been observed that an instance of rocksdb about 20GB in total
+// size which has run beyond 199 days, generated a big log file sized 96MB, with 492KB for each
+// day.
+//
+// Accordingly, default value for `rocksdb_log_file_time_to_roll` can be set to one day, and
+// that for `rocksdb_keep_log_file_num` can be set to 32, which means log files for recent one
+// month will be reserved.
+//
+// On the other hand, the max size of a log file is also be restricted to 8MB. In practice, the
+// size of logs over a day tends to be less than 1MB; however, once errors are reported very
+// frequently, the log file will grow larger and go far beyond several hundreds of KB.
+DSN_DEFINE_uint64("pegasus.server",
+ rocksdb_max_log_file_size,
+ 8 * 1024 * 1024,
+ "specify the maximal size of the info log file: once the log file is larger "
+ "than this option, a new info log file will be created; if this option is set "
+ "to 0, all logs will be written to one log file.");
+DSN_TAG_VARIABLE(rocksdb_max_log_file_size, FT_MUTABLE);
Review Comment:
> it’s hard to say this config is mutable, since for the already running rocksdb instances, this config is immutable. of course we can find some other way to update these configs, but that’s another matter.
Yes, actually all of these options are also immutable for rocksdb. The only method that reload these options dynamically, is to update these options by flags firstly for each replica server, then kill partitions and reopen them.
However, if the options of lots of partitions should be reload, instead of killing partitions, it's better to update options in configuration file and then restart the replica servers. Thus if it's necessary, we can launch another PR to update options dynamically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 commented on a diff in pull request #1009: feat: support to restrict the size of rocksdb info logs
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on code in PR #1009:
URL: https://github.com/apache/incubator-pegasus/pull/1009#discussion_r900702343
##########
src/server/pegasus_server_impl_init.cpp:
##########
@@ -95,6 +95,41 @@ DSN_DEFINE_uint64(
"batch-get operation iterate count exceed this threshold will be logged, 0 means no check");
DSN_TAG_VARIABLE(rocksdb_abnormal_batch_get_count_threshold, FT_MUTABLE);
+// In production environment, it has been observed that an instance of rocksdb about 20GB in total
+// size which has run beyond 199 days, generated a big log file sized 96MB, with 492KB for each
+// day.
+//
+// Accordingly, default value for `rocksdb_log_file_time_to_roll` can be set to one day, and
+// that for `rocksdb_keep_log_file_num` can be set to 32, which means log files for recent one
+// month will be reserved.
+//
+// On the other hand, the max size of a log file is also be restricted to 8MB. In practice, the
+// size of logs over a day tends to be less than 1MB; however, once errors are reported very
+// frequently, the log file will grow larger and go far beyond several hundreds of KB.
+DSN_DEFINE_uint64("pegasus.server",
+ rocksdb_max_log_file_size,
+ 8 * 1024 * 1024,
+ "specify the maximal size of the info log file: once the log file is larger "
+ "than this option, a new info log file will be created; if this option is set "
+ "to 0, all logs will be written to one log file.");
+DSN_TAG_VARIABLE(rocksdb_max_log_file_size, FT_MUTABLE);
Review Comment:
Yes, we can do it in the next pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org