You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by "ruojieranyishen (via GitHub)" <gi...@apache.org> on 2023/05/25 12:20:20 UTC
[GitHub] [incubator-pegasus] ruojieranyishen opened a new issue, #1488: Feature:Online Query and Dynamic Modification of Table-level RocksDB Options
ruojieranyishen opened a new issue, #1488:
URL: https://github.com/apache/incubator-pegasus/issues/1488
### Background
Currently, in the Pegasus 2.4.1 version, dynamic querying and modification of RocksDB options at the table level are not supported. While the Pegasus shell provides commands like `set_app_envs `and `create` for configuring certain table-level options, it does not include RocksDB options. Consequently, adding the configure table-level RocksDB options ability would enhance the app configuration flexibility in Pegasus.
### Goals
This issue aims to achieve the following goals:
1. Provide online modification functionality for table-level RocksDB configuration.
2. Provide online query functionality for table-level RocksDB configuration.
3. Research on the principles, interdependencies, and modification risks of the rocksdb options used in Pegasus with rocksdb-6.6.4 version, and compile a comprehensive documentation on the usage of table-level rocksdb options.
4. By modifying the logic of the Usage Scenario feature, unify the approach for setting table-level RocksDB options.
### Solution
Here are the solutions for goals 1 and 2.
The storage location of rocksdb option is as follows:
1. Utilize the config.ini file to provide default global RocksDB options information.
2. Store table-specific RocksDB options in the envs section of the app_info in ZooKeeper (zk).
Three scenarios to set specific rocksdb option：
1. [Create a replica with specific rocksdb option]
Set specify option by `create appname -e rocksdb.options ` command
If a replica is not created, use the app envs rocksdb option from metaserver to create. For a already created replica, because the app envs is not transmitted from the metaserver, use options from the RocksDB OPTION file as the reference.
2. [Modify a specific rocksdb option of an online replica]
Set specify option by `set_app_envs rocksdb.options` command
3. [Query the modified option online]
Get specify option by `get_app_envs` command
To validate the feasibility of the proposed solution, I will use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] ruojieranyishen commented on issue #1488: Feature:Online Query and Dynamic Modification of Table-level RocksDB Options
Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on issue #1488:
URL: https://github.com/apache/incubator-pegasus/issues/1488#issuecomment-1590742429
This is the specific creation process of replica, including my two modifications. Respectively deal with creating a new rocksdb and restarting rocksdb.
### Create a replica key process：
1. `new replica` ： Create a replica, which contains env information num_levels=5, and the actual rocksdb has not been created
2. `store_app_info(_app_info)` ： Persist app_info information to .app_info file
3. `_app.reset(replication_app_base::new_storage_instance(_app_info.app_type, this));` ：Construct pegasus_server_impl. Parameter this pointer is replica*, which contains env information num_levels=5.
4. `pegasus_server_impl::pegasus_server_impl `：Construct pegasus_server_impl. Initialize rocksdb option via FLAGS_rocksdb_*
5. `update_app_envs_before_open_db(envs);` ：Make the env information effective before creating the rocksdb instance.
6. `reset_usage_scenario_options` ： Load rocksdb option from OPTION-XXXXX file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] ruojieranyishen commented on issue #1488: Feature:Online Query and Dynamic Modification of Table-level RocksDB Options
Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on issue #1488:
URL: https://github.com/apache/incubator-pegasus/issues/1488#issuecomment-1590742937
I show the set_app_envs procedure using the write_buffer_size option. My third modification solves the online modification of dynamic rocksdb options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] ruojieranyishen commented on issue #1488: Feature:Online Query and Dynamic Modification of Table-level RocksDB Options
Posted by "ruojieranyishen (via GitHub)" <gi...@apache.org>.
ruojieranyishen commented on issue #1488:
URL: https://github.com/apache/incubator-pegasus/issues/1488#issuecomment-1590741238
This is the app creation process, I show the app envs passing through the num_levels option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org