You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by "ninsmiracle (via GitHub)" <gi...@apache.org> on 2024/08/20 02:12:08 UTC

[PR] feat: dynamic reload duplication config [incubator-pegasus]

ninsmiracle opened a new pull request, #2103:
URL: https://github.com/apache/incubator-pegasus/pull/2103

   ### What problem does this PR solve? <!--add issue link with summary if exists-->
   #2102 
   
   ### What is changed and how does it work?
     If the cluster maintainer push new config to target cluster, you can use pegasus-shell to make cluster reload new duplication config without nodes restart.
     I add some function to help servers get their config on time, and we can reload new config ONLY about duplication.
   
   
   ##### Tests <!-- At least one of them must be included. -->
   - Unit test
   - Manual test (add detailed scripts or steps below)
   - Cluster test
   
   ##### Code changes
   
   - Has exported function/method change
   Move and copy constructors of configuration class.
   Some function about reloading dup parameter into memory instance.
   
   ##### Related changes
   - Need to update the user documentation
   - Need to be included in the release note
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] feat: dynamic reload duplication config [incubator-pegasus]

Posted by "ninsmiracle (via GitHub)" <gi...@apache.org>.
ninsmiracle commented on code in PR #2103:
URL: https://github.com/apache/incubator-pegasus/pull/2103#discussion_r1726960500


##########
src/utils/configuration.h:
##########
@@ -43,11 +44,68 @@ namespace dsn {
 
 class configuration
 {
+private:
+    struct conf
+    {
+        std::string section;
+        std::string key;
+        std::string value;
+        int line;
+
+        bool present;

Review Comment:
   At begin I declaring `configuration(config_map&)` to use the mutex lock the map. If I not move private parameter before public function, an error will be reported during the compilation phase: the variable cannot be found.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] feat: dynamic reload duplication config [incubator-pegasus]

Posted by "ninsmiracle (via GitHub)" <gi...@apache.org>.
ninsmiracle commented on code in PR #2103:
URL: https://github.com/apache/incubator-pegasus/pull/2103#discussion_r1726929235


##########
src/common/test/err-config-test.ini:
##########
@@ -0,0 +1,106 @@
+; The MIT License (MIT)

Review Comment:
   This is a config file that I copy the file header of `src/common/test/config-test.ini`. And I don't know how to write the copyright owner of this config file.
   ```
   Copyright [yyyy] [name of copyright owner]
   
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] feat: dynamic reload duplication config [incubator-pegasus]

Posted by "acelyc111 (via GitHub)" <gi...@apache.org>.
acelyc111 commented on code in PR #2103:
URL: https://github.com/apache/incubator-pegasus/pull/2103#discussion_r1724706007


##########
src/common/duplication_common.cpp:
##########
@@ -103,6 +105,59 @@ class duplication_group_registry : public utils::singleton<duplication_group_reg
     }
 
     const std::set<uint8_t> &get_distinct_cluster_id_set() { return _distinct_cids; }
+    error_with<uint8_t> reload_duplication_config(std::string config_file)
+    {
+        if (config_file.length() == 0) {

Review Comment:
   ```suggestion
           if (config_file.empty()   ) {
   ```



##########
src/common/duplication_common.cpp:
##########
@@ -103,6 +105,59 @@ class duplication_group_registry : public utils::singleton<duplication_group_reg
     }
 
     const std::set<uint8_t> &get_distinct_cluster_id_set() { return _distinct_cids; }
+    error_with<uint8_t> reload_duplication_config(std::string config_file)
+    {
+        if (config_file.length() == 0) {
+            config_file = "config.ini";
+        }
+        const char *config_file_cstr = config_file.c_str();
+
+        std::map<std::string, uint8_t> new_group;
+        std::set<uint8_t> new_distinct_cids;
+        dsn::configuration old_config;
+
+        // reload default config.ini, user can point to another config file. update g_config here
+        if (!dsn_config_reload(config_file_cstr, nullptr, &old_config)) {
+            LOG_ERROR("Fail to reload config file {} \n", config_file_cstr);
+            return error_s::make(
+                ERR_OBJECT_NOT_FOUND,
+                " new `duplication-group` configured can not be read. Check your config.ini now");
+            ;

Review Comment:
   ```suggestion
   ```



##########
src/common/test/err-config-test.ini:
##########
@@ -0,0 +1,106 @@
+; The MIT License (MIT)

Review Comment:
   Use Apache 2.0 license.



##########
src/common/duplication_common.cpp:
##########
@@ -103,6 +105,59 @@ class duplication_group_registry : public utils::singleton<duplication_group_reg
     }
 
     const std::set<uint8_t> &get_distinct_cluster_id_set() { return _distinct_cids; }
+    error_with<uint8_t> reload_duplication_config(std::string config_file)
+    {
+        if (config_file.length() == 0) {
+            config_file = "config.ini";
+        }
+        const char *config_file_cstr = config_file.c_str();
+
+        std::map<std::string, uint8_t> new_group;
+        std::set<uint8_t> new_distinct_cids;
+        dsn::configuration old_config;
+
+        // reload default config.ini, user can point to another config file. update g_config here
+        if (!dsn_config_reload(config_file_cstr, nullptr, &old_config)) {
+            LOG_ERROR("Fail to reload config file {} \n", config_file_cstr);

Review Comment:
   There are some format issues, please check them.
   ```suggestion
               LOG_ERROR("Fail to reload config file {}", config_file_cstr);
   ```



##########
src/utils/config_api.cpp:
##########
@@ -39,6 +40,23 @@ bool dsn_config_load(const char *file, const char *arguments)
 
 void dsn_config_dump(std::ostream &os) { g_config.dump(os); }
 
+bool dsn_config_reload(const char *file,
+                       const char *arguments,
+                       /*out*/ dsn::configuration *old_config)
+{
+    *old_config = g_config;
+    dsn::configuration temp_config;
+    if (!temp_config.load(file, arguments)) {
+        // todo gns: add some error log

Review Comment:
   Use the standard TODO style.



##########
src/utils/configuration.h:
##########
@@ -43,11 +44,68 @@ namespace dsn {
 
 class configuration
 {
+private:
+    struct conf
+    {
+        std::string section;
+        std::string key;
+        std::string value;
+        int line;
+
+        bool present;

Review Comment:
   Could you please add some comments when you happened to modify the code? Thanks!



##########
src/utils/configuration.cpp:
##########
@@ -51,6 +51,20 @@ configuration::~configuration()
     _configs.clear();
 }
 
+void configuration::copy_configs(configuration &source_conf)
+{
+    source_conf._lock.lock();

Review Comment:
   _configs should be empty before filling values, right? Please check it.



##########
src/utils/configuration.cpp:
##########
@@ -51,6 +51,20 @@ configuration::~configuration()
     _configs.clear();
 }
 
+void configuration::copy_configs(configuration &source_conf)
+{
+    source_conf._lock.lock();
+    for (auto &section_kv : source_conf._configs) {

Review Comment:
   Use C++ 17 style.
   
   ```
   for (const auto [a, b] : xxx) {
     ...
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] feat: dynamic reload duplication config [incubator-pegasus]

Posted by "ninsmiracle (via GitHub)" <gi...@apache.org>.
ninsmiracle commented on code in PR #2103:
URL: https://github.com/apache/incubator-pegasus/pull/2103#discussion_r1726946885


##########
src/utils/configuration.cpp:
##########
@@ -51,6 +51,20 @@ configuration::~configuration()
     _configs.clear();
 }
 
+void configuration::copy_configs(configuration &source_conf)
+{
+    source_conf._lock.lock();

Review Comment:
   Before I call `configuration::copy_configs`, it  will call  `configuration::clear_configs` to make `_configs` clearly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] feat: dynamic reload duplication config [incubator-pegasus]

Posted by "ninsmiracle (via GitHub)" <gi...@apache.org>.
ninsmiracle commented on PR #2103:
URL: https://github.com/apache/incubator-pegasus/pull/2103#issuecomment-2297839450

   ### How to use?
   1. Push new config file (config.ini) with your own method.
   2. Open pegasus-shell and type `remote_command -t all dup-config-reload `
   3. Begin duplication as normal. Just like: `dup add -c c3tst-performance3 -p` 
   
   ### What can it do?
   This PR can ONLY reload the parameter about duplication, so that it will be safety.
   ```
   [[pegasus.clusters]]
      %{cluster.name} = %{meta_address}
       # duplication clusters already have
      c4tst-master = 10.xxx.xx.xx:22601,10.xxx.xx.x:22601
      c3tst-backup = 10.xxx.xx.xx:37001,10.xxx.xx.x:37001
      # If you want to add another cluster in this duplication situation
      xxxxx = xxxxxxxxxxxx
      
     [[duplication-group]]
     c4tst-master = 1 
     c3tst-backup = 2 
     xxxx = 3
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


Re: [PR] feat: dynamic reload duplication config [incubator-pegasus]

Posted by "acelyc111 (via GitHub)" <gi...@apache.org>.
acelyc111 commented on PR #2103:
URL: https://github.com/apache/incubator-pegasus/pull/2103#issuecomment-2370785943

   Thanks for the contribution! Before this pull request, let's discuss clearly the requirement in https://github.com/apache/incubator-pegasus/issues/2102.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org