You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "justinwwhuang (via GitHub)" <gi...@apache.org> on 2024/04/01 02:14:25 UTC

[PR] [INLONG-9906][Agent] Add configuration comparison logic and processing of comparison results [inlong]

justinwwhuang opened a new pull request, #9908:
URL: https://github.com/apache/inlong/pull/9908

   [INLONG-9906][Agent] Add configuration comparison logic and processing of comparison results
   - Fixes #9906
   
   ### Motivation
   
   Add configuration comparison logic and processing of comparison results
   ### Modifications
   
   Add configuration comparison logic and processing of comparison results
   
   ### Verifying this change
   
   *(Please pick either of the following options)*
   
   - [ ] This change is a trivial rework/code cleanup without any test coverage.
   
   - [ ] This change is already covered by existing tests, such as:
     *(please describe tests)*
   
   - [ ] This change added tests and can be verified as follows:
   
     *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
     - If a feature is not applicable for documentation, explain why?
     - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [INLONG-9906][Agent] Add configuration comparison logic and processing of comparison results [inlong]

Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #9908:
URL: https://github.com/apache/inlong/pull/9908#discussion_r1545930338


##########
inlong-agent/agent-installer/src/main/java/org/apache/inlong/agent/installer/ModuleManager.java:
##########
@@ -216,6 +218,147 @@ private boolean updateModules(List<ModuleConfig> managerModuleList) {
         managerModuleList.forEach((moduleConfig) -> {
             modulesFromManager.put(moduleConfig.getId(), moduleConfig);
         });
+        traverseManagerModulesToLocal(modulesFromManager);
+        traverseLocalModulesToManager(modulesFromManager);
+        return true;
+    }
+
+    private void traverseManagerModulesToLocal(Map<Integer, ModuleConfig> modulesFromManager) {
+        modulesFromManager.values().forEach((managerModule) -> {
+            ModuleConfig localModule = currentModules.get(managerModule.getId());
+            if (localModule == null) {
+                LOGGER.info("traverseManagerModulesToLocal module {} {} {} not found in local, add it",
+                        managerModule.getId(), managerModule.getName(), managerModule.getVersion());
+                addModule(managerModule);
+            } else {
+                if (managerModule.getMd5().equals(localModule.getMd5())) {
+                    LOGGER.info("traverseManagerModulesToLocal module {} {} {} md5 no change, do nothing",
+                            localModule.getId(), localModule.getName(), localModule.getVersion());
+                } else {
+                    LOGGER.info("traverseManagerModulesToLocal module {} {} {} md5 changed, update it",
+                            localModule.getId(), localModule.getName(), localModule.getVersion());
+                    updateModule(localModule, managerModule);
+                }
+            }
+        });
+    }
+
+    private void traverseLocalModulesToManager(Map<Integer, ModuleConfig> modulesFromManager) {
+        currentModules.values().forEach((localModule) -> {
+            ModuleConfig managerModule = modulesFromManager.get(localModule.getId());
+            if (managerModule == null) {
+                LOGGER.info("traverseLocalModulesToManager module {} {} {} not found in local, delete it",
+                        localModule.getId(), localModule.getName(), localModule.getVersion());
+                deleteModule(localModule);
+            }
+        });
+    }
+
+    private void addModule(ModuleConfig module) {
+        LOGGER.info("add module {} start", module.getName());
+        addAndSaveModuleConfig(module);
+        if (!downloadModule(module)) {
+            LOGGER.error("add module {} but download failed", module.getName());
+            return;
+        }
+        saveModuleState(module.getId(), ModuleStateEnum.DOWNLOADED);
+        installModule(module);
+        saveModuleState(module.getId(), ModuleStateEnum.INSTALLED);
+        startModule(module);
+        LOGGER.info("add module {} end", module.getId());
+    }
+
+    private void deleteModule(ModuleConfig module) {
+        LOGGER.info("delete module {} start", module.getId());
+        stopModule(module);
+        uninstallModule(module);
+        deleteAndSaveModuleConfig(module);
+        LOGGER.info("delete module {} end", module.getId());
+    }
+
+    private void updateModule(ModuleConfig localModule, ModuleConfig managerModule) {
+        LOGGER.info("update module {} start", localModule.getId());
+        deleteModule(localModule);
+        addModule(managerModule);
+        LOGGER.info("update module {} end", localModule.getId());
+    }
+
+    private void addAndSaveModuleConfig(ModuleConfig module) {
+        module.setState(ModuleStateEnum.NEW);
+        if (currentModules.containsKey(module.getId())) {
+            LOGGER.error("should not happen! module {} found! will force to replace it!", module.getId());
+        }
+        currentModules.put(module.getId(), module);
+        saveToLocalFile(confPath);
+    }
+
+    private void deleteAndSaveModuleConfig(ModuleConfig module) {
+        if (!currentModules.containsKey(module.getId())) {
+            LOGGER.error("should not happen! module {} not found!", module.getId());
+            return;
+        }
+        currentModules.remove(module.getId());
+        saveToLocalFile(confPath);
+    }
+
+    private boolean saveModuleState(Integer moduleId, ModuleStateEnum state) {
+        ModuleConfig module = currentModules.get(moduleId);
+        if (module == null) {
+            LOGGER.error("should not happen! module {} not found!", moduleId);
+            return false;
+        }
+        LOGGER.info("save module state to {} {}", moduleId, state);
+        module.setState(state);
+        saveToLocalFile(confPath);
+        return true;
+    }
+
+    private void installModule(ModuleConfig module) {
+        LOGGER.info("install module {} with cmd {}", module.getId(), module.getInstallCommand());
+        String ret = ExcuteLinux.exeCmd(module.getInstallCommand());
+        LOGGER.info("install module {} return {} ", module.getId(), ret);
+    }
+
+    private boolean startModule(ModuleConfig module) {
+        LOGGER.info("start module {} with cmd {}", module.getId(), module.getStartCommand());
+        for (int i = 0; i < module.getProcessesNum(); i++) {
+            String ret = ExcuteLinux.exeCmd(module.getStartCommand());
+            LOGGER.info("start [{}] module {} return {} ", i, module.getId(), ret);
+        }
+        if (isProcessAllStarted(module)) {
+            LOGGER.info("start module {} success", module.getId());
+            return true;
+        } else {
+            LOGGER.info("start module {} failed", module.getId());
+            return false;
+        }
+    }
+
+    private void stopModule(ModuleConfig module) {
+        LOGGER.info("stop module {} with cmd {}", module.getId(), module.getStopCommand());
+        String ret = ExcuteLinux.exeCmd(module.getStopCommand());
+        LOGGER.info("stop module {} return {} ", module.getId(), ret);
+    }
+
+    private void uninstallModule(ModuleConfig module) {
+        LOGGER.info("uninstall module {} with cmd {}", module.getId(), module.getUninstallCommand());
+        String ret = ExcuteLinux.exeCmd(module.getUninstallCommand());
+        LOGGER.info("uninstall module {} return {} ", module.getId(), ret);
+    }
+
+    private boolean isProcessAllStarted(ModuleConfig module) {
+        String ret = ExcuteLinux.exeCmd(module.getCheckCommand());
+        String[] processArray = ret.split("\n");
+        int cnt = 0;
+        for (int i = 0; i < processArray.length; i++) {
+            if (processArray[i].length() > 0) {
+                cnt++;
+            }
+        }
+        LOGGER.info("get module process num {} {}", module.getName(), cnt);
+        if (cnt < module.getProcessesNum()) {

Review Comment:
   return cnt >= module.getProcessesNum();



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [INLONG-9906][Agent] Add configuration comparison logic and processing of comparison results [inlong]

Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang commented on code in PR #9908:
URL: https://github.com/apache/inlong/pull/9908#discussion_r1545953050


##########
inlong-agent/agent-installer/src/main/java/org/apache/inlong/agent/installer/ModuleManager.java:
##########
@@ -216,9 +218,158 @@ private boolean updateModules(List<ModuleConfig> managerModuleList) {
         managerModuleList.forEach((moduleConfig) -> {
             modulesFromManager.put(moduleConfig.getId(), moduleConfig);
         });
+        traverseManagerModulesToLocal(modulesFromManager);
+        traverseLocalModulesToManager(modulesFromManager);
         return true;
     }
 
+    private void traverseManagerModulesToLocal(Map<Integer, ModuleConfig> modulesFromManager) {
+        modulesFromManager.values().forEach((managerModule) -> {
+            ModuleConfig localModule = currentModules.get(managerModule.getId());
+            if (localModule == null) {
+                LOGGER.info("traverseManagerModulesToLocal module {} {} {} not found in local, add it",
+                        managerModule.getId(), managerModule.getName(), managerModule.getVersion());
+                addModule(managerModule);
+            } else {
+                if (managerModule.getMd5().equals(localModule.getMd5())) {
+                    LOGGER.info("traverseManagerModulesToLocal module {} {} {} md5 no change, do nothing",
+                            localModule.getId(), localModule.getName(), localModule.getVersion());
+                } else {
+                    LOGGER.info("traverseManagerModulesToLocal module {} {} {} md5 changed, update it",
+                            localModule.getId(), localModule.getName(), localModule.getVersion());
+                    updateModule(localModule, managerModule);
+                }
+            }
+        });
+    }
+
+    private void traverseLocalModulesToManager(Map<Integer, ModuleConfig> modulesFromManager) {
+        currentModules.values().forEach((localModule) -> {
+            ModuleConfig managerModule = modulesFromManager.get(localModule.getId());
+            if (managerModule == null) {
+                LOGGER.info("traverseLocalModulesToManager module {} {} {} not found in local, delete it",
+                        localModule.getId(), localModule.getName(), localModule.getVersion());
+                deleteModule(localModule);
+            }
+        });
+    }
+
+    private void addModule(ModuleConfig module) {
+        LOGGER.info("add module {} start", module.getName());
+        addAndSaveModuleConfig(module);
+        if (!downloadModule(module)) {
+            LOGGER.error("add module {} but download failed", module.getName());
+            return;
+        }
+        saveModuleState(module.getId(), ModuleStateEnum.DOWNLOADED);
+        installModule(module);
+        saveModuleState(module.getId(), ModuleStateEnum.INSTALLED);
+        startModule(module);
+        LOGGER.info("add module {} end", module.getId());
+    }
+
+    private void deleteModule(ModuleConfig module) {
+        LOGGER.info("delete module {} start", module.getId());
+        stopModule(module);
+        uninstallModule(module);
+        deleteAndSaveModuleConfig(module);
+        LOGGER.info("delete module {} end", module.getId());
+    }
+
+    private void updateModule(ModuleConfig localModule, ModuleConfig managerModule) {
+        LOGGER.info("update module {} start", localModule.getId());
+        if (localModule.getPackageConfig().getMd5().equals(managerModule.getPackageConfig().getMd5())) {
+            LOGGER.info("package md5 changed, will reinstall", localModule.getId());
+            deleteModule(localModule);
+            addModule(managerModule);
+        } else {
+            LOGGER.info("package md5 no chang, will restart", localModule.getId());
+            restartModule(localModule, managerModule);
+        }
+        LOGGER.info("update module {} end", localModule.getId());
+    }
+
+    private void addAndSaveModuleConfig(ModuleConfig module) {
+        module.setState(ModuleStateEnum.NEW);
+        if (currentModules.containsKey(module.getId())) {
+            LOGGER.error("should not happen! module {} found! will force to replace it!", module.getId());
+        }
+        currentModules.put(module.getId(), module);
+        saveToLocalFile(confPath);
+    }
+
+    private void deleteAndSaveModuleConfig(ModuleConfig module) {
+        if (!currentModules.containsKey(module.getId())) {
+            LOGGER.error("should not happen! module {} not found!", module.getId());
+            return;
+        }
+        currentModules.remove(module.getId());
+        saveToLocalFile(confPath);
+    }
+
+    private boolean saveModuleState(Integer moduleId, ModuleStateEnum state) {
+        ModuleConfig module = currentModules.get(moduleId);
+        if (module == null) {
+            LOGGER.error("should not happen! module {} not found!", moduleId);
+            return false;
+        }
+        LOGGER.info("save module state to {} {}", moduleId, state);
+        module.setState(state);
+        saveToLocalFile(confPath);

Review Comment:
   ```suggestion
           module.setState(state);
           saveToLocalFile(confPath);
           LOGGER.info("save module state to {} {}", moduleId, state);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [INLONG-9906][Agent] Add configuration comparison logic and processing of comparison results [inlong]

Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang merged PR #9908:
URL: https://github.com/apache/inlong/pull/9908


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org