You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/11/06 08:19:45 UTC
[GitHub] [dolphinscheduler] DarkAssassinator opened a new pull request, #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
DarkAssassinator opened a new pull request, #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
+ This PR will close #11652
+ This is a part of #12436
## Brief change log
**1. Add a Task Remote Host Management**
+ Add a table named `t_ds_task_remote_host` in DB
+ Add a `Task Remote Host Management` page in `Security`
+ Support use test connet this host (use SSH)
**2. Add a column named `remoteHostCode` in `TaskInstance`, `TaskDefinition` and `TaskDefinitionLog` entities.**
**3. Add SSH code for shell task.**
+ Use `ACP` to controll SSH Session. And the pooled object just is session not channel. If DS need channel control in the future, i will add channel pooled object.
+ Limit SFTP upload rate and max file size.
+ Add SFTP monitor process.
## Limitations
+ DS SSH do not support CPU/Memory limit on the remote server. Because other non-DS cluster servers may not necessarily deploy this feature.
+ SSH pool just controll session, not channel.
+ Now just support Shell and Python.
## Verify this pull request
This pull request is already covered by existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1305519120
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [71 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [24.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1017946956
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -2007,3 +2010,25 @@ CREATE TABLE `t_ds_fav_task`
) ENGINE = InnoDB
AUTO_INCREMENT = 1
DEFAULT CHARSET = utf8;
+
+-- ----------------------------
+-- Table structure for t_ds_task_remote_host
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_task_remote_host`;
+CREATE TABLE `t_ds_task_remote_host`
Review Comment:
If we add it to `connection center`, we can also migrate your `Add a Task Remote Host Management page in Security` into `connection center`, which may keep our web UI simple and unify
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1304766879
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [71 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [24.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1313735706
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1022366647
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java:
##########
@@ -190,11 +220,105 @@ public TaskResponse run(String execCommand) throws IOException, InterruptedExcep
return result;
}
+ if (bashTaskExecutionContext.getSessionHost() != null) {
+ runningOnSSH = true;
+ }
+
+ if (runningOnSSH && SystemUtils.IS_OS_WINDOWS) {
+ logger.error("SSH does not support Windows systems");
+ TaskExecutionContextCacheManager.removeByTaskInstanceId(taskInstanceId);
+ return result;
+ }
+
String commandFilePath = buildCommandFilePath();
// create command file if not exists
createCommandFileIfNotExists(execCommand, commandFilePath);
+ result = runCommandFile(commandFilePath);
+
+ return result;
+ }
+
+ private TaskResponse runCommandFile(String commandFilePath) throws IOException, InterruptedException {
+ return runningOnSSH ? runOnSSH(commandFilePath) : runOnLocalProcess(commandFilePath);
+ }
+
+ private TaskResponse runOnSSH(String commandFilePath) {
+ TaskResponse result = new TaskResponse();
+ SSHSessionHolder sessionHolder = null;
+ this.sessionHost = bashTaskExecutionContext.getSessionHost();
+ try {
+ sessionHolder = SSHSessionPool.getSessionHolder(sessionHost);
+ sessionHolder.setSftpConfig(SSHSessionPool.getSftpConfig());
+ logger.info("borrow session:{} success", sessionHolder);
+
+ boolean uploadRes =
+ sessionHolder.sftpDir(taskRequest.getExecutePath(), taskRequest.getExecutePath(), logger);
+ if (!uploadRes) {
+ logger.error("upload task {} execute path to remote session {} failed", taskRequest.getExecutePath(),
+ sessionHost.toString());
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ return result;
+ }
+
+ if (taskRequest.getEnvFile() != null) {
+ boolean uploadEnvRes =
+ sessionHolder.sftpDir(taskRequest.getEnvFile(), taskRequest.getExecutePath(), logger);
+ if (!uploadEnvRes) {
+ logger.error("upload task {} execute path to remote session {} failed", taskRequest.getEnvFile(),
+ sessionHost.toString());
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ return result;
+ }
+ }
+
+ // because JSch .chmod is not stable, so use the 'chmod -R' instead
+ logger.info("update remote path's permission:{} on session:{} to 755", taskRequest.getExecutePath(),
+ sessionHost.toString());
+ String chmodCommand = "chmod -R 755 " + taskRequest.getExecutePath();
Review Comment:
> > > Why it should chmod 755? I'm not sure whether it will cause some CVE? cc @ruanwenjun
> >
> >
> > because remote server should keep same as DS worker, and just chmod the execute path, if there are any worry, we can change to 600. WDYT
>
> If chmod 600 can work, I think it is better.
ohh sorry, my mistake, not 600, chmod 700 should be ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1021607061
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/TaskRemoteHostController.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.controller;
+
+import static org.apache.dolphinscheduler.api.enums.Status.CREATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.DELETE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_PAGE_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.SUCCESS;
+import static org.apache.dolphinscheduler.api.enums.Status.TEST_CONNECT_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.UPDATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.VARIFY_TASK_REMOTE_HOST_ERROR;
+
+import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation;
+import org.apache.dolphinscheduler.api.dto.TaskRemoteHostDTO;
+import org.apache.dolphinscheduler.api.exceptions.ApiException;
+import org.apache.dolphinscheduler.api.service.TaskRemoteHostService;
+import org.apache.dolphinscheduler.api.utils.PageInfo;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.api.vo.TaskRemoteHostVO;
+import org.apache.dolphinscheduler.common.constants.Constants;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestAttribute;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.ResponseStatus;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.Parameters;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+/**
+ * task remote host controller
+ */
+@Tag(name = "TASK_REMOTE_HOST_TAG")
+@RestController
+@RequestMapping("/remote_host")
+public class TaskRemoteHostController {
+
+ @Autowired
+ private TaskRemoteHostService taskRemoteHostService;
+
+ @Operation(summary = "createTaskRemoteHost", description = "CREATE_TASK_REMOTE_HOST_NOTES")
+ @PostMapping(value = "/create")
+ @ResponseStatus(HttpStatus.CREATED)
+ @ApiException(CREATE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result createTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ int result = taskRemoteHostService.createTaskRemoteHost(loginUser, taskRemoteHostDTO);
+ return result > 0 ? Result.success() : Result.error(CREATE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "updateTaskRemoteHost", description = "UPDATE_TASK_REMOTE_HOST_NOTES")
+ @PutMapping(value = "/update/{code}")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(UPDATE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result updateTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @PathVariable("code") Long code,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ int result = taskRemoteHostService.updateTaskRemoteHost(code, loginUser, taskRemoteHostDTO);
+ return result > 0 ? Result.success() : Result.error(UPDATE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "delete", description = "DELETE_TASK_REMOTE_HOST_NOTES")
+ @Parameters({
+ @Parameter(name = "code", description = "TASK_REMOTE_HOST_CODE", schema = @Schema(implementation = long.class, example = "123456", required = true))
+ })
+ @PostMapping(value = "/delete")
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1315216082
Hi @SbloodyS @caishunfeng, please PTAL again, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1313750088
> 4 Security Hotspots
Sure, but these [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) are not really issue, just `password` word
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1017957901
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -2007,3 +2010,25 @@ CREATE TABLE `t_ds_fav_task`
) ENGINE = InnoDB
AUTO_INCREMENT = 1
DEFAULT CHARSET = utf8;
+
+-- ----------------------------
+-- Table structure for t_ds_task_remote_host
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_task_remote_host`;
+CREATE TABLE `t_ds_task_remote_host`
Review Comment:
> > Add a Task Remote Host Management page in Security
>
> If we add it to `connection center`, we can also migrate it into `connection center`, which may keep our web UI simple and unify
yes, but the conclusion we discussed bi-week was to temporarily in `Security`, after `Connection center` fine, we will migrate it into `Connection center`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-code-scanning[bot] commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1014804237
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/TaskRemoteHostController.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.controller;
+
+import static org.apache.dolphinscheduler.api.enums.Status.CREATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.DELETE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_PAGE_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.SUCCESS;
+import static org.apache.dolphinscheduler.api.enums.Status.TEST_CONNECT_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.UPDATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.VARIFY_TASK_REMOTE_HOST_ERROR;
+
+import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation;
+import org.apache.dolphinscheduler.api.dto.TaskRemoteHostDTO;
+import org.apache.dolphinscheduler.api.exceptions.ApiException;
+import org.apache.dolphinscheduler.api.service.TaskRemoteHostService;
+import org.apache.dolphinscheduler.api.utils.PageInfo;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.api.vo.TaskRemoteHostVO;
+import org.apache.dolphinscheduler.common.constants.Constants;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestAttribute;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.ResponseStatus;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.Parameters;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+/**
+ * task remote host controller
+ */
+@Tag(name = "TASK_REMOTE_HOST_TAG")
+@RestController
+@RequestMapping("/remote_host")
+public class TaskRemoteHostController {
+
+ @Autowired
+ private TaskRemoteHostService taskRemoteHostService;
+
+ @Operation(summary = "createTaskRemoteHost", description = "CREATE_TASK_REMOTE_HOST_NOTES")
+ @PostMapping(value = "/create")
+ @ResponseStatus(HttpStatus.CREATED)
+ @ApiException(CREATE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result createTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ int result = taskRemoteHostService.createTaskRemoteHost(loginUser, taskRemoteHostDTO);
+ return result > 0 ? Result.success() : Result.error(CREATE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "updateTaskRemoteHost", description = "UPDATE_TASK_REMOTE_HOST_NOTES")
+ @PutMapping(value = "/update/{code}")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(UPDATE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result updateTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @PathVariable("code") Long code,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ int result = taskRemoteHostService.updateTaskRemoteHost(code, loginUser, taskRemoteHostDTO);
+ return result > 0 ? Result.success() : Result.error(UPDATE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "delete", description = "DELETE_TASK_REMOTE_HOST_NOTES")
+ @Parameters({
+ @Parameter(name = "code", description = "TASK_REMOTE_HOST_CODE", schema = @Schema(implementation = long.class, example = "123456", required = true))
+ })
+ @PostMapping(value = "/delete")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(DELETE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result deleteTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestParam("taskRemoteHostCode") Long code) {
+ int result = taskRemoteHostService.deleteByCode(code, loginUser);
+ return result > 0 ? Result.success() : Result.error(DELETE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "queryTaskRemoteHostListPaging", description = "QUERY_TASK_REMOTE_HOST_PAGE_NOTES")
+ @Parameters({
+ @Parameter(name = "searchVal", description = "SEARCH_VAL", schema = @Schema(implementation = String.class)),
+ @Parameter(name = "pageSize", description = "PAGE_SIZE", required = true, schema = @Schema(implementation = int.class, example = "20")),
+ @Parameter(name = "pageNo", description = "PAGE_NO", required = true, schema = @Schema(implementation = int.class, example = "1"))
+ })
+ @GetMapping(value = "/list-paging")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(QUERY_TASK_REMOTE_HOST_PAGE_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result<PageInfo<TaskRemoteHostVO>> queryTaskRemoteHostListPaging(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestParam(value = "searchVal", required = false) String searchVal,
+ @RequestParam("pageSize") Integer pageSize,
+ @RequestParam("pageNo") Integer pageNo) {
+ PageInfo<TaskRemoteHostVO> taskRemoteHostVOPageInfo =
+ taskRemoteHostService.queryTaskRemoteHostListPaging(loginUser, searchVal, pageNo, pageSize);
+ return Result.success(taskRemoteHostVOPageInfo);
+ }
+
+ @Operation(summary = "queryTaskRemoteHostList", description = "QUERY_ALL_TASK_REMOTE_HOST_LIST_NOTES")
+ @GetMapping(value = "/query-remote-host-list")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(QUERY_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result<List<TaskRemoteHostVO>> queryTaskRemoteHostList(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) {
+ List<TaskRemoteHostVO> taskRemoteHostVOList = taskRemoteHostService.queryAllTaskRemoteHosts(loginUser);
+ return Result.success(taskRemoteHostVOList);
+ }
+
+ @Operation(summary = "textConnect", description = "TEXT_CONNECT_HOST_NOTES")
+ @PostMapping(value = "/test-connect")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(TEST_CONNECT_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result testConnect(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ boolean result = taskRemoteHostService.testConnect(taskRemoteHostDTO);
+ return result ? Result.success(SUCCESS) : Result.error(TEST_CONNECT_ERROR);
+ }
+
+ @Operation(summary = "verifyTaskRemoteHost", description = "VERIFY_TASK_REMOTE_HOST_NOTES")
+ @Parameters({
+ @Parameter(name = "taskRemoteHostName", description = "TASK_REMOTE_HOST_NAME", required = true, schema = @Schema(implementation = String.class))
+ })
+ @PostMapping(value = "/verify-host")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(VARIFY_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result verifyTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
Review Comment:
## Useless parameter
The parameter 'loginUser' is never used.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/2241)
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/TaskRemoteHostController.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.controller;
+
+import static org.apache.dolphinscheduler.api.enums.Status.CREATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.DELETE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_PAGE_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.SUCCESS;
+import static org.apache.dolphinscheduler.api.enums.Status.TEST_CONNECT_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.UPDATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.VARIFY_TASK_REMOTE_HOST_ERROR;
+
+import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation;
+import org.apache.dolphinscheduler.api.dto.TaskRemoteHostDTO;
+import org.apache.dolphinscheduler.api.exceptions.ApiException;
+import org.apache.dolphinscheduler.api.service.TaskRemoteHostService;
+import org.apache.dolphinscheduler.api.utils.PageInfo;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.api.vo.TaskRemoteHostVO;
+import org.apache.dolphinscheduler.common.constants.Constants;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestAttribute;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.ResponseStatus;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.Parameters;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+/**
+ * task remote host controller
+ */
+@Tag(name = "TASK_REMOTE_HOST_TAG")
+@RestController
+@RequestMapping("/remote_host")
+public class TaskRemoteHostController {
+
+ @Autowired
+ private TaskRemoteHostService taskRemoteHostService;
+
+ @Operation(summary = "createTaskRemoteHost", description = "CREATE_TASK_REMOTE_HOST_NOTES")
+ @PostMapping(value = "/create")
+ @ResponseStatus(HttpStatus.CREATED)
+ @ApiException(CREATE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result createTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ int result = taskRemoteHostService.createTaskRemoteHost(loginUser, taskRemoteHostDTO);
+ return result > 0 ? Result.success() : Result.error(CREATE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "updateTaskRemoteHost", description = "UPDATE_TASK_REMOTE_HOST_NOTES")
+ @PutMapping(value = "/update/{code}")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(UPDATE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result updateTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @PathVariable("code") Long code,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ int result = taskRemoteHostService.updateTaskRemoteHost(code, loginUser, taskRemoteHostDTO);
+ return result > 0 ? Result.success() : Result.error(UPDATE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "delete", description = "DELETE_TASK_REMOTE_HOST_NOTES")
+ @Parameters({
+ @Parameter(name = "code", description = "TASK_REMOTE_HOST_CODE", schema = @Schema(implementation = long.class, example = "123456", required = true))
+ })
+ @PostMapping(value = "/delete")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(DELETE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result deleteTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestParam("taskRemoteHostCode") Long code) {
+ int result = taskRemoteHostService.deleteByCode(code, loginUser);
+ return result > 0 ? Result.success() : Result.error(DELETE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "queryTaskRemoteHostListPaging", description = "QUERY_TASK_REMOTE_HOST_PAGE_NOTES")
+ @Parameters({
+ @Parameter(name = "searchVal", description = "SEARCH_VAL", schema = @Schema(implementation = String.class)),
+ @Parameter(name = "pageSize", description = "PAGE_SIZE", required = true, schema = @Schema(implementation = int.class, example = "20")),
+ @Parameter(name = "pageNo", description = "PAGE_NO", required = true, schema = @Schema(implementation = int.class, example = "1"))
+ })
+ @GetMapping(value = "/list-paging")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(QUERY_TASK_REMOTE_HOST_PAGE_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result<PageInfo<TaskRemoteHostVO>> queryTaskRemoteHostListPaging(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestParam(value = "searchVal", required = false) String searchVal,
+ @RequestParam("pageSize") Integer pageSize,
+ @RequestParam("pageNo") Integer pageNo) {
+ PageInfo<TaskRemoteHostVO> taskRemoteHostVOPageInfo =
+ taskRemoteHostService.queryTaskRemoteHostListPaging(loginUser, searchVal, pageNo, pageSize);
+ return Result.success(taskRemoteHostVOPageInfo);
+ }
+
+ @Operation(summary = "queryTaskRemoteHostList", description = "QUERY_ALL_TASK_REMOTE_HOST_LIST_NOTES")
+ @GetMapping(value = "/query-remote-host-list")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(QUERY_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result<List<TaskRemoteHostVO>> queryTaskRemoteHostList(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser) {
+ List<TaskRemoteHostVO> taskRemoteHostVOList = taskRemoteHostService.queryAllTaskRemoteHosts(loginUser);
+ return Result.success(taskRemoteHostVOList);
+ }
+
+ @Operation(summary = "textConnect", description = "TEXT_CONNECT_HOST_NOTES")
+ @PostMapping(value = "/test-connect")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(TEST_CONNECT_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result testConnect(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
Review Comment:
## Useless parameter
The parameter 'loginUser' is never used.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/2242)
##########
dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/mapper/TaskRemoteHostMapperTest.java:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.dao.mapper;
+
+import org.apache.dolphinscheduler.dao.BaseDaoTest;
+import org.apache.dolphinscheduler.dao.entity.TaskRemoteHost;
+
+import java.util.List;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.springframework.beans.factory.annotation.Autowired;
+
+import com.baomidou.mybatisplus.core.metadata.IPage;
+import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
+
+public class TaskRemoteHostMapperTest extends BaseDaoTest {
+
+ @Autowired
+ private TaskRemoteHostMapper taskRemoteHostMapper;
+
+ private TaskRemoteHost insertOne() {
+ TaskRemoteHost taskRemoteHost = new TaskRemoteHost();
+ taskRemoteHost.setCode(1L);
+ taskRemoteHost.setName("app01 server");
+ taskRemoteHost.setIp("localhost");
+ taskRemoteHost.setPort(22);
+ taskRemoteHost.setAccount("foo");
+ taskRemoteHost.setPassword("foo123");
+ taskRemoteHost.setDescription("app01 server description");
+ taskRemoteHost.setOperator(1);
+ taskRemoteHostMapper.insert(taskRemoteHost);
+ return taskRemoteHost;
+ }
+
+ @BeforeEach
+ public void setUp() {
+ clearTestData();
+ }
+
+ @AfterEach
+ public void after() {
+ clearTestData();
+ }
+
+ public void clearTestData() {
+ taskRemoteHostMapper.queryAllTaskRemoteHostList().forEach(taskRemoteHost -> {
+ taskRemoteHostMapper.deleteByCode(taskRemoteHost.getCode());
+ });
+ }
+
+ @Test
+ public void testUpdate() {
+ TaskRemoteHost taskRemoteHost = insertOne();
+ taskRemoteHost.setDescription("update description");
+ int update = taskRemoteHostMapper.updateById(taskRemoteHost);
+ Assertions.assertEquals(1, update);
+ }
+
+ @Test
+ public void testDelete() {
+ TaskRemoteHost taskRemoteHost = insertOne();
+ int delete = taskRemoteHostMapper.deleteById(taskRemoteHost);
+ Assertions.assertEquals(1, delete);
+ }
+
+ @Test
+ public void testQueryByTaskRemoteHostName() {
+ TaskRemoteHost taskRemoteHost = insertOne();
+ TaskRemoteHost result = taskRemoteHostMapper.queryByTaskRemoteHostName(taskRemoteHost.getName());
+ Assertions.assertEquals(taskRemoteHost.getName(), result.getName());
+ }
+
+ @Test
+ public void testQueryByTaskRemoteHostCode() {
+ TaskRemoteHost taskRemoteHost = insertOne();
+ TaskRemoteHost result = taskRemoteHostMapper.queryByTaskRemoteHostCode(taskRemoteHost.getCode());
+ Assertions.assertEquals(taskRemoteHost.getCode(), result.getCode());
+ }
+
+ @Test
+ public void testQueryTaskRemoteHostListPaging() {
+ TaskRemoteHost entity = insertOne();
Review Comment:
## Unread local variable
Variable 'TaskRemoteHost entity' is never read.
[Show more details](https://github.com/apache/dolphinscheduler/security/code-scanning/2243)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1313750534
> > SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
> > [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [72 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
> > [](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density
&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
>
> @DarkAssassinator please check the sonar result and fix it.
Sure, but these [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) are not really issue, just password word
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1312521545
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [72 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1021607979
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/TaskRemoteHostController.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.controller;
+
+import static org.apache.dolphinscheduler.api.enums.Status.CREATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.DELETE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_PAGE_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.SUCCESS;
+import static org.apache.dolphinscheduler.api.enums.Status.TEST_CONNECT_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.UPDATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.VARIFY_TASK_REMOTE_HOST_ERROR;
+
+import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation;
+import org.apache.dolphinscheduler.api.dto.TaskRemoteHostDTO;
+import org.apache.dolphinscheduler.api.exceptions.ApiException;
+import org.apache.dolphinscheduler.api.service.TaskRemoteHostService;
+import org.apache.dolphinscheduler.api.utils.PageInfo;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.api.vo.TaskRemoteHostVO;
+import org.apache.dolphinscheduler.common.constants.Constants;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestAttribute;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.ResponseStatus;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.Parameters;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+/**
+ * task remote host controller
+ */
+@Tag(name = "TASK_REMOTE_HOST_TAG")
+@RestController
+@RequestMapping("/remote_host")
+public class TaskRemoteHostController {
+
+ @Autowired
+ private TaskRemoteHostService taskRemoteHostService;
+
+ @Operation(summary = "createTaskRemoteHost", description = "CREATE_TASK_REMOTE_HOST_NOTES")
+ @PostMapping(value = "/create")
Review Comment:
done
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TaskRemoteHostServiceImpl.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.service.impl;
+
+import static org.apache.dolphinscheduler.api.constants.ApiFuncIdentificationConstant.TASK_REMOTE_HOST_CREATE;
+import static org.apache.dolphinscheduler.api.constants.ApiFuncIdentificationConstant.TASK_REMOTE_HOST_DELETE;
+import static org.apache.dolphinscheduler.api.constants.ApiFuncIdentificationConstant.TASK_REMOTE_HOST_EDIT;
+
+import org.apache.dolphinscheduler.api.dto.TaskRemoteHostDTO;
+import org.apache.dolphinscheduler.api.enums.Status;
+import org.apache.dolphinscheduler.api.exceptions.ServiceException;
+import org.apache.dolphinscheduler.api.service.TaskRemoteHostService;
+import org.apache.dolphinscheduler.api.utils.PageInfo;
+import org.apache.dolphinscheduler.api.vo.TaskRemoteHostVO;
+import org.apache.dolphinscheduler.common.enums.AuthorizationType;
+import org.apache.dolphinscheduler.common.enums.UserType;
+import org.apache.dolphinscheduler.common.utils.CodeGenerateUtils;
+import org.apache.dolphinscheduler.common.utils.CodeGenerateUtils.CodeGenerateException;
+import org.apache.dolphinscheduler.common.utils.NetUtils;
+import org.apache.dolphinscheduler.dao.entity.TaskInstance;
+import org.apache.dolphinscheduler.dao.entity.TaskRemoteHost;
+import org.apache.dolphinscheduler.dao.entity.User;
+import org.apache.dolphinscheduler.dao.mapper.TaskInstanceMapper;
+import org.apache.dolphinscheduler.dao.mapper.TaskRemoteHostMapper;
+import org.apache.dolphinscheduler.plugin.task.api.model.SSHSessionHost;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.DSSessionAbandonedConfig;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.DSSessionPoolConfig;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.SSHResponse;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.SSHSessionHolder;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.SSHSessionPool;
+import org.apache.dolphinscheduler.service.utils.Constants;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.BeanUtils;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+import org.springframework.transaction.annotation.Transactional;
+
+import com.baomidou.mybatisplus.core.metadata.IPage;
+import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
+
+@Service
+public class TaskRemoteHostServiceImpl extends BaseServiceImpl implements TaskRemoteHostService {
+
+ private static final Logger logger = LoggerFactory.getLogger(TaskRemoteHostServiceImpl.class);
+
+ @Autowired
+ private TaskRemoteHostMapper taskRemoteHostMapper;
+
+ @Autowired
+ private TaskInstanceMapper taskInstanceMapper;
+
+ @Override
+ @Transactional
+ public int createTaskRemoteHost(User loginUser, TaskRemoteHostDTO taskRemoteHostDTO) {
+ checkTaskRemoteHostDTO(taskRemoteHostDTO);
+
+ checkOperatorPermissions(loginUser, null, AuthorizationType.TASK_REMOTE_TASK, TASK_REMOTE_HOST_CREATE);
+
+ if (isExistSameName(taskRemoteHostDTO.getName())) {
+ throw new ServiceException(Status.TASK_REMOTE_HOST_EXIST, taskRemoteHostDTO.getName());
+ }
+
+ TaskRemoteHost remoteHost = new TaskRemoteHost();
+ BeanUtils.copyProperties(taskRemoteHostDTO, remoteHost);
+ long remoteHostCode;
+ try {
+ remoteHostCode = CodeGenerateUtils.getInstance().genCode();
+ } catch (CodeGenerateException e) {
+ throw new ServiceException(Status.INTERNAL_SERVER_ERROR_ARGS);
+ }
+ remoteHost.setCode(remoteHostCode);
+ remoteHost.setOperator(loginUser.getId());
+ remoteHost.setCreateTime(new Date());
+ remoteHost.setUpdateTime(new Date());
+
+ int result = taskRemoteHostMapper.insert(remoteHost);
+ if (result > 0) {
+ permissionPostHandle(AuthorizationType.TASK_REMOTE_TASK, loginUser.getId(),
+ Collections.singletonList(remoteHost.getId()), logger);
+ logger.info("Create remote host successes, host name {}.", remoteHost.getName());
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1313000172
> SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
>
> [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [72 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
>
> [](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&v
iew=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
@DarkAssassinator please check the sonar result and fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1021541060
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ssh/SSHSessionHolder.java:
##########
@@ -0,0 +1,348 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.api.ssh;
+
+import org.apache.dolphinscheduler.plugin.task.api.model.SSHSessionHost;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.jcraft.jsch.Channel;
+import com.jcraft.jsch.ChannelExec;
+import com.jcraft.jsch.ChannelSftp;
+import com.jcraft.jsch.JSch;
+import com.jcraft.jsch.JSchException;
+import com.jcraft.jsch.Session;
+import com.jcraft.jsch.SftpException;
+
+/**
+ * ACP pooled object: JSch Session Wrapper
+ */
+public class SSHSessionHolder {
+
+ private static final Logger logger = LoggerFactory.getLogger(SSHSessionHolder.class);
+
+ private static final String EXEC_TYPE = "exec";
+
+ private static final String SFTP_TYPE = "sftp";
+
+ private static final String SHELL_TYPE = "shell";
+
+ private static final int DEFAULT_CONNECT_TIMEOUT = 5000;
+
+ private static final int KB_UNIT = 1024;
+
+ public static final String SINGLE_SLASH = "/";
+
+ private Session session;
+
+ private final SSHSessionHost sessionHost;
+
+ private final String id;
+
+ private SftpConfig sftpConfig;
+
+ public SSHSessionHolder(SSHSessionHost sessionHost) {
+ this.sessionHost = sessionHost;
+ this.id = UUID.randomUUID().toString();
+ this.session = null;
+ }
+
+ public void connect() throws JSchException {
+ this.connect(DEFAULT_CONNECT_TIMEOUT);
+ }
+
+ public void connect(int timeoutMills) throws JSchException {
+ JSch jSch = new JSch();
+ this.session = jSch.getSession(sessionHost.getAccount(), sessionHost.getIp(), sessionHost.getPort());
+ this.session.setTimeout(timeoutMills);
+ this.session.setConfig("StrictHostKeyChecking", "no");
+ // other authorization methods can be considered in the future
+ this.session.setPassword(sessionHost.getPassword());
+ this.session.connect();
+ logger.info("Connected to ssh session: {}, session id: {}", sessionHost.toString(), id);
+ }
+
+ public void disconnect() {
+ if (session != null) {
+ session.disconnect();
+ }
+ }
+
+ public boolean isConnected() {
+ return session.isConnected();
+ }
+
+ public void keepAlive() throws Exception {
+ if (session != null) {
+ session.sendKeepAliveMsg();
+ }
+ }
+
+ public SSHResponse execCommand(String command) {
+ return this.execCommand(command, -1, logger);
+ }
+
+ public SSHResponse execCommand(String command, long timeout, Logger logger) {
+ return this.execCommand(createChannelExec(), command, timeout, logger);
+ }
+
+ public SSHResponse execCommand(ChannelExec channelExec, String command, long timeout, Logger customLogger) {
+ customLogger.info("Executing command {} on session:{}", command, this);
+ channelExec.setCommand(command);
+ channelExec.setInputStream(null);
+ channelExec.setErrStream(System.err);
+ // it will kill process when channel disconnect, default true
+ channelExec.setPty(true);
+
+ SSHResponse response = new SSHResponse();
+ try (
+ InputStream in = channelExec.getInputStream();
+ InputStream err = channelExec.getErrStream();) {
+ channelExec.connect(DEFAULT_CONNECT_TIMEOUT);
+
+ long timeoutSave = timeout > 0 ? timeout : Integer.MAX_VALUE;
+ List<String> out = new ArrayList<String>();
+ byte[] outSave = new byte[1024];
+ while (true) {
+ timeoutSave--;
+ while (in.available() > 0) {
+ int i = in.read(outSave, 0, 1024);
+ if (i < 0) {
+ break;
+ }
+ String line = new String(outSave, 0, i, StandardCharsets.UTF_8);
+ out.add(line);
+ customLogger.info(line);
+ }
+ while (err.available() > 0) {
+ int i = err.read(outSave, 0, 1024);
+ if (i < 0) {
+ break;
+ }
+ String line = new String(outSave, 0, i, StandardCharsets.UTF_8);
+ out.add(line);
+ customLogger.error(line);
+ }
+ if (channelExec.isClosed() || channelExec.isEOF()) {
+ response.setExitCode(channelExec.getExitStatus());
+ break;
+ }
+ TimeUnit.MILLISECONDS.sleep(1000);
+ if (timeoutSave < 0) {
+ customLogger.error("Exec command {} on session {} timed out", command, this);
+ throw new SSHException("Exec command " + command + " on session " + this + " timed out");
+ }
+ }
+ response.setOut(out);
+ logger.info("Exec command {} on session {} finished, exit code: {}", command, this, response.getExitCode());
+ return response;
+ } catch (Exception e) {
+ throw new SSHException("Exec command " + command + " on session " + this + " failed", e);
+ } finally {
+ close(channelExec);
+ }
+ }
+
+ public boolean sftpDir(String localDirPath, String remoteDirPath) {
+ return sftpDir(localDirPath, remoteDirPath, logger);
+ }
+
+ public boolean sftpDir(String localDirPath, String remoteDirPath, Logger customLogger) {
+ return sftpDir(createChannelSftp(), localDirPath, remoteDirPath, sftpConfig.isEnableUploadMonitor(),
+ sftpConfig.getMaxUploadRate(), sftpConfig.getMaxFileSize(), customLogger);
+ }
+
+ /**
+ * Sftp local directory to remote host,
+ * @param channelSftp SSH sftp channel
+ * @param localDirPath local directory path
+ * @param remoteDirPath remote target directory path
+ * @param enableUploadMonitor enable upload monitor thread
+ * @param maxUploadRate max upload rate, if negative, will not limit
+ * @param maxFileSize max file size, if negative, will not limit
+ * @param customLogger custom logger, default local
+ * @return sftp result
+ */
+ public boolean sftpDir(ChannelSftp channelSftp, String localDirPath, String remoteDirPath,
+ boolean enableUploadMonitor, int maxUploadRate, int maxFileSize, Logger customLogger) {
+ customLogger.info("Start to sftp local dir: {} to {}:{}", localDirPath, sessionHost.toString(), remoteDirPath);
+
+ File file = new File(localDirPath);
+ if (!file.exists()) {
+ customLogger.error("{} not exists.", localDirPath);
+ return false;
+ }
+
+ try {
+ channelSftp.connect(DEFAULT_CONNECT_TIMEOUT);
+ try {
+ channelSftp.cd(remoteDirPath);
+ } catch (SftpException e) {
+ if (!createDirOnRemote(remoteDirPath)) {
+ customLogger.error("Create directory:{} on remote:{} failed, so exit.", remoteDirPath,
+ sessionHost.toString());
+ return false;
+ }
+ }
+
+ long totalSize = file.length();
+
+ if (maxFileSize >= 0) {
+ if (totalSize > (long) maxFileSize * KB_UNIT * KB_UNIT) {
+ customLogger.error("The size of :{} has exceeded the maximum size:{}, size: {}", totalSize,
+ maxFileSize, localDirPath);
+ return false;
+ }
+ }
+
+ if (file.isDirectory()) {
+ File[] files = file.listFiles();
+ if (files == null || files.length == 0) {
+ customLogger.error("{} is a empty directory.", localDirPath);
+ return false;
+ }
+ for (File subFile : files) {
+ String sf = subFile.getCanonicalPath();
+ if (subFile.isDirectory()) {
+ String mkdirPath = remoteDirPath + "/" + subFile.getName();
+ try {
+ channelSftp.cd(mkdirPath);
+ } catch (SftpException e) {
+ if (!createDirOnRemote(mkdirPath)) {
+ customLogger.error("Could not create directory {} on remote session:{}", mkdirPath,
+ sessionHost.toString());
+ return false;
+ }
+ }
+ if (!sftpDir(createChannelSftp(), sf, mkdirPath, enableUploadMonitor, maxUploadRate, -1,
+ customLogger)) {
+ customLogger.error("sftp {} to {}:{} failed.", sf, sessionHost.toString(), mkdirPath);
+ return false;
+ }
+ } else {
+ upload(channelSftp, sf, remoteDirPath, enableUploadMonitor, maxUploadRate);
+ }
+ }
+ } else {
+ upload(channelSftp, file.getCanonicalPath(), remoteDirPath, enableUploadMonitor, maxUploadRate);
+ }
+ return true;
+ } catch (Exception e) {
+ throw new SSHException(
+ "sftp " + localDirPath + " to " + sessionHost.toString() + ":" + remoteDirPath + " failed.", e);
+ }
+ }
+
+ /**
+ * Create a directory on the remote server
+ * Because JSch does not support one-time creation of multi-layer directories, so just use `mkdir -p` instead
+ * @param remoteDirPath remote directory path
+ * @return result
+ */
+ public boolean createDirOnRemote(String remoteDirPath) {
+ logger.info("create directory:{} on remote:{}", remoteDirPath, sessionHost.toString());
+ SSHResponse response = execCommand("mkdir -p " + remoteDirPath);
+ return response.getExitCode() == 0;
+ }
+
+ public void clearPath(String path) {
+ if (SINGLE_SLASH.equals(path)) {
+ return;
+ }
+ execCommand("rm -rf " + path);
Review Comment:
> Please don't mark it as resolved if it's not addressed. cc @zhongjiajie @caishunfeng
Sorry, sure, and i think that `File.delete` cannot delete the remote dir and files, so i think that this case has resolved. WDYT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1319616485
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1305522735
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [71 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [24.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1312522883
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [72 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1304765019
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/12736?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#12736](https://codecov.io/gh/apache/dolphinscheduler/pull/12736?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f1ac7d5) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/913525343e98541b46cbc048058c4b2a67400804?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9135253) will **decrease** coverage by `0.22%`.
> The diff coverage is `22.24%`.
```diff
@@ Coverage Diff @@
## dev #12736 +/- ##
============================================
- Coverage 39.15% 38.92% -0.23%
- Complexity 4207 4244 +37
============================================
Files 1045 1056 +11
Lines 39704 40261 +557
Branches 4571 4643 +72
============================================
+ Hits 15545 15672 +127
- Misses 22392 22803 +411
- Partials 1767 1786 +19
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/12736?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...r/api/constants/ApiFuncIdentificationConstant.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnN0YW50cy9BcGlGdW5jSWRlbnRpZmljYXRpb25Db25zdGFudC5qYXZh) | `88.23% <ø> (ø)` | |
| [...lphinscheduler/common/enums/AuthorizationType.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL2VudW1zL0F1dGhvcml6YXRpb25UeXBlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...apache/dolphinscheduler/common/utils/NetUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3V0aWxzL05ldFV0aWxzLmphdmE=) | `48.61% <0.00%> (-1.04%)` | :arrow_down: |
| [...he/dolphinscheduler/dao/entity/TaskDefinition.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYW8vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGFvL2VudGl0eS9UYXNrRGVmaW5pdGlvbi5qYXZh) | `31.57% <ø> (ø)` | |
| [...dolphinscheduler/dao/entity/TaskDefinitionLog.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYW8vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGFvL2VudGl0eS9UYXNrRGVmaW5pdGlvbkxvZy5qYXZh) | `19.04% <0.00%> (-0.47%)` | :arrow_down: |
| [...ache/dolphinscheduler/dao/entity/TaskInstance.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYW8vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGFvL2VudGl0eS9UYXNrSW5zdGFuY2UuamF2YQ==) | `45.45% <ø> (ø)` | |
| [...server/master/runner/MasterSchedulerBootstrap.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvTWFzdGVyU2NoZWR1bGVyQm9vdHN0cmFwLmphdmE=) | `0.00% <ø> (ø)` | |
| [...erver/master/runner/StreamTaskExecuteRunnable.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvU3RyZWFtVGFza0V4ZWN1dGVSdW5uYWJsZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...duler/plugin/task/api/AbstractCommandExecutor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9BYnN0cmFjdENvbW1hbmRFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...dolphinscheduler/plugin/task/api/ProcessUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9Qcm9jZXNzVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| ... and [26 more](https://codecov.io/gh/apache/dolphinscheduler/pull/12736/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1315200637
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1017944320
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -2007,3 +2010,25 @@ CREATE TABLE `t_ds_fav_task`
) ENGINE = InnoDB
AUTO_INCREMENT = 1
DEFAULT CHARSET = utf8;
+
+-- ----------------------------
+-- Table structure for t_ds_task_remote_host
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_task_remote_host`;
+CREATE TABLE `t_ds_task_remote_host`
Review Comment:
I do not think adding a table for a task type is a good idea, can we ship it to our data source center? I know that we will change its name from `datasource center` to `connection center`, which can have various types of connection instead of database only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1319614042
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1021028685
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/TaskRemoteHostController.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.controller;
+
+import static org.apache.dolphinscheduler.api.enums.Status.CREATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.DELETE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_PAGE_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.SUCCESS;
+import static org.apache.dolphinscheduler.api.enums.Status.TEST_CONNECT_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.UPDATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.VARIFY_TASK_REMOTE_HOST_ERROR;
+
+import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation;
+import org.apache.dolphinscheduler.api.dto.TaskRemoteHostDTO;
+import org.apache.dolphinscheduler.api.exceptions.ApiException;
+import org.apache.dolphinscheduler.api.service.TaskRemoteHostService;
+import org.apache.dolphinscheduler.api.utils.PageInfo;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.api.vo.TaskRemoteHostVO;
+import org.apache.dolphinscheduler.common.constants.Constants;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestAttribute;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.ResponseStatus;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.Parameters;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+/**
+ * task remote host controller
+ */
+@Tag(name = "TASK_REMOTE_HOST_TAG")
+@RestController
+@RequestMapping("/remote_host")
+public class TaskRemoteHostController {
+
+ @Autowired
+ private TaskRemoteHostService taskRemoteHostService;
+
+ @Operation(summary = "createTaskRemoteHost", description = "CREATE_TASK_REMOTE_HOST_NOTES")
+ @PostMapping(value = "/create")
Review Comment:
maybe it should remove the value dut to default PostMapping means `create`? cc @zhongjiajie
```suggestion
@PostMapping()
```
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/TaskRemoteHostController.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.controller;
+
+import static org.apache.dolphinscheduler.api.enums.Status.CREATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.DELETE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_PAGE_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.SUCCESS;
+import static org.apache.dolphinscheduler.api.enums.Status.TEST_CONNECT_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.UPDATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.VARIFY_TASK_REMOTE_HOST_ERROR;
+
+import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation;
+import org.apache.dolphinscheduler.api.dto.TaskRemoteHostDTO;
+import org.apache.dolphinscheduler.api.exceptions.ApiException;
+import org.apache.dolphinscheduler.api.service.TaskRemoteHostService;
+import org.apache.dolphinscheduler.api.utils.PageInfo;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.api.vo.TaskRemoteHostVO;
+import org.apache.dolphinscheduler.common.constants.Constants;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestAttribute;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.ResponseStatus;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.Parameters;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+/**
+ * task remote host controller
+ */
+@Tag(name = "TASK_REMOTE_HOST_TAG")
+@RestController
+@RequestMapping("/remote_host")
+public class TaskRemoteHostController {
+
+ @Autowired
+ private TaskRemoteHostService taskRemoteHostService;
+
+ @Operation(summary = "createTaskRemoteHost", description = "CREATE_TASK_REMOTE_HOST_NOTES")
+ @PostMapping(value = "/create")
+ @ResponseStatus(HttpStatus.CREATED)
+ @ApiException(CREATE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result createTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ int result = taskRemoteHostService.createTaskRemoteHost(loginUser, taskRemoteHostDTO);
+ return result > 0 ? Result.success() : Result.error(CREATE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "updateTaskRemoteHost", description = "UPDATE_TASK_REMOTE_HOST_NOTES")
+ @PutMapping(value = "/update/{code}")
+ @ResponseStatus(HttpStatus.OK)
+ @ApiException(UPDATE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result updateTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @PathVariable("code") Long code,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ int result = taskRemoteHostService.updateTaskRemoteHost(code, loginUser, taskRemoteHostDTO);
+ return result > 0 ? Result.success() : Result.error(UPDATE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "delete", description = "DELETE_TASK_REMOTE_HOST_NOTES")
+ @Parameters({
+ @Parameter(name = "code", description = "TASK_REMOTE_HOST_CODE", schema = @Schema(implementation = long.class, example = "123456", required = true))
+ })
+ @PostMapping(value = "/delete")
Review Comment:
```suggestion
@DeleteMapping(value = "/delete")
```
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java:
##########
@@ -190,11 +220,105 @@ public TaskResponse run(String execCommand) throws IOException, InterruptedExcep
return result;
}
+ if (bashTaskExecutionContext.getSessionHost() != null) {
+ runningOnSSH = true;
+ }
+
+ if (runningOnSSH && SystemUtils.IS_OS_WINDOWS) {
+ logger.error("SSH does not support Windows systems");
+ TaskExecutionContextCacheManager.removeByTaskInstanceId(taskInstanceId);
+ return result;
+ }
+
String commandFilePath = buildCommandFilePath();
// create command file if not exists
createCommandFileIfNotExists(execCommand, commandFilePath);
+ result = runCommandFile(commandFilePath);
+
+ return result;
+ }
+
+ private TaskResponse runCommandFile(String commandFilePath) throws IOException, InterruptedException {
+ return runningOnSSH ? runOnSSH(commandFilePath) : runOnLocalProcess(commandFilePath);
+ }
+
+ private TaskResponse runOnSSH(String commandFilePath) {
+ TaskResponse result = new TaskResponse();
+ SSHSessionHolder sessionHolder = null;
+ this.sessionHost = bashTaskExecutionContext.getSessionHost();
+ try {
+ sessionHolder = SSHSessionPool.getSessionHolder(sessionHost);
+ sessionHolder.setSftpConfig(SSHSessionPool.getSftpConfig());
+ logger.info("borrow session:{} success", sessionHolder);
+
+ boolean uploadRes =
+ sessionHolder.sftpDir(taskRequest.getExecutePath(), taskRequest.getExecutePath(), logger);
+ if (!uploadRes) {
+ logger.error("upload task {} execute path to remote session {} failed", taskRequest.getExecutePath(),
+ sessionHost.toString());
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ return result;
+ }
+
+ if (taskRequest.getEnvFile() != null) {
+ boolean uploadEnvRes =
+ sessionHolder.sftpDir(taskRequest.getEnvFile(), taskRequest.getExecutePath(), logger);
+ if (!uploadEnvRes) {
+ logger.error("upload task {} execute path to remote session {} failed", taskRequest.getEnvFile(),
+ sessionHost.toString());
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ return result;
+ }
+ }
+
+ // because JSch .chmod is not stable, so use the 'chmod -R' instead
+ logger.info("update remote path's permission:{} on session:{} to 755", taskRequest.getExecutePath(),
+ sessionHost.toString());
+ String chmodCommand = "chmod -R 755 " + taskRequest.getExecutePath();
Review Comment:
Why it should chmod 755? I'm not sure whether it will cause some CVE? cc @ruanwenjun
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql:
##########
@@ -2032,3 +2032,24 @@ CREATE TABLE t_ds_fav_task
user_id int NOT NULL,
PRIMARY KEY (id)
);
+
+--
+-- Table structure for t_ds_task_remote_host
+--
+
+DROP TABLE IF EXISTS t_ds_task_remote_host CASCADE;
+CREATE TABLE t_ds_task_remote_host
+(
+ id int NOT NULL AUTO_INCREMENT,
+ code bigint(20) NOT NULL ,
+ name varchar(100) NOT NULL ,
+ ip varchar(100) NOT NULL ,
+ port int NOT NULL ,
+ account varchar(100) NOT NULL ,
+ password varchar(64) NOT NULL ,
+ operator int DEFAULT NULL ,
+ description text,
+ create_time timestamp DEFAULT NULL ,
+ update_time timestamp DEFAULT NULL ,
+ PRIMARY KEY (id)
Review Comment:
format
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ssh/PooledSSHSessionFactory.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.api.ssh;
+
+import org.apache.dolphinscheduler.plugin.task.api.model.SSHSessionHost;
+
+import org.apache.commons.pool2.BaseKeyedPooledObjectFactory;
+import org.apache.commons.pool2.DestroyMode;
+import org.apache.commons.pool2.PooledObject;
+import org.apache.commons.pool2.impl.DefaultPooledObject;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * ACP SSH Session factory
+ */
+public class PooledSSHSessionFactory extends BaseKeyedPooledObjectFactory<SSHSessionHost, SSHSessionHolder> {
+
+ private static final Logger logger = LoggerFactory.getLogger(PooledSSHSessionFactory.class);
+
+ @Override
+ public SSHSessionHolder create(SSHSessionHost sshSessionHost) throws Exception {
+ SSHSessionHolder pooledObject = new SSHSessionHolder(sshSessionHost);
+ pooledObject.connect();
+ return pooledObject;
+ }
+
+ @Override
+ public PooledObject<SSHSessionHolder> wrap(SSHSessionHolder sshSessionHolder) {
+ return new DefaultPooledObject<>(sshSessionHolder);
+ }
+
+ @Override
+ public void destroyObject(SSHSessionHost key, PooledObject<SSHSessionHolder> p,
+ DestroyMode destroyMode) throws Exception {
+ logger.info("destroy session {}", p.getObject().toString());
+ p.getObject().disconnect();
+ }
+
+ @Override
+ public boolean validateObject(SSHSessionHost key, PooledObject<SSHSessionHolder> p) {
+ if (p.getObject().isConnected()) {
+ try {
+ p.getObject().keepAlive();
+ return true;
+ } catch (Exception e) {
+ logger.error("Cannot send alive msg to session of {}", p.getObject().toString(), e);
Review Comment:
```suggestion
logger.error("Cannot send alive msg to session of {}", p.getObject(), e);
```
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/TaskRemoteHostController.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.controller;
+
+import static org.apache.dolphinscheduler.api.enums.Status.CREATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.DELETE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.QUERY_TASK_REMOTE_HOST_PAGE_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.SUCCESS;
+import static org.apache.dolphinscheduler.api.enums.Status.TEST_CONNECT_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.UPDATE_TASK_REMOTE_HOST_ERROR;
+import static org.apache.dolphinscheduler.api.enums.Status.VARIFY_TASK_REMOTE_HOST_ERROR;
+
+import org.apache.dolphinscheduler.api.aspect.AccessLogAnnotation;
+import org.apache.dolphinscheduler.api.dto.TaskRemoteHostDTO;
+import org.apache.dolphinscheduler.api.exceptions.ApiException;
+import org.apache.dolphinscheduler.api.service.TaskRemoteHostService;
+import org.apache.dolphinscheduler.api.utils.PageInfo;
+import org.apache.dolphinscheduler.api.utils.Result;
+import org.apache.dolphinscheduler.api.vo.TaskRemoteHostVO;
+import org.apache.dolphinscheduler.common.constants.Constants;
+import org.apache.dolphinscheduler.dao.entity.User;
+
+import java.util.List;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestAttribute;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.ResponseStatus;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.Parameters;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+/**
+ * task remote host controller
+ */
+@Tag(name = "TASK_REMOTE_HOST_TAG")
+@RestController
+@RequestMapping("/remote_host")
+public class TaskRemoteHostController {
+
+ @Autowired
+ private TaskRemoteHostService taskRemoteHostService;
+
+ @Operation(summary = "createTaskRemoteHost", description = "CREATE_TASK_REMOTE_HOST_NOTES")
+ @PostMapping(value = "/create")
+ @ResponseStatus(HttpStatus.CREATED)
+ @ApiException(CREATE_TASK_REMOTE_HOST_ERROR)
+ @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
+ public Result createTaskRemoteHost(@Parameter(hidden = true) @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
+ @RequestBody TaskRemoteHostDTO taskRemoteHostDTO) {
+ int result = taskRemoteHostService.createTaskRemoteHost(loginUser, taskRemoteHostDTO);
+ return result > 0 ? Result.success() : Result.error(CREATE_TASK_REMOTE_HOST_ERROR);
+ }
+
+ @Operation(summary = "updateTaskRemoteHost", description = "UPDATE_TASK_REMOTE_HOST_NOTES")
+ @PutMapping(value = "/update/{code}")
Review Comment:
same here, the PutMapping is update action defaultly.
```suggestion
@PutMapping(value = "/{code}")
```
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TaskRemoteHostServiceImpl.java:
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.api.service.impl;
+
+import static org.apache.dolphinscheduler.api.constants.ApiFuncIdentificationConstant.TASK_REMOTE_HOST_CREATE;
+import static org.apache.dolphinscheduler.api.constants.ApiFuncIdentificationConstant.TASK_REMOTE_HOST_DELETE;
+import static org.apache.dolphinscheduler.api.constants.ApiFuncIdentificationConstant.TASK_REMOTE_HOST_EDIT;
+
+import org.apache.dolphinscheduler.api.dto.TaskRemoteHostDTO;
+import org.apache.dolphinscheduler.api.enums.Status;
+import org.apache.dolphinscheduler.api.exceptions.ServiceException;
+import org.apache.dolphinscheduler.api.service.TaskRemoteHostService;
+import org.apache.dolphinscheduler.api.utils.PageInfo;
+import org.apache.dolphinscheduler.api.vo.TaskRemoteHostVO;
+import org.apache.dolphinscheduler.common.enums.AuthorizationType;
+import org.apache.dolphinscheduler.common.enums.UserType;
+import org.apache.dolphinscheduler.common.utils.CodeGenerateUtils;
+import org.apache.dolphinscheduler.common.utils.CodeGenerateUtils.CodeGenerateException;
+import org.apache.dolphinscheduler.common.utils.NetUtils;
+import org.apache.dolphinscheduler.dao.entity.TaskInstance;
+import org.apache.dolphinscheduler.dao.entity.TaskRemoteHost;
+import org.apache.dolphinscheduler.dao.entity.User;
+import org.apache.dolphinscheduler.dao.mapper.TaskInstanceMapper;
+import org.apache.dolphinscheduler.dao.mapper.TaskRemoteHostMapper;
+import org.apache.dolphinscheduler.plugin.task.api.model.SSHSessionHost;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.DSSessionAbandonedConfig;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.DSSessionPoolConfig;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.SSHResponse;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.SSHSessionHolder;
+import org.apache.dolphinscheduler.plugin.task.api.ssh.SSHSessionPool;
+import org.apache.dolphinscheduler.service.utils.Constants;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.BeanUtils;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+import org.springframework.transaction.annotation.Transactional;
+
+import com.baomidou.mybatisplus.core.metadata.IPage;
+import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
+
+@Service
+public class TaskRemoteHostServiceImpl extends BaseServiceImpl implements TaskRemoteHostService {
+
+ private static final Logger logger = LoggerFactory.getLogger(TaskRemoteHostServiceImpl.class);
+
+ @Autowired
+ private TaskRemoteHostMapper taskRemoteHostMapper;
+
+ @Autowired
+ private TaskInstanceMapper taskInstanceMapper;
+
+ @Override
+ @Transactional
+ public int createTaskRemoteHost(User loginUser, TaskRemoteHostDTO taskRemoteHostDTO) {
+ checkTaskRemoteHostDTO(taskRemoteHostDTO);
+
+ checkOperatorPermissions(loginUser, null, AuthorizationType.TASK_REMOTE_TASK, TASK_REMOTE_HOST_CREATE);
+
+ if (isExistSameName(taskRemoteHostDTO.getName())) {
+ throw new ServiceException(Status.TASK_REMOTE_HOST_EXIST, taskRemoteHostDTO.getName());
+ }
+
+ TaskRemoteHost remoteHost = new TaskRemoteHost();
+ BeanUtils.copyProperties(taskRemoteHostDTO, remoteHost);
+ long remoteHostCode;
+ try {
+ remoteHostCode = CodeGenerateUtils.getInstance().genCode();
+ } catch (CodeGenerateException e) {
+ throw new ServiceException(Status.INTERNAL_SERVER_ERROR_ARGS);
+ }
+ remoteHost.setCode(remoteHostCode);
+ remoteHost.setOperator(loginUser.getId());
+ remoteHost.setCreateTime(new Date());
+ remoteHost.setUpdateTime(new Date());
+
+ int result = taskRemoteHostMapper.insert(remoteHost);
+ if (result > 0) {
+ permissionPostHandle(AuthorizationType.TASK_REMOTE_TASK, loginUser.getId(),
+ Collections.singletonList(remoteHost.getId()), logger);
+ logger.info("Create remote host successes, host name {}.", remoteHost.getName());
+ }
Review Comment:
In this case, I think checking result > 0 is meanless, due to not multi db write operations after inserting. Defaultly, it will success if no exception. WDYT?
```suggestion
taskRemoteHostMapper.insert(remoteHost);
permissionPostHandle(AuthorizationType.TASK_REMOTE_TASK, loginUser.getId(),
Collections.singletonList(remoteHost.getId()), logger);
logger.info("Create remote host successes, host name {}.", remoteHost.getName());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1321898297
> I have one question that in #12552, DS supports to define file parameter. It seems like you don't consider this scenario, output files might be created under remote host's execution directory, however, you only fetch stdout and neglect output files.
this is a new add scenario, i will check it. and please help to review other codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1327502455
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1681426570
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1304766533
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [71 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [24.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1313733556
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1021608439
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ssh/PooledSSHSessionFactory.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.api.ssh;
+
+import org.apache.dolphinscheduler.plugin.task.api.model.SSHSessionHost;
+
+import org.apache.commons.pool2.BaseKeyedPooledObjectFactory;
+import org.apache.commons.pool2.DestroyMode;
+import org.apache.commons.pool2.PooledObject;
+import org.apache.commons.pool2.impl.DefaultPooledObject;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * ACP SSH Session factory
+ */
+public class PooledSSHSessionFactory extends BaseKeyedPooledObjectFactory<SSHSessionHost, SSHSessionHolder> {
+
+ private static final Logger logger = LoggerFactory.getLogger(PooledSSHSessionFactory.class);
+
+ @Override
+ public SSHSessionHolder create(SSHSessionHost sshSessionHost) throws Exception {
+ SSHSessionHolder pooledObject = new SSHSessionHolder(sshSessionHost);
+ pooledObject.connect();
+ return pooledObject;
+ }
+
+ @Override
+ public PooledObject<SSHSessionHolder> wrap(SSHSessionHolder sshSessionHolder) {
+ return new DefaultPooledObject<>(sshSessionHolder);
+ }
+
+ @Override
+ public void destroyObject(SSHSessionHost key, PooledObject<SSHSessionHolder> p,
+ DestroyMode destroyMode) throws Exception {
+ logger.info("destroy session {}", p.getObject().toString());
+ p.getObject().disconnect();
+ }
+
+ @Override
+ public boolean validateObject(SSHSessionHost key, PooledObject<SSHSessionHolder> p) {
+ if (p.getObject().isConnected()) {
+ try {
+ p.getObject().keepAlive();
+ return true;
+ } catch (Exception e) {
+ logger.error("Cannot send alive msg to session of {}", p.getObject().toString(), e);
Review Comment:
done
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_h2.sql:
##########
@@ -2032,3 +2032,24 @@ CREATE TABLE t_ds_fav_task
user_id int NOT NULL,
PRIMARY KEY (id)
);
+
+--
+-- Table structure for t_ds_task_remote_host
+--
+
+DROP TABLE IF EXISTS t_ds_task_remote_host CASCADE;
+CREATE TABLE t_ds_task_remote_host
+(
+ id int NOT NULL AUTO_INCREMENT,
+ code bigint(20) NOT NULL ,
+ name varchar(100) NOT NULL ,
+ ip varchar(100) NOT NULL ,
+ port int NOT NULL ,
+ account varchar(100) NOT NULL ,
+ password varchar(64) NOT NULL ,
+ operator int DEFAULT NULL ,
+ description text,
+ create_time timestamp DEFAULT NULL ,
+ update_time timestamp DEFAULT NULL ,
+ PRIMARY KEY (id)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1021040610
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ssh/SSHSessionHolder.java:
##########
@@ -0,0 +1,348 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.api.ssh;
+
+import org.apache.dolphinscheduler.plugin.task.api.model.SSHSessionHost;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.jcraft.jsch.Channel;
+import com.jcraft.jsch.ChannelExec;
+import com.jcraft.jsch.ChannelSftp;
+import com.jcraft.jsch.JSch;
+import com.jcraft.jsch.JSchException;
+import com.jcraft.jsch.Session;
+import com.jcraft.jsch.SftpException;
+
+/**
+ * ACP pooled object: JSch Session Wrapper
+ */
+public class SSHSessionHolder {
+
+ private static final Logger logger = LoggerFactory.getLogger(SSHSessionHolder.class);
+
+ private static final String EXEC_TYPE = "exec";
+
+ private static final String SFTP_TYPE = "sftp";
+
+ private static final String SHELL_TYPE = "shell";
+
+ private static final int DEFAULT_CONNECT_TIMEOUT = 5000;
+
+ private static final int KB_UNIT = 1024;
+
+ public static final String SINGLE_SLASH = "/";
+
+ private Session session;
+
+ private final SSHSessionHost sessionHost;
+
+ private final String id;
+
+ private SftpConfig sftpConfig;
+
+ public SSHSessionHolder(SSHSessionHost sessionHost) {
+ this.sessionHost = sessionHost;
+ this.id = UUID.randomUUID().toString();
+ this.session = null;
+ }
+
+ public void connect() throws JSchException {
+ this.connect(DEFAULT_CONNECT_TIMEOUT);
+ }
+
+ public void connect(int timeoutMills) throws JSchException {
+ JSch jSch = new JSch();
+ this.session = jSch.getSession(sessionHost.getAccount(), sessionHost.getIp(), sessionHost.getPort());
+ this.session.setTimeout(timeoutMills);
+ this.session.setConfig("StrictHostKeyChecking", "no");
+ // other authorization methods can be considered in the future
+ this.session.setPassword(sessionHost.getPassword());
+ this.session.connect();
+ logger.info("Connected to ssh session: {}, session id: {}", sessionHost.toString(), id);
+ }
+
+ public void disconnect() {
+ if (session != null) {
+ session.disconnect();
+ }
+ }
+
+ public boolean isConnected() {
+ return session.isConnected();
+ }
+
+ public void keepAlive() throws Exception {
+ if (session != null) {
+ session.sendKeepAliveMsg();
+ }
+ }
+
+ public SSHResponse execCommand(String command) {
+ return this.execCommand(command, -1, logger);
+ }
+
+ public SSHResponse execCommand(String command, long timeout, Logger logger) {
+ return this.execCommand(createChannelExec(), command, timeout, logger);
+ }
+
+ public SSHResponse execCommand(ChannelExec channelExec, String command, long timeout, Logger customLogger) {
+ customLogger.info("Executing command {} on session:{}", command, this);
+ channelExec.setCommand(command);
+ channelExec.setInputStream(null);
+ channelExec.setErrStream(System.err);
+ // it will kill process when channel disconnect, default true
+ channelExec.setPty(true);
+
+ SSHResponse response = new SSHResponse();
+ try (
+ InputStream in = channelExec.getInputStream();
+ InputStream err = channelExec.getErrStream();) {
+ channelExec.connect(DEFAULT_CONNECT_TIMEOUT);
+
+ long timeoutSave = timeout > 0 ? timeout : Integer.MAX_VALUE;
+ List<String> out = new ArrayList<String>();
+ byte[] outSave = new byte[1024];
+ while (true) {
+ timeoutSave--;
+ while (in.available() > 0) {
+ int i = in.read(outSave, 0, 1024);
+ if (i < 0) {
+ break;
+ }
+ String line = new String(outSave, 0, i, StandardCharsets.UTF_8);
+ out.add(line);
+ customLogger.info(line);
+ }
+ while (err.available() > 0) {
+ int i = err.read(outSave, 0, 1024);
+ if (i < 0) {
+ break;
+ }
+ String line = new String(outSave, 0, i, StandardCharsets.UTF_8);
+ out.add(line);
+ customLogger.error(line);
+ }
+ if (channelExec.isClosed() || channelExec.isEOF()) {
+ response.setExitCode(channelExec.getExitStatus());
+ break;
+ }
+ TimeUnit.MILLISECONDS.sleep(1000);
+ if (timeoutSave < 0) {
+ customLogger.error("Exec command {} on session {} timed out", command, this);
+ throw new SSHException("Exec command " + command + " on session " + this + " timed out");
+ }
+ }
+ response.setOut(out);
+ logger.info("Exec command {} on session {} finished, exit code: {}", command, this, response.getExitCode());
+ return response;
+ } catch (Exception e) {
+ throw new SSHException("Exec command " + command + " on session " + this + " failed", e);
+ } finally {
+ close(channelExec);
+ }
+ }
+
+ public boolean sftpDir(String localDirPath, String remoteDirPath) {
+ return sftpDir(localDirPath, remoteDirPath, logger);
+ }
+
+ public boolean sftpDir(String localDirPath, String remoteDirPath, Logger customLogger) {
+ return sftpDir(createChannelSftp(), localDirPath, remoteDirPath, sftpConfig.isEnableUploadMonitor(),
+ sftpConfig.getMaxUploadRate(), sftpConfig.getMaxFileSize(), customLogger);
+ }
+
+ /**
+ * Sftp local directory to remote host,
+ * @param channelSftp SSH sftp channel
+ * @param localDirPath local directory path
+ * @param remoteDirPath remote target directory path
+ * @param enableUploadMonitor enable upload monitor thread
+ * @param maxUploadRate max upload rate, if negative, will not limit
+ * @param maxFileSize max file size, if negative, will not limit
+ * @param customLogger custom logger, default local
+ * @return sftp result
+ */
+ public boolean sftpDir(ChannelSftp channelSftp, String localDirPath, String remoteDirPath,
+ boolean enableUploadMonitor, int maxUploadRate, int maxFileSize, Logger customLogger) {
+ customLogger.info("Start to sftp local dir: {} to {}:{}", localDirPath, sessionHost.toString(), remoteDirPath);
+
+ File file = new File(localDirPath);
+ if (!file.exists()) {
+ customLogger.error("{} not exists.", localDirPath);
+ return false;
+ }
+
+ try {
+ channelSftp.connect(DEFAULT_CONNECT_TIMEOUT);
+ try {
+ channelSftp.cd(remoteDirPath);
+ } catch (SftpException e) {
+ if (!createDirOnRemote(remoteDirPath)) {
+ customLogger.error("Create directory:{} on remote:{} failed, so exit.", remoteDirPath,
+ sessionHost.toString());
+ return false;
+ }
+ }
+
+ long totalSize = file.length();
+
+ if (maxFileSize >= 0) {
+ if (totalSize > (long) maxFileSize * KB_UNIT * KB_UNIT) {
+ customLogger.error("The size of :{} has exceeded the maximum size:{}, size: {}", totalSize,
+ maxFileSize, localDirPath);
+ return false;
+ }
+ }
+
+ if (file.isDirectory()) {
+ File[] files = file.listFiles();
+ if (files == null || files.length == 0) {
+ customLogger.error("{} is a empty directory.", localDirPath);
+ return false;
+ }
+ for (File subFile : files) {
+ String sf = subFile.getCanonicalPath();
+ if (subFile.isDirectory()) {
+ String mkdirPath = remoteDirPath + "/" + subFile.getName();
+ try {
+ channelSftp.cd(mkdirPath);
+ } catch (SftpException e) {
+ if (!createDirOnRemote(mkdirPath)) {
+ customLogger.error("Could not create directory {} on remote session:{}", mkdirPath,
+ sessionHost.toString());
+ return false;
+ }
+ }
+ if (!sftpDir(createChannelSftp(), sf, mkdirPath, enableUploadMonitor, maxUploadRate, -1,
+ customLogger)) {
+ customLogger.error("sftp {} to {}:{} failed.", sf, sessionHost.toString(), mkdirPath);
+ return false;
+ }
+ } else {
+ upload(channelSftp, sf, remoteDirPath, enableUploadMonitor, maxUploadRate);
+ }
+ }
+ } else {
+ upload(channelSftp, file.getCanonicalPath(), remoteDirPath, enableUploadMonitor, maxUploadRate);
+ }
+ return true;
+ } catch (Exception e) {
+ throw new SSHException(
+ "sftp " + localDirPath + " to " + sessionHost.toString() + ":" + remoteDirPath + " failed.", e);
+ }
+ }
+
+ /**
+ * Create a directory on the remote server
+ * Because JSch does not support one-time creation of multi-layer directories, so just use `mkdir -p` instead
+ * @param remoteDirPath remote directory path
+ * @return result
+ */
+ public boolean createDirOnRemote(String remoteDirPath) {
+ logger.info("create directory:{} on remote:{}", remoteDirPath, sessionHost.toString());
+ SSHResponse response = execCommand("mkdir -p " + remoteDirPath);
+ return response.getExitCode() == 0;
+ }
+
+ public void clearPath(String path) {
+ if (SINGLE_SLASH.equals(path)) {
+ return;
+ }
+ execCommand("rm -rf " + path);
Review Comment:
Please don't mark it as resolved if it's not addressed. cc @zhongjiajie @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1022236048
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java:
##########
@@ -190,11 +220,105 @@ public TaskResponse run(String execCommand) throws IOException, InterruptedExcep
return result;
}
+ if (bashTaskExecutionContext.getSessionHost() != null) {
+ runningOnSSH = true;
+ }
+
+ if (runningOnSSH && SystemUtils.IS_OS_WINDOWS) {
+ logger.error("SSH does not support Windows systems");
+ TaskExecutionContextCacheManager.removeByTaskInstanceId(taskInstanceId);
+ return result;
+ }
+
String commandFilePath = buildCommandFilePath();
// create command file if not exists
createCommandFileIfNotExists(execCommand, commandFilePath);
+ result = runCommandFile(commandFilePath);
+
+ return result;
+ }
+
+ private TaskResponse runCommandFile(String commandFilePath) throws IOException, InterruptedException {
+ return runningOnSSH ? runOnSSH(commandFilePath) : runOnLocalProcess(commandFilePath);
+ }
+
+ private TaskResponse runOnSSH(String commandFilePath) {
+ TaskResponse result = new TaskResponse();
+ SSHSessionHolder sessionHolder = null;
+ this.sessionHost = bashTaskExecutionContext.getSessionHost();
+ try {
+ sessionHolder = SSHSessionPool.getSessionHolder(sessionHost);
+ sessionHolder.setSftpConfig(SSHSessionPool.getSftpConfig());
+ logger.info("borrow session:{} success", sessionHolder);
+
+ boolean uploadRes =
+ sessionHolder.sftpDir(taskRequest.getExecutePath(), taskRequest.getExecutePath(), logger);
+ if (!uploadRes) {
+ logger.error("upload task {} execute path to remote session {} failed", taskRequest.getExecutePath(),
+ sessionHost.toString());
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ return result;
+ }
+
+ if (taskRequest.getEnvFile() != null) {
+ boolean uploadEnvRes =
+ sessionHolder.sftpDir(taskRequest.getEnvFile(), taskRequest.getExecutePath(), logger);
+ if (!uploadEnvRes) {
+ logger.error("upload task {} execute path to remote session {} failed", taskRequest.getEnvFile(),
+ sessionHost.toString());
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ return result;
+ }
+ }
+
+ // because JSch .chmod is not stable, so use the 'chmod -R' instead
+ logger.info("update remote path's permission:{} on session:{} to 755", taskRequest.getExecutePath(),
+ sessionHost.toString());
+ String chmodCommand = "chmod -R 755 " + taskRequest.getExecutePath();
Review Comment:
> > Why it should chmod 755? I'm not sure whether it will cause some CVE? cc @ruanwenjun
>
> because remote server should keep same as DS worker, and just chmod the execute path, if there are any worry, we can change to 600. WDYT
If chmod 600 can work, I think it is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1021546423
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/AbstractCommandExecutor.java:
##########
@@ -190,11 +220,105 @@ public TaskResponse run(String execCommand) throws IOException, InterruptedExcep
return result;
}
+ if (bashTaskExecutionContext.getSessionHost() != null) {
+ runningOnSSH = true;
+ }
+
+ if (runningOnSSH && SystemUtils.IS_OS_WINDOWS) {
+ logger.error("SSH does not support Windows systems");
+ TaskExecutionContextCacheManager.removeByTaskInstanceId(taskInstanceId);
+ return result;
+ }
+
String commandFilePath = buildCommandFilePath();
// create command file if not exists
createCommandFileIfNotExists(execCommand, commandFilePath);
+ result = runCommandFile(commandFilePath);
+
+ return result;
+ }
+
+ private TaskResponse runCommandFile(String commandFilePath) throws IOException, InterruptedException {
+ return runningOnSSH ? runOnSSH(commandFilePath) : runOnLocalProcess(commandFilePath);
+ }
+
+ private TaskResponse runOnSSH(String commandFilePath) {
+ TaskResponse result = new TaskResponse();
+ SSHSessionHolder sessionHolder = null;
+ this.sessionHost = bashTaskExecutionContext.getSessionHost();
+ try {
+ sessionHolder = SSHSessionPool.getSessionHolder(sessionHost);
+ sessionHolder.setSftpConfig(SSHSessionPool.getSftpConfig());
+ logger.info("borrow session:{} success", sessionHolder);
+
+ boolean uploadRes =
+ sessionHolder.sftpDir(taskRequest.getExecutePath(), taskRequest.getExecutePath(), logger);
+ if (!uploadRes) {
+ logger.error("upload task {} execute path to remote session {} failed", taskRequest.getExecutePath(),
+ sessionHost.toString());
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ return result;
+ }
+
+ if (taskRequest.getEnvFile() != null) {
+ boolean uploadEnvRes =
+ sessionHolder.sftpDir(taskRequest.getEnvFile(), taskRequest.getExecutePath(), logger);
+ if (!uploadEnvRes) {
+ logger.error("upload task {} execute path to remote session {} failed", taskRequest.getEnvFile(),
+ sessionHost.toString());
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ return result;
+ }
+ }
+
+ // because JSch .chmod is not stable, so use the 'chmod -R' instead
+ logger.info("update remote path's permission:{} on session:{} to 755", taskRequest.getExecutePath(),
+ sessionHost.toString());
+ String chmodCommand = "chmod -R 755 " + taskRequest.getExecutePath();
Review Comment:
> Why it should chmod 755? I'm not sure whether it will cause some CVE? cc @ruanwenjun
because remote server should keep same as DS worker, and just chmod the execute path, if there are any worry, we can change to 600. WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1015310593
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -2007,3 +2010,23 @@ CREATE TABLE `t_ds_fav_task`
) ENGINE = InnoDB
AUTO_INCREMENT = 1
DEFAULT CHARSET = utf8;
+
+-- ----------------------------
+-- Table structure for t_ds_task_remote_host
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_task_remote_host`;
+CREATE TABLE `t_ds_task_remote_host`
+(
+ `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'id',
+ `code` bigint(20) NOT NULL DEFAULT '0' COMMENT 'encoding',
Review Comment:
> It's better to add indexes in this table. So as postgresql.
Sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1014952861
##########
dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql:
##########
@@ -2007,3 +2010,23 @@ CREATE TABLE `t_ds_fav_task`
) ENGINE = InnoDB
AUTO_INCREMENT = 1
DEFAULT CHARSET = utf8;
+
+-- ----------------------------
+-- Table structure for t_ds_task_remote_host
+-- ----------------------------
+DROP TABLE IF EXISTS `t_ds_task_remote_host`;
+CREATE TABLE `t_ds_task_remote_host`
+(
+ `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'id',
+ `code` bigint(20) NOT NULL DEFAULT '0' COMMENT 'encoding',
Review Comment:
It's better to add indexes in this table. So as postgresql.
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ssh/SSHSessionHolder.java:
##########
@@ -0,0 +1,348 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.api.ssh;
+
+import org.apache.dolphinscheduler.plugin.task.api.model.SSHSessionHost;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.jcraft.jsch.Channel;
+import com.jcraft.jsch.ChannelExec;
+import com.jcraft.jsch.ChannelSftp;
+import com.jcraft.jsch.JSch;
+import com.jcraft.jsch.JSchException;
+import com.jcraft.jsch.Session;
+import com.jcraft.jsch.SftpException;
+
+/**
+ * ACP pooled object: JSch Session Wrapper
+ */
+public class SSHSessionHolder {
+
+ private static final Logger logger = LoggerFactory.getLogger(SSHSessionHolder.class);
+
+ private static final String EXEC_TYPE = "exec";
+
+ private static final String SFTP_TYPE = "sftp";
+
+ private static final String SHELL_TYPE = "shell";
+
+ private static final int DEFAULT_CONNECT_TIMEOUT = 5000;
+
+ private static final int KB_UNIT = 1024;
+
+ public static final String SINGLE_SLASH = "/";
+
+ private Session session;
+
+ private final SSHSessionHost sessionHost;
+
+ private final String id;
+
+ private SftpConfig sftpConfig;
+
+ public SSHSessionHolder(SSHSessionHost sessionHost) {
+ this.sessionHost = sessionHost;
+ this.id = UUID.randomUUID().toString();
+ this.session = null;
+ }
+
+ public void connect() throws JSchException {
+ this.connect(DEFAULT_CONNECT_TIMEOUT);
+ }
+
+ public void connect(int timeoutMills) throws JSchException {
+ JSch jSch = new JSch();
+ this.session = jSch.getSession(sessionHost.getAccount(), sessionHost.getIp(), sessionHost.getPort());
+ this.session.setTimeout(timeoutMills);
+ this.session.setConfig("StrictHostKeyChecking", "no");
+ // other authorization methods can be considered in the future
+ this.session.setPassword(sessionHost.getPassword());
+ this.session.connect();
+ logger.info("Connected to ssh session: {}, session id: {}", sessionHost.toString(), id);
+ }
+
+ public void disconnect() {
+ if (session != null) {
+ session.disconnect();
+ }
+ }
+
+ public boolean isConnected() {
+ return session.isConnected();
+ }
+
+ public void keepAlive() throws Exception {
+ if (session != null) {
+ session.sendKeepAliveMsg();
+ }
+ }
+
+ public SSHResponse execCommand(String command) {
+ return this.execCommand(command, -1, logger);
+ }
+
+ public SSHResponse execCommand(String command, long timeout, Logger logger) {
+ return this.execCommand(createChannelExec(), command, timeout, logger);
+ }
+
+ public SSHResponse execCommand(ChannelExec channelExec, String command, long timeout, Logger customLogger) {
+ customLogger.info("Executing command {} on session:{}", command, this);
+ channelExec.setCommand(command);
+ channelExec.setInputStream(null);
+ channelExec.setErrStream(System.err);
+ // it will kill process when channel disconnect, default true
+ channelExec.setPty(true);
+
+ SSHResponse response = new SSHResponse();
+ try (
+ InputStream in = channelExec.getInputStream();
+ InputStream err = channelExec.getErrStream();) {
+ channelExec.connect(DEFAULT_CONNECT_TIMEOUT);
+
+ long timeoutSave = timeout > 0 ? timeout : Integer.MAX_VALUE;
+ List<String> out = new ArrayList<String>();
+ byte[] outSave = new byte[1024];
+ while (true) {
+ timeoutSave--;
+ while (in.available() > 0) {
+ int i = in.read(outSave, 0, 1024);
+ if (i < 0) {
+ break;
+ }
+ String line = new String(outSave, 0, i, StandardCharsets.UTF_8);
+ out.add(line);
+ customLogger.info(line);
+ }
+ while (err.available() > 0) {
+ int i = err.read(outSave, 0, 1024);
+ if (i < 0) {
+ break;
+ }
+ String line = new String(outSave, 0, i, StandardCharsets.UTF_8);
+ out.add(line);
+ customLogger.error(line);
+ }
+ if (channelExec.isClosed() || channelExec.isEOF()) {
+ response.setExitCode(channelExec.getExitStatus());
+ break;
+ }
+ TimeUnit.MILLISECONDS.sleep(1000);
+ if (timeoutSave < 0) {
+ customLogger.error("Exec command {} on session {} timed out", command, this);
+ throw new SSHException("Exec command " + command + " on session " + this + " timed out");
+ }
+ }
+ response.setOut(out);
+ logger.info("Exec command {} on session {} finished, exit code: {}", command, this, response.getExitCode());
+ return response;
+ } catch (Exception e) {
+ throw new SSHException("Exec command " + command + " on session " + this + " failed", e);
+ } finally {
+ close(channelExec);
+ }
+ }
+
+ public boolean sftpDir(String localDirPath, String remoteDirPath) {
+ return sftpDir(localDirPath, remoteDirPath, logger);
+ }
+
+ public boolean sftpDir(String localDirPath, String remoteDirPath, Logger customLogger) {
+ return sftpDir(createChannelSftp(), localDirPath, remoteDirPath, sftpConfig.isEnableUploadMonitor(),
+ sftpConfig.getMaxUploadRate(), sftpConfig.getMaxFileSize(), customLogger);
+ }
+
+ /**
+ * Sftp local directory to remote host,
+ * @param channelSftp SSH sftp channel
+ * @param localDirPath local directory path
+ * @param remoteDirPath remote target directory path
+ * @param enableUploadMonitor enable upload monitor thread
+ * @param maxUploadRate max upload rate, if negative, will not limit
+ * @param maxFileSize max file size, if negative, will not limit
+ * @param customLogger custom logger, default local
+ * @return sftp result
+ */
+ public boolean sftpDir(ChannelSftp channelSftp, String localDirPath, String remoteDirPath,
+ boolean enableUploadMonitor, int maxUploadRate, int maxFileSize, Logger customLogger) {
+ customLogger.info("Start to sftp local dir: {} to {}:{}", localDirPath, sessionHost.toString(), remoteDirPath);
+
+ File file = new File(localDirPath);
+ if (!file.exists()) {
+ customLogger.error("{} not exists.", localDirPath);
+ return false;
+ }
+
+ try {
+ channelSftp.connect(DEFAULT_CONNECT_TIMEOUT);
+ try {
+ channelSftp.cd(remoteDirPath);
+ } catch (SftpException e) {
+ if (!createDirOnRemote(remoteDirPath)) {
+ customLogger.error("Create directory:{} on remote:{} failed, so exit.", remoteDirPath,
+ sessionHost.toString());
+ return false;
+ }
+ }
+
+ long totalSize = file.length();
+
+ if (maxFileSize >= 0) {
+ if (totalSize > (long) maxFileSize * KB_UNIT * KB_UNIT) {
+ customLogger.error("The size of :{} has exceeded the maximum size:{}, size: {}", totalSize,
+ maxFileSize, localDirPath);
+ return false;
+ }
+ }
+
+ if (file.isDirectory()) {
+ File[] files = file.listFiles();
+ if (files == null || files.length == 0) {
+ customLogger.error("{} is a empty directory.", localDirPath);
+ return false;
+ }
+ for (File subFile : files) {
+ String sf = subFile.getCanonicalPath();
+ if (subFile.isDirectory()) {
+ String mkdirPath = remoteDirPath + "/" + subFile.getName();
+ try {
+ channelSftp.cd(mkdirPath);
+ } catch (SftpException e) {
+ if (!createDirOnRemote(mkdirPath)) {
+ customLogger.error("Could not create directory {} on remote session:{}", mkdirPath,
+ sessionHost.toString());
+ return false;
+ }
+ }
+ if (!sftpDir(createChannelSftp(), sf, mkdirPath, enableUploadMonitor, maxUploadRate, -1,
+ customLogger)) {
+ customLogger.error("sftp {} to {}:{} failed.", sf, sessionHost.toString(), mkdirPath);
+ return false;
+ }
+ } else {
+ upload(channelSftp, sf, remoteDirPath, enableUploadMonitor, maxUploadRate);
+ }
+ }
+ } else {
+ upload(channelSftp, file.getCanonicalPath(), remoteDirPath, enableUploadMonitor, maxUploadRate);
+ }
+ return true;
+ } catch (Exception e) {
+ throw new SSHException(
+ "sftp " + localDirPath + " to " + sessionHost.toString() + ":" + remoteDirPath + " failed.", e);
+ }
+ }
+
+ /**
+ * Create a directory on the remote server
+ * Because JSch does not support one-time creation of multi-layer directories, so just use `mkdir -p` instead
+ * @param remoteDirPath remote directory path
+ * @return result
+ */
+ public boolean createDirOnRemote(String remoteDirPath) {
+ logger.info("create directory:{} on remote:{}", remoteDirPath, sessionHost.toString());
+ SSHResponse response = execCommand("mkdir -p " + remoteDirPath);
+ return response.getExitCode() == 0;
+ }
+
+ public void clearPath(String path) {
+ if (SINGLE_SLASH.equals(path)) {
+ return;
+ }
+ execCommand("rm -rf " + path);
Review Comment:
I think it's better to use `File.delete` to perform operations instead of shell commands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1315263349
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1495163797
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1664793685
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] closed pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
URL: https://github.com/apache/dolphinscheduler/pull/12736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1315195233
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1327508343
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1313951454
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1315268607
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#issuecomment-1313962465
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=12736)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL) [45 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=12736&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list) [23.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=12736&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DarkAssassinator commented on a diff in pull request #12736: [Feature-11652][2022Q4 RoadMap] DS support remote SSH task.
Posted by GitBox <gi...@apache.org>.
DarkAssassinator commented on code in PR #12736:
URL: https://github.com/apache/dolphinscheduler/pull/12736#discussion_r1015311331
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/ssh/SSHSessionHolder.java:
##########
@@ -0,0 +1,348 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.api.ssh;
+
+import org.apache.dolphinscheduler.plugin.task.api.model.SSHSessionHost;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.jcraft.jsch.Channel;
+import com.jcraft.jsch.ChannelExec;
+import com.jcraft.jsch.ChannelSftp;
+import com.jcraft.jsch.JSch;
+import com.jcraft.jsch.JSchException;
+import com.jcraft.jsch.Session;
+import com.jcraft.jsch.SftpException;
+
+/**
+ * ACP pooled object: JSch Session Wrapper
+ */
+public class SSHSessionHolder {
+
+ private static final Logger logger = LoggerFactory.getLogger(SSHSessionHolder.class);
+
+ private static final String EXEC_TYPE = "exec";
+
+ private static final String SFTP_TYPE = "sftp";
+
+ private static final String SHELL_TYPE = "shell";
+
+ private static final int DEFAULT_CONNECT_TIMEOUT = 5000;
+
+ private static final int KB_UNIT = 1024;
+
+ public static final String SINGLE_SLASH = "/";
+
+ private Session session;
+
+ private final SSHSessionHost sessionHost;
+
+ private final String id;
+
+ private SftpConfig sftpConfig;
+
+ public SSHSessionHolder(SSHSessionHost sessionHost) {
+ this.sessionHost = sessionHost;
+ this.id = UUID.randomUUID().toString();
+ this.session = null;
+ }
+
+ public void connect() throws JSchException {
+ this.connect(DEFAULT_CONNECT_TIMEOUT);
+ }
+
+ public void connect(int timeoutMills) throws JSchException {
+ JSch jSch = new JSch();
+ this.session = jSch.getSession(sessionHost.getAccount(), sessionHost.getIp(), sessionHost.getPort());
+ this.session.setTimeout(timeoutMills);
+ this.session.setConfig("StrictHostKeyChecking", "no");
+ // other authorization methods can be considered in the future
+ this.session.setPassword(sessionHost.getPassword());
+ this.session.connect();
+ logger.info("Connected to ssh session: {}, session id: {}", sessionHost.toString(), id);
+ }
+
+ public void disconnect() {
+ if (session != null) {
+ session.disconnect();
+ }
+ }
+
+ public boolean isConnected() {
+ return session.isConnected();
+ }
+
+ public void keepAlive() throws Exception {
+ if (session != null) {
+ session.sendKeepAliveMsg();
+ }
+ }
+
+ public SSHResponse execCommand(String command) {
+ return this.execCommand(command, -1, logger);
+ }
+
+ public SSHResponse execCommand(String command, long timeout, Logger logger) {
+ return this.execCommand(createChannelExec(), command, timeout, logger);
+ }
+
+ public SSHResponse execCommand(ChannelExec channelExec, String command, long timeout, Logger customLogger) {
+ customLogger.info("Executing command {} on session:{}", command, this);
+ channelExec.setCommand(command);
+ channelExec.setInputStream(null);
+ channelExec.setErrStream(System.err);
+ // it will kill process when channel disconnect, default true
+ channelExec.setPty(true);
+
+ SSHResponse response = new SSHResponse();
+ try (
+ InputStream in = channelExec.getInputStream();
+ InputStream err = channelExec.getErrStream();) {
+ channelExec.connect(DEFAULT_CONNECT_TIMEOUT);
+
+ long timeoutSave = timeout > 0 ? timeout : Integer.MAX_VALUE;
+ List<String> out = new ArrayList<String>();
+ byte[] outSave = new byte[1024];
+ while (true) {
+ timeoutSave--;
+ while (in.available() > 0) {
+ int i = in.read(outSave, 0, 1024);
+ if (i < 0) {
+ break;
+ }
+ String line = new String(outSave, 0, i, StandardCharsets.UTF_8);
+ out.add(line);
+ customLogger.info(line);
+ }
+ while (err.available() > 0) {
+ int i = err.read(outSave, 0, 1024);
+ if (i < 0) {
+ break;
+ }
+ String line = new String(outSave, 0, i, StandardCharsets.UTF_8);
+ out.add(line);
+ customLogger.error(line);
+ }
+ if (channelExec.isClosed() || channelExec.isEOF()) {
+ response.setExitCode(channelExec.getExitStatus());
+ break;
+ }
+ TimeUnit.MILLISECONDS.sleep(1000);
+ if (timeoutSave < 0) {
+ customLogger.error("Exec command {} on session {} timed out", command, this);
+ throw new SSHException("Exec command " + command + " on session " + this + " timed out");
+ }
+ }
+ response.setOut(out);
+ logger.info("Exec command {} on session {} finished, exit code: {}", command, this, response.getExitCode());
+ return response;
+ } catch (Exception e) {
+ throw new SSHException("Exec command " + command + " on session " + this + " failed", e);
+ } finally {
+ close(channelExec);
+ }
+ }
+
+ public boolean sftpDir(String localDirPath, String remoteDirPath) {
+ return sftpDir(localDirPath, remoteDirPath, logger);
+ }
+
+ public boolean sftpDir(String localDirPath, String remoteDirPath, Logger customLogger) {
+ return sftpDir(createChannelSftp(), localDirPath, remoteDirPath, sftpConfig.isEnableUploadMonitor(),
+ sftpConfig.getMaxUploadRate(), sftpConfig.getMaxFileSize(), customLogger);
+ }
+
+ /**
+ * Sftp local directory to remote host,
+ * @param channelSftp SSH sftp channel
+ * @param localDirPath local directory path
+ * @param remoteDirPath remote target directory path
+ * @param enableUploadMonitor enable upload monitor thread
+ * @param maxUploadRate max upload rate, if negative, will not limit
+ * @param maxFileSize max file size, if negative, will not limit
+ * @param customLogger custom logger, default local
+ * @return sftp result
+ */
+ public boolean sftpDir(ChannelSftp channelSftp, String localDirPath, String remoteDirPath,
+ boolean enableUploadMonitor, int maxUploadRate, int maxFileSize, Logger customLogger) {
+ customLogger.info("Start to sftp local dir: {} to {}:{}", localDirPath, sessionHost.toString(), remoteDirPath);
+
+ File file = new File(localDirPath);
+ if (!file.exists()) {
+ customLogger.error("{} not exists.", localDirPath);
+ return false;
+ }
+
+ try {
+ channelSftp.connect(DEFAULT_CONNECT_TIMEOUT);
+ try {
+ channelSftp.cd(remoteDirPath);
+ } catch (SftpException e) {
+ if (!createDirOnRemote(remoteDirPath)) {
+ customLogger.error("Create directory:{} on remote:{} failed, so exit.", remoteDirPath,
+ sessionHost.toString());
+ return false;
+ }
+ }
+
+ long totalSize = file.length();
+
+ if (maxFileSize >= 0) {
+ if (totalSize > (long) maxFileSize * KB_UNIT * KB_UNIT) {
+ customLogger.error("The size of :{} has exceeded the maximum size:{}, size: {}", totalSize,
+ maxFileSize, localDirPath);
+ return false;
+ }
+ }
+
+ if (file.isDirectory()) {
+ File[] files = file.listFiles();
+ if (files == null || files.length == 0) {
+ customLogger.error("{} is a empty directory.", localDirPath);
+ return false;
+ }
+ for (File subFile : files) {
+ String sf = subFile.getCanonicalPath();
+ if (subFile.isDirectory()) {
+ String mkdirPath = remoteDirPath + "/" + subFile.getName();
+ try {
+ channelSftp.cd(mkdirPath);
+ } catch (SftpException e) {
+ if (!createDirOnRemote(mkdirPath)) {
+ customLogger.error("Could not create directory {} on remote session:{}", mkdirPath,
+ sessionHost.toString());
+ return false;
+ }
+ }
+ if (!sftpDir(createChannelSftp(), sf, mkdirPath, enableUploadMonitor, maxUploadRate, -1,
+ customLogger)) {
+ customLogger.error("sftp {} to {}:{} failed.", sf, sessionHost.toString(), mkdirPath);
+ return false;
+ }
+ } else {
+ upload(channelSftp, sf, remoteDirPath, enableUploadMonitor, maxUploadRate);
+ }
+ }
+ } else {
+ upload(channelSftp, file.getCanonicalPath(), remoteDirPath, enableUploadMonitor, maxUploadRate);
+ }
+ return true;
+ } catch (Exception e) {
+ throw new SSHException(
+ "sftp " + localDirPath + " to " + sessionHost.toString() + ":" + remoteDirPath + " failed.", e);
+ }
+ }
+
+ /**
+ * Create a directory on the remote server
+ * Because JSch does not support one-time creation of multi-layer directories, so just use `mkdir -p` instead
+ * @param remoteDirPath remote directory path
+ * @return result
+ */
+ public boolean createDirOnRemote(String remoteDirPath) {
+ logger.info("create directory:{} on remote:{}", remoteDirPath, sessionHost.toString());
+ SSHResponse response = execCommand("mkdir -p " + remoteDirPath);
+ return response.getExitCode() == 0;
+ }
+
+ public void clearPath(String path) {
+ if (SINGLE_SLASH.equals(path)) {
+ return;
+ }
+ execCommand("rm -rf " + path);
Review Comment:
> I think it's better to use `File.delete` to perform operations instead of shell commands.
Becuase these codes need to clear the path on the remote SSH server, so use the `rm -rf` command, not `File.delete`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org