You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/03/05 10:17:54 UTC
[GitHub] [incubator-dolphinscheduler] gitriver opened a new pull request #4965: 针对SPARK,YARN类型的任务kill时，才需要获取appId并根据appId kill yarn上的任务,其它类型的任务不需要
gitriver opened a new pull request #4965:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/4965
## *Tips*
- *Thanks very much for contributing to Apache DolphinScheduler.*
- *Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds checkstyle plugin.)*
## Brief change log
*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] CalvinKirs commented on pull request #4965: [Improvement]kill yarn type task optimization
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #4965:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/4965#issuecomment-792763669
code style check fails, you can refer this docs:https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gitriver commented on a change in pull request #4965: [Improvement]kill yarn type task optimization
Posted by GitBox <gi...@apache.org>.
gitriver commented on a change in pull request #4965:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/4965#discussion_r589878758
##########
File path: dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/processor/TaskKillProcessor.java
##########
@@ -129,11 +127,19 @@ public void process(Channel channel, Command command) {
OSUtils.exeCmd(cmd);
- // find log and kill yarn job
- appIds = killYarnJob(Host.of(taskExecutionContext.getHost()).getIp(),
- taskExecutionContext.getLogPath(),
- taskExecutionContext.getExecutePath(),
- taskExecutionContext.getTenantCode());
+ TaskType anEnum = EnumUtils.getEnum(TaskType.class, taskExecutionContext.getTaskType());
+ switch (anEnum) {
+ case SPARK:
+ case FLINK:
+ // find log and kill yarn job
+ appIds = killYarnJob(Host.of(taskExecutionContext.getHost()).getIp(),
+ taskExecutionContext.getLogPath(),
+ taskExecutionContext.getExecutePath(),
+ taskExecutionContext.getTenantCode());
+ break;
+ default:
+ appIds = Collections.EMPTY_LIST;
+ }
Review comment:
not all job is run on yarn cluster,
this place need optimize
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] zhuangchong commented on a change in pull request #4965: 针对SPARK,YARN类型的任务kill时，才需要获取appId并根据appId kill yarn上的任务,其它类型的任务不需要
Posted by GitBox <gi...@apache.org>.
zhuangchong commented on a change in pull request #4965:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/4965#discussion_r588187338
##########
File path: dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/worker/processor/TaskKillProcessor.java
##########
@@ -129,11 +127,19 @@ public void process(Channel channel, Command command) {
OSUtils.exeCmd(cmd);
- // find log and kill yarn job
- appIds = killYarnJob(Host.of(taskExecutionContext.getHost()).getIp(),
- taskExecutionContext.getLogPath(),
- taskExecutionContext.getExecutePath(),
- taskExecutionContext.getTenantCode());
+ TaskType anEnum = EnumUtils.getEnum(TaskType.class, taskExecutionContext.getTaskType());
+ switch (anEnum) {
+ case SPARK:
+ case FLINK:
+ // find log and kill yarn job
+ appIds = killYarnJob(Host.of(taskExecutionContext.getHost()).getIp(),
+ taskExecutionContext.getLogPath(),
+ taskExecutionContext.getExecutePath(),
+ taskExecutionContext.getTenantCode());
+ break;
+ default:
+ appIds = Collections.EMPTY_LIST;
+ }
Review comment:
hive-sql,mr,YARN tasks started in shell/pythod scripts, and so on. These all involve killing yarn tasks, and this change is not recommended here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] sonarcloud[bot] commented on pull request #4965: 针对SPARK,YARN类型的任务kill时，才需要获取appId并根据appId kill yarn上的任务,其它类型的任务不需要
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #4965:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/4965#issuecomment-791333990
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=VULNERABILITY)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=SECURITY_HOTSPOT) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=SECURITY_HOTSPOT)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=4965&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=4965&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=4965&metric=new_coverage&view=list)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=4965&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=4965&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org