You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "rickchengx (via GitHub)" <gi...@apache.org> on 2023/07/11 06:55:50 UTC
[GitHub] [dolphinscheduler] rickchengx opened a new pull request, #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
rickchengx opened a new pull request, #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504
<!--Thanks very much for contributing to Apache DolphinScheduler, we are happy that you want to help us improve DolphinScheduler! -->
## Purpose of the pull request
* close: #14503
## Brief change log
* `doKill()` must be running before `cancelApplication()` so that `pstree` could get all the sub process id and kill them
## Verify this pull request
<img width="511" alt="截屏2023-07-11 14 46 38" src="https://github.com/apache/dolphinscheduler/assets/38122586/aff6f7a5-eec3-4d09-822a-eabf9bf46773">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1638022450
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14504)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1630305203
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14504)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS merged pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS merged PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1631789475
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14504)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on a diff in pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on code in PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#discussion_r1261898498
##########
dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/processor/WorkerTaskKillProcessor.java:
##########
@@ -98,6 +96,7 @@ public void process(Channel channel, Message message) {
}
boolean result = doKill(taskExecutionContext);
+ this.cancelApplication(taskInstanceId);
Review Comment:
> I cannot reproduce this on my mac.
I think in some cases, `process.destroy` cannot kill the subprocess.
<img width="637" alt="截屏2023-07-13 10 12 39" src="https://github.com/apache/dolphinscheduler/assets/38122586/384ce08a-40e3-46ce-8a2d-27ca08461a1e">
https://stackoverflow.com/questions/5879791/how-to-kill-subprocesses-of-a-java-process
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on a diff in pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on code in PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#discussion_r1260482673
##########
dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/processor/WorkerTaskKillProcessor.java:
##########
@@ -98,6 +96,7 @@ public void process(Channel channel, Message message) {
}
boolean result = doKill(taskExecutionContext);
+ this.cancelApplication(taskInstanceId);
Review Comment:
Do u mean that all the system process (include subprocess) and the remote process (like yarn) should be killed in `cancelApplication` in each task plugin ?
* but the `process.destroy()` cannot kill the subprocess. like `sleep 300` in the shell task. the `sleep 300`
is still running after we stop the workflow instance.
<img width="438" alt="截屏2023-07-11 14 31 07" src="https://github.com/apache/dolphinscheduler/assets/38122586/5462dd75-4686-4f11-b075-953330e1594a">
* And also the remote process now is killed in `doKill()`, not in each task plugin.
##########
dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/processor/WorkerTaskKillProcessor.java:
##########
@@ -98,6 +96,7 @@ public void process(Channel channel, Message message) {
}
boolean result = doKill(taskExecutionContext);
+ this.cancelApplication(taskInstanceId);
Review Comment:
Do u mean that all the system process (include subprocess) and the remote process (like yarn) should be killed in `cancelApplication` in each task plugin ?
* but the `process.destroy()` cannot kill the subprocess. like `sleep 300` in the shell task. the `sleep 300`
is still running after we stop the workflow instance.
<img width="438" alt="截屏2023-07-11 14 31 07" src="https://github.com/apache/dolphinscheduler/assets/38122586/5462dd75-4686-4f11-b075-953330e1594a">
* And also the remote process now is killed in `doKill()`, not in each task plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1634234696
> > I cannot reproduce this on my mac.
>
> ### How to reproduce
> > create a shell task with sleep 300
> > kill the task
> > sleep 300 is still running
>
> Hi, @zhongjiajie @Radeity @SbloodyS Could you guys please take a time to see if this bug can be reproduced?
>
> On my mac, the task is killed at 14:48 <img alt="image" width="1122" src="https://user-images.githubusercontent.com/38122586/253188970-7a0e82b4-24eb-4e59-b622-e148eb64838f.png">
>
> but `sleep 300` is still running at 14:49 <img alt="截屏2023-07-13 14 50 13" width="431" src="https://user-images.githubusercontent.com/38122586/253189300-75737d6a-3a22-433c-a88d-ec41737f9831.png">
Hi, @rickchengx , I can reproduce this bug on my linux server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1631793486
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14504)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1630329389
Hi, @Radeity could you please help review this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on code in PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#discussion_r1259832096
##########
dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/processor/WorkerTaskKillProcessor.java:
##########
@@ -98,6 +96,7 @@ public void process(Channel channel, Message message) {
}
boolean result = doKill(taskExecutionContext);
+ this.cancelApplication(taskInstanceId);
Review Comment:
I hope we can remove the `doKill` method, the kill operation should be implemented by task plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on code in PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#discussion_r1261322321
##########
dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/processor/WorkerTaskKillProcessor.java:
##########
@@ -98,6 +96,7 @@ public void process(Channel channel, Message message) {
}
boolean result = doKill(taskExecutionContext);
+ this.cancelApplication(taskInstanceId);
Review Comment:
I cannot reproduce this on my mac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on a diff in pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on code in PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#discussion_r1261898498
##########
dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/processor/WorkerTaskKillProcessor.java:
##########
@@ -98,6 +96,7 @@ public void process(Channel channel, Message message) {
}
boolean result = doKill(taskExecutionContext);
+ this.cancelApplication(taskInstanceId);
Review Comment:
> I cannot reproduce this on my mac.
I think in some cases, `process.destroy` cannot kill the subprocess.
##########
dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/processor/WorkerTaskKillProcessor.java:
##########
@@ -98,6 +96,7 @@ public void process(Channel channel, Message message) {
}
boolean result = doKill(taskExecutionContext);
+ this.cancelApplication(taskInstanceId);
Review Comment:
> I cannot reproduce this on my mac.
I think in some cases, `process.destroy` cannot kill the subprocess.
<img width="637" alt="截屏2023-07-13 10 12 39" src="https://github.com/apache/dolphinscheduler/assets/38122586/384ce08a-40e3-46ce-8a2d-27ca08461a1e">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1634248812
> I hope we can remove the `doKill` method, the kill operation should be implemented by task plugin.
Agree with you, current kill operation seems mess.
However, for this PR, this fix does work. If `process.destroy` cannot kill the subprocess, we can only get subprocess id in `getPidsStr` before the main process is killed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1630288770
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14504)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on a diff in pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on code in PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#discussion_r1260482673
##########
dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/processor/WorkerTaskKillProcessor.java:
##########
@@ -98,6 +96,7 @@ public void process(Channel channel, Message message) {
}
boolean result = doKill(taskExecutionContext);
+ this.cancelApplication(taskInstanceId);
Review Comment:
@ruanwenjun Do u mean that all the system process (include subprocess) and the remote process (like yarn) should be killed in `cancelApplication` in each task plugin ?
* but the `process.destroy()` cannot kill the subprocess. like `sleep 300` in the shell task. the `sleep 300`
is still running after we stop the workflow instance.
<img width="438" alt="截屏2023-07-11 14 31 07" src="https://github.com/apache/dolphinscheduler/assets/38122586/5462dd75-4686-4f11-b075-953330e1594a">
* And also the remote process now is killed in `doKill()`, not in each task plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1631784259
## [Codecov](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#14504](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (afabd3a) into [dev](https://app.codecov.io/gh/apache/dolphinscheduler/commit/c3c2dda861e2472b8f282ee889e5d609d0d6b36c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (c3c2dda) will **increase** coverage by `0.00%`.
> The diff coverage is `58.00%`.
> :exclamation: Current head afabd3a differs from pull request most recent head 5acc38e. Consider uploading reports for the commit 5acc38e to get more accurate results
```diff
@@ Coverage Diff @@
## dev #14504 +/- ##
=========================================
Coverage 38.72% 38.73%
+ Complexity 4577 4574 -3
=========================================
Files 1237 1237
Lines 43496 43487 -9
Branches 4817 4790 -27
=========================================
- Hits 16845 16843 -2
+ Misses 24781 24776 -5
+ Partials 1870 1868 -2
```
| [Impacted Files](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...scheduler/api/controller/DataSourceController.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRGF0YVNvdXJjZUNvbnRyb2xsZXIuamF2YQ==) | `36.36% <0.00%> (-2.35%)` | :arrow_down: |
| [...e/dolphinscheduler/common/constants/Constants.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL2NvbnN0YW50cy9Db25zdGFudHMuamF2YQ==) | `75.00% <ø> (ø)` | |
| [...er/plugin/task/api/utils/DataQualityConstants.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS91dGlscy9EYXRhUXVhbGl0eUNvbnN0YW50cy5qYXZh) | `0.00% <ø> (ø)` | |
| [...nscheduler/plugin/task/api/utils/ProcessUtils.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS91dGlscy9Qcm9jZXNzVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...rver/worker/processor/WorkerTaskKillProcessor.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci13b3JrZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci9wcm9jZXNzb3IvV29ya2VyVGFza0tpbGxQcm9jZXNzb3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...eduler/api/service/impl/DataSourceServiceImpl.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9EYXRhU291cmNlU2VydmljZUltcGwuamF2YQ==) | `50.83% <58.33%> (+2.36%)` | :arrow_up: |
| [.../org/apache/dolphinscheduler/api/enums/Status.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2VudW1zL1N0YXR1cy5qYXZh) | `100.00% <100.00%> (ø)` | |
| [...ler/data/quality/flow/batch/reader/JdbcReader.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1kYXRhLXF1YWxpdHkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGF0YS9xdWFsaXR5L2Zsb3cvYmF0Y2gvcmVhZGVyL0pkYmNSZWFkZXIuamF2YQ==) | `95.00% <100.00%> (ø)` | |
| [...ler/data/quality/flow/batch/writer/JdbcWriter.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1kYXRhLXF1YWxpdHkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGF0YS9xdWFsaXR5L2Zsb3cvYmF0Y2gvd3JpdGVyL0pkYmNXcml0ZXIuamF2YQ==) | `85.00% <100.00%> (ø)` | |
| [...cheduler/plugin/task/dq/utils/RuleParserUtils.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stZGF0YXF1YWxpdHkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svZHEvdXRpbHMvUnVsZVBhcnNlclV0aWxzLmphdmE=) | `87.03% <100.00%> (+0.04%)` | :arrow_up: |
... and [37 files with indirect coverage changes](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14504/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1630949305
This is strategy, when we kill the shell task, we will use `process.destroy();` to cancel the process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1638026516
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14504)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14504&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14504&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1633667104
> I cannot reproduce this on my mac.
### How to reproduce
> create a shell task with sleep 300
kill the task
sleep 300 is still running
Hi, @zhongjiajie @Radeity Could you guys please take a time to reproduce this bug ?
On my mac, the task is killed at 14:48
<img width="1122" alt="image" src="https://github.com/apache/dolphinscheduler/assets/38122586/7a0e82b4-24eb-4e59-b622-e148eb64838f">
but `sleep 300` is still running at 14:49
<img width="431" alt="截屏2023-07-13 14 50 13" src="https://github.com/apache/dolphinscheduler/assets/38122586/75737d6a-3a22-433c-a88d-ec41737f9831">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on pull request #14504: [Fix-14503][Worker] Fix the problem of subprocess cannot be killed
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on PR #14504:
URL: https://github.com/apache/dolphinscheduler/pull/14504#issuecomment-1635542351
@ruanwenjun I think we could fix this bug first through this PR. And we could improve the kill logic to use the `cancelApplication()` in each task plugin. It looks like a lot of work. WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org