You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/06/16 09:52:35 UTC
[GitHub] [dolphinscheduler] ruanwenjun opened a new pull request, #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
ruanwenjun opened a new pull request, #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479
## Purpose of the pull request
Close #10470, maybe we need to remove the variable in processor, then we don't need to create it in each task, or we can use threadlocal to store the variable.
## Brief change log
- Don't cache the CommonTaskProcessor in TaskProcessorFactory
- Add some log
## Verify this pull request
Trigger multiple process instance, then see all process instance can run successful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun merged pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
Posted by GitBox <gi...@apache.org>.
ruanwenjun merged PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479#discussion_r898916672
##########
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProcessInstance.java:
##########
@@ -265,12 +265,13 @@ public ProcessInstance() {
*/
public ProcessInstance(ProcessDefinition processDefinition) {
this.processDefinition = processDefinition;
+ // todo: the name is not unique
this.name = processDefinition.getName()
- + "-"
- +
- processDefinition.getVersion()
- + "-"
- +
+ + "-"
+ +
+ processDefinition.getVersion()
Review Comment:
nit: what about also simplify this with `String.join` or `Joiner` in guava, it should be improved in readability?
##########
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProcessInstance.java:
##########
@@ -265,12 +265,13 @@ public ProcessInstance() {
*/
public ProcessInstance(ProcessDefinition processDefinition) {
this.processDefinition = processDefinition;
+ // todo: the name is not unique
Review Comment:
Can we do this in this PR?
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumer.java:
##########
@@ -190,16 +190,18 @@ protected boolean dispatchTask(TaskPriority taskPriority) {
return true;
}
}
-
result = dispatcher.dispatch(executionContext);
if (result) {
+ logger.info("Master success dispatch task to worker, taskInstanceId: {}", taskPriority.getTaskId());
addDispatchEvent(context, executionContext);
+ } else {
+ logger.info("Master failed to dispatch task to worker, taskInstanceId: {}", taskPriority.getTaskId());
}
} catch (RuntimeException e) {
- logger.error("dispatch error: ", e);
+ logger.error("Master dispatch task to worker error: ", e);
} catch (ExecuteException e) {
- logger.error("dispatch error: {}", e.getMessage());
+ logger.error("Master dispatch task to worker error: {}", e.getMessage());
Review Comment:
nit: also change to this so we can get the exception stack?
```suggestion
logger.error("Master dispatch task to worker error: {}", e);
```
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/processor/queue/TaskEvent.java:
##########
@@ -27,10 +27,12 @@
import java.util.Date;
import io.netty.channel.Channel;
+import lombok.Data;
/**
* task event
*/
+@Data
Review Comment:
💯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479#issuecomment-1157631764
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10479)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10479&metric=new_coverage&view=list) [10.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10479&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10479&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10479&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479#discussion_r899002399
##########
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProcessInstance.java:
##########
@@ -265,12 +265,13 @@ public ProcessInstance() {
*/
public ProcessInstance(ProcessDefinition processDefinition) {
this.processDefinition = processDefinition;
+ // todo: the name is not unique
Review Comment:
Right now, the `processInstanceName` is just used in front-end, we can change the `DateUtils.getCurrentTimeStamp()` to the snowflake id, but I am not sure if this is need to change, so I just add a todo in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479#issuecomment-1157626153
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10479](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4015a2c) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/f512e3c31f4850c7272e868b291193a58c661d57?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f512e3c) will **decrease** coverage by `0.04%`.
> The diff coverage is `21.12%`.
```diff
@@ Coverage Diff @@
## dev #10479 +/- ##
============================================
- Coverage 40.90% 40.85% -0.05%
+ Complexity 4853 4843 -10
============================================
Files 883 885 +2
Lines 35972 35974 +2
Branches 3991 3993 +2
============================================
- Hits 14715 14698 -17
- Misses 19802 19820 +18
- Partials 1455 1456 +1
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/dolphinscheduler/common/utils/FileUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3V0aWxzL0ZpbGVVdGlscy5qYXZh) | `52.94% <0.00%> (-3.31%)` | :arrow_down: |
| [...e/dolphinscheduler/dao/entity/ProcessInstance.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYW8vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGFvL2VudGl0eS9Qcm9jZXNzSW5zdGFuY2UuamF2YQ==) | `65.56% <0.00%> (-0.44%)` | :arrow_down: |
| [...ver/master/consumer/TaskPriorityQueueConsumer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9jb25zdW1lci9UYXNrUHJpb3JpdHlRdWV1ZUNvbnN1bWVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ler/server/master/dispatch/ExecutorDispatcher.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9FeGVjdXRvckRpc3BhdGNoZXIuamF2YQ==) | `0.00% <ø> (ø)` | |
| [...duler/server/master/processor/queue/TaskEvent.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9wcm9jZXNzb3IvcXVldWUvVGFza0V2ZW50LmphdmE=) | `50.00% <0.00%> (-17.95%)` | :arrow_down: |
| [.../master/processor/queue/TaskExecuteThreadPool.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9wcm9jZXNzb3IvcXVldWUvVGFza0V4ZWN1dGVUaHJlYWRQb29sLmphdmE=) | `2.17% <0.00%> (-0.10%)` | :arrow_down: |
| [.../server/master/runner/WorkflowExecuteRunnable.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvV29ya2Zsb3dFeGVjdXRlUnVubmFibGUuamF2YQ==) | `7.74% <0.00%> (-0.01%)` | :arrow_down: |
| [...erver/master/runner/WorkflowExecuteThreadPool.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvV29ya2Zsb3dFeGVjdXRlVGhyZWFkUG9vbC5qYXZh) | `1.40% <ø> (ø)` | |
| [...server/master/runner/task/CommonTaskProcessor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvdGFzay9Db21tb25UYXNrUHJvY2Vzc29yLmphdmE=) | `4.10% <0.00%> (ø)` | |
| [...inscheduler/server/worker/config/WorkerConfig.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci13b3JrZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci9jb25maWcvV29ya2VyQ29uZmlnLmphdmE=) | `76.47% <ø> (ø)` | |
| ... and [18 more](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f512e3c...4015a2c](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org