You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/06/16 09:52:35 UTC

[GitHub] [dolphinscheduler] ruanwenjun opened a new pull request, #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe

ruanwenjun opened a new pull request, #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479

   ## Purpose of the pull request
   
   Close #10470, maybe we need to remove the variable in processor, then we don't need to create it in each task, or we can use threadlocal to store the variable.
   
   ## Brief change log
   
   - Don't cache the CommonTaskProcessor in TaskProcessorFactory
   - Add some log
   ## Verify this pull request
   
   Trigger multiple process instance, then see all process instance can run successful.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun merged pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe

Posted by GitBox <gi...@apache.org>.
ruanwenjun merged PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe

Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479#discussion_r898916672


##########
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProcessInstance.java:
##########
@@ -265,12 +265,13 @@ public ProcessInstance() {
      */
     public ProcessInstance(ProcessDefinition processDefinition) {
         this.processDefinition = processDefinition;
+        // todo: the name is not unique
         this.name = processDefinition.getName()
-            + "-"
-            +
-            processDefinition.getVersion()
-            + "-"
-            +
+                + "-"
+                +
+                processDefinition.getVersion()

Review Comment:
   nit: what about also simplify this with `String.join` or `Joiner` in guava, it should be improved in readability?



##########
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProcessInstance.java:
##########
@@ -265,12 +265,13 @@ public ProcessInstance() {
      */
     public ProcessInstance(ProcessDefinition processDefinition) {
         this.processDefinition = processDefinition;
+        // todo: the name is not unique

Review Comment:
   Can we do this in this PR?



##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumer.java:
##########
@@ -190,16 +190,18 @@ protected boolean dispatchTask(TaskPriority taskPriority) {
                     return true;
                 }
             }
-
             result = dispatcher.dispatch(executionContext);
 
             if (result) {
+                logger.info("Master success dispatch task to worker, taskInstanceId: {}", taskPriority.getTaskId());
                 addDispatchEvent(context, executionContext);
+            } else {
+                logger.info("Master failed to dispatch task to worker, taskInstanceId: {}", taskPriority.getTaskId());
             }
         } catch (RuntimeException e) {
-            logger.error("dispatch error: ", e);
+            logger.error("Master dispatch task to worker error: ", e);
         } catch (ExecuteException e) {
-            logger.error("dispatch error: {}", e.getMessage());
+            logger.error("Master dispatch task to worker error: {}", e.getMessage());

Review Comment:
   nit: also change to this so we can get the exception stack?
   
   
   ```suggestion
               logger.error("Master dispatch task to worker error: {}", e);
   ```
   



##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/processor/queue/TaskEvent.java:
##########
@@ -27,10 +27,12 @@
 import java.util.Date;
 
 import io.netty.channel.Channel;
+import lombok.Data;
 
 /**
  * task event
  */
+@Data

Review Comment:
   💯 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479#issuecomment-1157631764

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10479)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10479&resolved=false&types=CODE_SMELL)
   
   [![10.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '10.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10479&metric=new_coverage&view=list) [10.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10479&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10479&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10479&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479#discussion_r899002399


##########
dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/ProcessInstance.java:
##########
@@ -265,12 +265,13 @@ public ProcessInstance() {
      */
     public ProcessInstance(ProcessDefinition processDefinition) {
         this.processDefinition = processDefinition;
+        // todo: the name is not unique

Review Comment:
   Right now, the `processInstanceName` is just used in front-end, we can change the `DateUtils.getCurrentTimeStamp()` to the snowflake id, but I am not sure if this is need to change, so I just add a todo in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10479: Fix TaskProcessorFactory#getTaskProcessor get common processor is not thread safe

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10479:
URL: https://github.com/apache/dolphinscheduler/pull/10479#issuecomment-1157626153

   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#10479](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4015a2c) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/f512e3c31f4850c7272e868b291193a58c661d57?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f512e3c) will **decrease** coverage by `0.04%`.
   > The diff coverage is `21.12%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev   #10479      +/-   ##
   ============================================
   - Coverage     40.90%   40.85%   -0.05%     
   + Complexity     4853     4843      -10     
   ============================================
     Files           883      885       +2     
     Lines         35972    35974       +2     
     Branches       3991     3993       +2     
   ============================================
   - Hits          14715    14698      -17     
   - Misses        19802    19820      +18     
   - Partials       1455     1456       +1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...pache/dolphinscheduler/common/utils/FileUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL3V0aWxzL0ZpbGVVdGlscy5qYXZh) | `52.94% <0.00%> (-3.31%)` | :arrow_down: |
   | [...e/dolphinscheduler/dao/entity/ProcessInstance.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYW8vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGFvL2VudGl0eS9Qcm9jZXNzSW5zdGFuY2UuamF2YQ==) | `65.56% <0.00%> (-0.44%)` | :arrow_down: |
   | [...ver/master/consumer/TaskPriorityQueueConsumer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9jb25zdW1lci9UYXNrUHJpb3JpdHlRdWV1ZUNvbnN1bWVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ler/server/master/dispatch/ExecutorDispatcher.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9FeGVjdXRvckRpc3BhdGNoZXIuamF2YQ==) | `0.00% <ø> (ø)` | |
   | [...duler/server/master/processor/queue/TaskEvent.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9wcm9jZXNzb3IvcXVldWUvVGFza0V2ZW50LmphdmE=) | `50.00% <0.00%> (-17.95%)` | :arrow_down: |
   | [.../master/processor/queue/TaskExecuteThreadPool.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9wcm9jZXNzb3IvcXVldWUvVGFza0V4ZWN1dGVUaHJlYWRQb29sLmphdmE=) | `2.17% <0.00%> (-0.10%)` | :arrow_down: |
   | [.../server/master/runner/WorkflowExecuteRunnable.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvV29ya2Zsb3dFeGVjdXRlUnVubmFibGUuamF2YQ==) | `7.74% <0.00%> (-0.01%)` | :arrow_down: |
   | [...erver/master/runner/WorkflowExecuteThreadPool.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvV29ya2Zsb3dFeGVjdXRlVGhyZWFkUG9vbC5qYXZh) | `1.40% <ø> (ø)` | |
   | [...server/master/runner/task/CommonTaskProcessor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvdGFzay9Db21tb25UYXNrUHJvY2Vzc29yLmphdmE=) | `4.10% <0.00%> (ø)` | |
   | [...inscheduler/server/worker/config/WorkerConfig.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci13b3JrZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci9jb25maWcvV29ya2VyQ29uZmlnLmphdmE=) | `76.47% <ø> (ø)` | |
   | ... and [18 more](https://codecov.io/gh/apache/dolphinscheduler/pull/10479/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f512e3c...4015a2c](https://codecov.io/gh/apache/dolphinscheduler/pull/10479?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org