You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "qingwli (via GitHub)" <gi...@apache.org> on 2023/06/30 08:33:16 UTC
[GitHub] [dolphinscheduler] qingwli opened a new pull request, #14433: [Fix] Fix running task instance throught api gots failed
qingwli opened a new pull request, #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433
When calling API to run task instances like spark or data quality job will get errors;
`/dolphinscheduler/projects/{projectCode}/executors/task-instance/{code}/start`
The reason is when running task instances and building taskExecutionContext lost `DataQualityTaskExecutionContext` and `K8sTaskExecutionContext`.
Extract public methods to solve the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on code in PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433#discussion_r1252849966
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/execute/TaskExecutionContextFactory.java:
##########
@@ -121,33 +121,39 @@ public TaskExecutionContext createTaskExecutionContext(TaskInstance taskInstance
.orElse(null);
setTaskResourceInfo(resources);
- // TODO to be optimized
- DataQualityTaskExecutionContext dataQualityTaskExecutionContext = null;
- if (TASK_TYPE_DATA_QUALITY.equalsIgnoreCase(taskInstance.getTaskType())) {
- dataQualityTaskExecutionContext = new DataQualityTaskExecutionContext();
- setDataQualityTaskRelation(dataQualityTaskExecutionContext, taskInstance, workflowInstance.getTenantCode());
- }
-
- K8sTaskExecutionContext k8sTaskExecutionContext = setK8sTaskRelation(taskInstance);
-
Map<String, Property> businessParamsMap = curingParamsService.preBuildBusinessParams(workflowInstance);
AbstractParameters baseParam = taskPluginManager.getParameters(ParametersNode.builder()
.taskType(taskInstance.getTaskType()).taskParams(taskInstance.getTaskParams()).build());
Map<String, Property> propertyMap =
curingParamsService.paramParsingPreparation(taskInstance, baseParam, workflowInstance);
- return TaskExecutionContextBuilder.get()
+ TaskExecutionContext taskExecutionContext = TaskExecutionContextBuilder.get()
.buildWorkflowInstanceHost(masterConfig.getMasterAddress())
.buildTaskInstanceRelatedInfo(taskInstance)
.buildTaskDefinitionRelatedInfo(taskInstance.getTaskDefine())
.buildProcessInstanceRelatedInfo(taskInstance.getProcessInstance())
.buildProcessDefinitionRelatedInfo(taskInstance.getProcessDefine())
.buildResourceParametersInfo(resources)
- .buildDataQualityTaskExecutionContext(dataQualityTaskExecutionContext)
- .buildK8sTaskRelatedInfo(k8sTaskExecutionContext)
.buildBusinessParamsMap(businessParamsMap)
.buildParamInfo(propertyMap)
.create();
+
+ buildTaskExecutionContext(taskExecutionContext, taskInstance, workflowInstance.getTenantCode());
+ return taskExecutionContext;
+ }
+
+ public void buildTaskExecutionContext(TaskExecutionContext taskExecutionContext, TaskInstance taskInstance,
Review Comment:
This method will cause confusion, this is a side-effect method, and we don't know what will happen.
It's better to split to `setDataQualityTaskExecutionContext` and `setK8sTaskRelatedInfo`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433#issuecomment-1614462521
## [Codecov](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14433?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#14433](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14433?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (92be099) into [dev](https://app.codecov.io/gh/apache/dolphinscheduler/commit/f186cd5dab609ff5adbbe0d8b9088b69a273ab67?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (f186cd5) will **decrease** coverage by `0.02%`.
> The diff coverage is `0.00%`.
> :exclamation: Current head 92be099 differs from pull request most recent head 416dff1. Consider uploading reports for the commit 416dff1 to get more accurate results
```diff
@@ Coverage Diff @@
## dev #14433 +/- ##
============================================
- Coverage 38.55% 38.54% -0.02%
Complexity 4556 4556
============================================
Files 1236 1236
Lines 43456 43461 +5
Branches 4814 4814
============================================
- Hits 16756 16751 -5
- Misses 24842 24850 +8
- Partials 1858 1860 +2
```
| [Impacted Files](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14433?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...erver/master/runner/StreamTaskExecuteRunnable.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14433?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvU3RyZWFtVGFza0V4ZWN1dGVSdW5uYWJsZS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...er/runner/execute/TaskExecutionContextFactory.java](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14433?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvZXhlY3V0ZS9UYXNrRXhlY3V0aW9uQ29udGV4dEZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (ø)` | |
... and [7 files with indirect coverage changes](https://app.codecov.io/gh/apache/dolphinscheduler/pull/14433/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433#issuecomment-1621484226
Please retry analysis of this Pull-Request directly on [SonarCloud](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] qingwli commented on pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "qingwli (via GitHub)" <gi...@apache.org>.
qingwli commented on PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433#issuecomment-1621202945
PTAL @caishunfeng @ruanwenjun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433#issuecomment-1621483690
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14433)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_duplicated_lines_density&view=list)
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] qingwli commented on a diff in pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "qingwli (via GitHub)" <gi...@apache.org>.
qingwli commented on code in PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433#discussion_r1252859760
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/execute/TaskExecutionContextFactory.java:
##########
@@ -121,33 +121,39 @@ public TaskExecutionContext createTaskExecutionContext(TaskInstance taskInstance
.orElse(null);
setTaskResourceInfo(resources);
- // TODO to be optimized
- DataQualityTaskExecutionContext dataQualityTaskExecutionContext = null;
- if (TASK_TYPE_DATA_QUALITY.equalsIgnoreCase(taskInstance.getTaskType())) {
- dataQualityTaskExecutionContext = new DataQualityTaskExecutionContext();
- setDataQualityTaskRelation(dataQualityTaskExecutionContext, taskInstance, workflowInstance.getTenantCode());
- }
-
- K8sTaskExecutionContext k8sTaskExecutionContext = setK8sTaskRelation(taskInstance);
-
Map<String, Property> businessParamsMap = curingParamsService.preBuildBusinessParams(workflowInstance);
AbstractParameters baseParam = taskPluginManager.getParameters(ParametersNode.builder()
.taskType(taskInstance.getTaskType()).taskParams(taskInstance.getTaskParams()).build());
Map<String, Property> propertyMap =
curingParamsService.paramParsingPreparation(taskInstance, baseParam, workflowInstance);
- return TaskExecutionContextBuilder.get()
+ TaskExecutionContext taskExecutionContext = TaskExecutionContextBuilder.get()
.buildWorkflowInstanceHost(masterConfig.getMasterAddress())
.buildTaskInstanceRelatedInfo(taskInstance)
.buildTaskDefinitionRelatedInfo(taskInstance.getTaskDefine())
.buildProcessInstanceRelatedInfo(taskInstance.getProcessInstance())
.buildProcessDefinitionRelatedInfo(taskInstance.getProcessDefine())
.buildResourceParametersInfo(resources)
- .buildDataQualityTaskExecutionContext(dataQualityTaskExecutionContext)
- .buildK8sTaskRelatedInfo(k8sTaskExecutionContext)
.buildBusinessParamsMap(businessParamsMap)
.buildParamInfo(propertyMap)
.create();
+
+ buildTaskExecutionContext(taskExecutionContext, taskInstance, workflowInstance.getTenantCode());
+ return taskExecutionContext;
+ }
+
+ public void buildTaskExecutionContext(TaskExecutionContext taskExecutionContext, TaskInstance taskInstance,
Review Comment:
Thanks for your suggestion, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433#issuecomment-1614472178
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14433)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_duplicated_lines_density&view=list)
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433#issuecomment-1614475451
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=14433)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=14433&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=14433&metric=new_duplicated_lines_density&view=list)
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx commented on pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433#issuecomment-1619368766
rerun the failed CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] rickchengx merged pull request #14433: [Fix] Fix running task instance throught api gots failed
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx merged PR #14433:
URL: https://github.com/apache/dolphinscheduler/pull/14433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org