You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/07/29 05:14:08 UTC
[GitHub] [dolphinscheduler] kyoty opened a new pull request #5912: [NEW FEATURE][FIX-4385]Complement asks has the ability to configure parallelsim
kyoty opened a new pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912
## Purpose of the pull request
Implements the new feature discussed in #4385
Close #4385
## Brief change log
1.`Serial execution` keep the same.
2.`Parallel execution` ：**Add a checkbox: Configure parallelism. If checked, the parameter of parallelism is required.** Otherwise, the parallelism of the complement task will be 1: 1 with the number of tasks .
+ If enable Configure parallelism. the parameter of parallelism must be a certain positive number.
a empty stror a str contains non-numberic character or a non-positive number would casue a alert like below:
+ If the parallism is a positive number like below, it would run like below:
we assume the parallism is 7
we cam see that 7 process definitions has been created and is running.
Take the process: sql-3-20210729113046224 as an example, we can see that it contains a set of tasks and all of them work as expected.
## Verify this pull request
This pull request is already covered by existing tests and also manually verified the change by testing locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs merged pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
CalvinKirs merged pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #5912: [NEW FEATURE][FIX-4385]Complement add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888818515
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list) [84.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter edited a comment on pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888813979
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5912](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5435920) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/5e343d3fd21ceb2bf141e7b70524b5b3eb5fb87b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e343d3) will **increase** coverage by `1.72%`.
> The diff coverage is `91.66%`.
[](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## dev #5912 +/- ##
============================================
+ Coverage 45.03% 46.76% +1.72%
- Complexity 3656 3745 +89
============================================
Files 606 604 -2
Lines 24876 24826 -50
Branches 2835 2835
============================================
+ Hits 11204 11610 +406
+ Misses 12600 12074 -526
- Partials 1072 1142 +70
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...inscheduler/api/controller/ExecutorController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRXhlY3V0b3JDb250cm9sbGVyLmphdmE=) | `8.33% <ø> (ø)` | |
| [...cheduler/api/service/impl/ExecutorServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9FeGVjdXRvclNlcnZpY2VJbXBsLmphdmE=) | `50.00% <91.66%> (+0.89%)` | :arrow_up: |
| [...olphinscheduler/api/controller/BaseController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvQmFzZUNvbnRyb2xsZXIuamF2YQ==) | `39.43% <0.00%> (-14.32%)` | :arrow_down: |
| [...ache/dolphinscheduler/server/log/LoggerServer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL2xvZy9Mb2dnZXJTZXJ2ZXIuamF2YQ==) | `78.26% <0.00%> (-8.70%)` | :arrow_down: |
| [...scheduler/api/service/impl/ProjectServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9Qcm9qZWN0U2VydmljZUltcGwuamF2YQ==) | `83.33% <0.00%> (-3.58%)` | :arrow_down: |
| [...nscheduler/server/worker/task/spark/SparkTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL3NwYXJrL1NwYXJrVGFzay5qYXZh) | `72.50% <0.00%> (-3.06%)` | :arrow_down: |
| [.../org/apache/dolphinscheduler/api/utils/Result.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL1Jlc3VsdC5qYXZh) | `90.62% <0.00%> (-2.93%)` | :arrow_down: |
| [...er/api/controller/ProcessDefinitionController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvUHJvY2Vzc0RlZmluaXRpb25Db250cm9sbGVyLmphdmE=) | `59.72% <0.00%> (-1.71%)` | :arrow_down: |
| [...dolphinscheduler/remote/future/ResponseFuture.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL2Z1dHVyZS9SZXNwb25zZUZ1dHVyZS5qYXZh) | `81.35% <0.00%> (-1.70%)` | :arrow_down: |
| [...olphinscheduler/plugin/alert/email/MailSender.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hbGVydC1wbHVnaW4vZG9scGhpbnNjaGVkdWxlci1hbGVydC1lbWFpbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9wbHVnaW4vYWxlcnQvZW1haWwvTWFpbFNlbmRlci5qYXZh) | `72.91% <0.00%> (-1.39%)` | :arrow_down: |
| ... and [52 more](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [5e343d3...5435920](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on a change in pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on a change in pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#discussion_r683479472
##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
##########
@@ -553,14 +555,21 @@ private int createCommand(CommandType commandType, int processDefineId,
}
}
if (!CollectionUtils.isEmpty(listDate)) {
- // loop by schedule date
- for (Date date : listDate) {
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(date));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(date));
+
+ int effectThreadsCount = expectedParallelismNumber == null ? 1 : Math.min(listDate.size(), expectedParallelismNumber);
+ logger.info("In parallel mode, current expectedParallelismNumber:{}", expectedParallelismNumber);
+
+ int average = listDate.size() / effectThreadsCount;
+ int slice = listDate.size() % effectThreadsCount == 0 ? average : average + 1;
+
+ Lists.partition(listDate, slice).stream().forEach(partition -> {
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(partition.get(0)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(partition.get(partition.size() - 1)));
command.setCommandParam(JSONUtils.toJsonString(cmdParam));
processService.createCommand(command);
- }
- return listDate.size();
+ });
+
+ return effectThreadsCount;
Review comment:
@lenboo hi,PTAL,thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter edited a comment on pull request #5912: [NEW FEATURE][FIX-4385]Complement add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888813979
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5912](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7d2a079) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/5e343d3fd21ceb2bf141e7b70524b5b3eb5fb87b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e343d3) will **increase** coverage by `1.80%`.
> The diff coverage is `77.77%`.
[](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## dev #5912 +/- ##
============================================
+ Coverage 45.03% 46.84% +1.80%
- Complexity 3656 3745 +89
============================================
Files 606 603 -3
Lines 24876 24799 -77
Branches 2835 2830 -5
============================================
+ Hits 11204 11616 +412
+ Misses 12600 12044 -556
- Partials 1072 1139 +67
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...inscheduler/api/controller/ExecutorController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRXhlY3V0b3JDb250cm9sbGVyLmphdmE=) | `8.33% <ø> (ø)` | |
| [...cheduler/api/service/impl/ExecutorServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9FeGVjdXRvclNlcnZpY2VJbXBsLmphdmE=) | `49.12% <77.77%> (+0.01%)` | :arrow_up: |
| [...nscheduler/server/worker/task/spark/SparkTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL3NwYXJrL1NwYXJrVGFzay5qYXZh) | `72.50% <0.00%> (-3.06%)` | :arrow_down: |
| [...er/server/worker/task/procedure/ProcedureTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL3Byb2NlZHVyZS9Qcm9jZWR1cmVUYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../api/utils/exportprocess/TaskNodeParamFactory.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL2V4cG9ydHByb2Nlc3MvVGFza05vZGVQYXJhbUZhY3RvcnkuamF2YQ==) | | |
| [...duler/api/utils/exportprocess/DataSourceParam.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL2V4cG9ydHByb2Nlc3MvRGF0YVNvdXJjZVBhcmFtLmphdmE=) | | |
| [...eduler/api/utils/exportprocess/DependentParam.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL2V4cG9ydHByb2Nlc3MvRGVwZW5kZW50UGFyYW0uamF2YQ==) | | |
| [...hinscheduler/server/worker/task/http/HttpTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL2h0dHAvSHR0cFRhc2suamF2YQ==) | `2.70% <0.00%> (+0.11%)` | :arrow_up: |
| [...nscheduler/server/worker/task/flink/FlinkTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL2ZsaW5rL0ZsaW5rVGFzay5qYXZh) | `7.31% <0.00%> (+0.79%)` | :arrow_up: |
| [...scheduler/server/worker/task/mr/MapReduceTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL21yL01hcFJlZHVjZVRhc2suamF2YQ==) | `7.89% <0.00%> (+0.91%)` | :arrow_up: |
| ... and [19 more](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [5e343d3...7d2a079](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter edited a comment on pull request #5912: [NEW FEATURE][FIX-4385]Complement add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888813979
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5912](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6790d9d) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/5e343d3fd21ceb2bf141e7b70524b5b3eb5fb87b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e343d3) will **increase** coverage by `1.80%`.
> The diff coverage is `77.77%`.
> :exclamation: Current head 6790d9d differs from pull request most recent head 7d2a079. Consider uploading reports for the commit 7d2a079 to get more accurate results
[](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## dev #5912 +/- ##
============================================
+ Coverage 45.03% 46.84% +1.80%
- Complexity 3656 3746 +90
============================================
Files 606 603 -3
Lines 24876 24799 -77
Branches 2835 2830 -5
============================================
+ Hits 11204 11617 +413
+ Misses 12600 12044 -556
- Partials 1072 1138 +66
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...inscheduler/api/controller/ExecutorController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRXhlY3V0b3JDb250cm9sbGVyLmphdmE=) | `8.33% <ø> (ø)` | |
| [...cheduler/api/service/impl/ExecutorServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9FeGVjdXRvclNlcnZpY2VJbXBsLmphdmE=) | `49.12% <77.77%> (+0.01%)` | :arrow_up: |
| [...nscheduler/server/worker/task/spark/SparkTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL3NwYXJrL1NwYXJrVGFzay5qYXZh) | `72.50% <0.00%> (-3.06%)` | :arrow_down: |
| [...er/server/worker/task/procedure/ProcedureTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL3Byb2NlZHVyZS9Qcm9jZWR1cmVUYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../api/utils/exportprocess/TaskNodeParamFactory.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL2V4cG9ydHByb2Nlc3MvVGFza05vZGVQYXJhbUZhY3RvcnkuamF2YQ==) | | |
| [...eduler/api/utils/exportprocess/DependentParam.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL2V4cG9ydHByb2Nlc3MvRGVwZW5kZW50UGFyYW0uamF2YQ==) | | |
| [...duler/api/utils/exportprocess/DataSourceParam.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL2V4cG9ydHByb2Nlc3MvRGF0YVNvdXJjZVBhcmFtLmphdmE=) | | |
| [...hinscheduler/server/worker/task/http/HttpTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL2h0dHAvSHR0cFRhc2suamF2YQ==) | `2.70% <0.00%> (+0.11%)` | :arrow_up: |
| [...nscheduler/server/worker/task/flink/FlinkTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL2ZsaW5rL0ZsaW5rVGFzay5qYXZh) | `7.31% <0.00%> (+0.79%)` | :arrow_up: |
| [...scheduler/server/worker/task/mr/MapReduceTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL21yL01hcFJlZHVjZVRhc2suamF2YQ==) | `7.89% <0.00%> (+0.91%)` | :arrow_up: |
| ... and [20 more](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [5e343d3...7d2a079](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #5912: [NEW FEATURE][FIX-4385]Complement asks has the ability to configure parallelsim
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888813979
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#5912](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1497268) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/5e343d3fd21ceb2bf141e7b70524b5b3eb5fb87b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5e343d3) will **increase** coverage by `1.79%`.
> The diff coverage is `77.77%`.
> :exclamation: Current head 1497268 differs from pull request most recent head 6790d9d. Consider uploading reports for the commit 6790d9d to get more accurate results
[](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## dev #5912 +/- ##
============================================
+ Coverage 45.03% 46.83% +1.79%
- Complexity 3656 3744 +88
============================================
Files 606 603 -3
Lines 24876 24799 -77
Branches 2835 2830 -5
============================================
+ Hits 11204 11615 +411
+ Misses 12600 12045 -555
- Partials 1072 1139 +67
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...inscheduler/api/controller/ExecutorController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRXhlY3V0b3JDb250cm9sbGVyLmphdmE=) | `8.33% <ø> (ø)` | |
| [...cheduler/api/service/impl/ExecutorServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9FeGVjdXRvclNlcnZpY2VJbXBsLmphdmE=) | `49.12% <77.77%> (+0.01%)` | :arrow_up: |
| [...nscheduler/server/worker/task/spark/SparkTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL3NwYXJrL1NwYXJrVGFzay5qYXZh) | `72.50% <0.00%> (-3.06%)` | :arrow_down: |
| [...er/server/worker/task/procedure/ProcedureTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL3Byb2NlZHVyZS9Qcm9jZWR1cmVUYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...duler/api/utils/exportprocess/DataSourceParam.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL2V4cG9ydHByb2Nlc3MvRGF0YVNvdXJjZVBhcmFtLmphdmE=) | | |
| [.../api/utils/exportprocess/TaskNodeParamFactory.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL2V4cG9ydHByb2Nlc3MvVGFza05vZGVQYXJhbUZhY3RvcnkuamF2YQ==) | | |
| [...eduler/api/utils/exportprocess/DependentParam.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3V0aWxzL2V4cG9ydHByb2Nlc3MvRGVwZW5kZW50UGFyYW0uamF2YQ==) | | |
| [...hinscheduler/server/worker/task/http/HttpTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL2h0dHAvSHR0cFRhc2suamF2YQ==) | `2.70% <0.00%> (+0.11%)` | :arrow_up: |
| [...nscheduler/server/worker/task/flink/FlinkTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL2ZsaW5rL0ZsaW5rVGFzay5qYXZh) | `7.31% <0.00%> (+0.79%)` | :arrow_up: |
| [...scheduler/server/worker/task/mr/MapReduceTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL3dvcmtlci90YXNrL21yL01hcFJlZHVjZVRhc2suamF2YQ==) | `7.89% <0.00%> (+0.91%)` | :arrow_up: |
| ... and [19 more](https://codecov.io/gh/apache/dolphinscheduler/pull/5912/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [5e343d3...6790d9d](https://codecov.io/gh/apache/dolphinscheduler/pull/5912?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhuangchong commented on a change in pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
zhuangchong commented on a change in pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#discussion_r683967092
##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
##########
@@ -553,14 +555,21 @@ private int createCommand(CommandType commandType, int processDefineId,
}
}
if (!CollectionUtils.isEmpty(listDate)) {
- // loop by schedule date
- for (Date date : listDate) {
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(date));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(date));
+
+ int effectThreadsCount = expectedParallelismNumber == null ? 1 : Math.min(listDate.size(), expectedParallelismNumber);
+ logger.info("In parallel mode, current expectedParallelismNumber:{}", expectedParallelismNumber);
+
+ int average = listDate.size() / effectThreadsCount;
+ int slice = listDate.size() % effectThreadsCount == 0 ? average : average + 1;
+
+ Lists.partition(listDate, slice).stream().forEach(partition -> {
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(partition.get(0)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(partition.get(partition.size() - 1)));
Review comment:
With the current change, I think the last mission cannot be executed.
```
List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionId(processDefinitionId);
List<Date> listDate = Lists.newLinkedList();
if (!CollectionUtils.isEmpty(schedules)) {
for (Schedule schedule : schedules) {
listDate.addAll(CronUtils.getSelfFireDateList(startDate, endDate, schedule.getCrontab()));
}
}
// get first fire date
Iterator<Date> iterator = null;
Date scheduleDate;
if (!CollectionUtils.isEmpty(listDate)) {
iterator = listDate.iterator();
scheduleDate = iterator.next();
processInstance.setScheduleTime(scheduleDate);
processService.updateProcessInstance(processInstance);
} else {
scheduleDate = processInstance.getScheduleTime();
if (scheduleDate == null) {
scheduleDate = startDate;
}
}
```
[executeComplementProcess](https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java)
```
public static List<Date> getSelfFireDateList(Date startTime, Date endTime, CronExpression cronExpression) {
List<Date> dateList = new ArrayList<>();
while (Stopper.isRunning()) {
startTime = cronExpression.getNextValidTimeAfter(startTime);
if (startTime.after(endTime) || startTime.equals(endTime)) {
break;
}
dateList.add(startTime);
}
return dateList;
}
```
[getSelfFireDateList](https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/quartz/cron/CronUtils.java)
I think the condition judgment of startTime.equals(endTime) should be removed in the getSelfFireDateList method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #5912: [NEW FEATURE][FIX-4385]Complement add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888819590
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list) [84.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #5912: [NEW FEATURE][FIX-4385]Complement add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888818515
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list) [84.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kyoty commented on a change in pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
kyoty commented on a change in pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#discussion_r684576386
##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
##########
@@ -553,14 +555,21 @@ private int createCommand(CommandType commandType, int processDefineId,
}
}
if (!CollectionUtils.isEmpty(listDate)) {
- // loop by schedule date
- for (Date date : listDate) {
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(date));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(date));
+
+ int effectThreadsCount = expectedParallelismNumber == null ? 1 : Math.min(listDate.size(), expectedParallelismNumber);
+ logger.info("In parallel mode, current expectedParallelismNumber:{}", expectedParallelismNumber);
+
+ int average = listDate.size() / effectThreadsCount;
+ int slice = listDate.size() % effectThreadsCount == 0 ? average : average + 1;
+
+ Lists.partition(listDate, slice).stream().forEach(partition -> {
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(partition.get(0)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(partition.get(partition.size() - 1)));
Review comment:
Thanks @zhuangchong.
In fact, when startTime is same as endTime, the listDate would be empty, and then the code would run into the [ExecutorServiceImpl.java#L567](https://github.com/apache/dolphinscheduler/blob/f5675170311ccc3cc8e6f88e74c956220297e8cd/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java#L567) like below:
```java
else {
// loop by day
int runCunt = 0;
while (!start.after(end)) {
runCunt += 1;
cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(start));
cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(start));
command.setCommandParam(JSONUtils.toJsonString(cmdParam));
processService.createCommand(command);
start = DateUtils.getSomeDay(start, 1);
}
return runCunt;
}
```
the runCunt would be 1.
The part I updated will not affect this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kyoty commented on a change in pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
kyoty commented on a change in pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#discussion_r689212950
##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
##########
@@ -553,14 +555,21 @@ private int createCommand(CommandType commandType, int processDefineId,
}
}
if (!CollectionUtils.isEmpty(listDate)) {
- // loop by schedule date
- for (Date date : listDate) {
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(date));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(date));
+
+ int effectThreadsCount = expectedParallelismNumber == null ? 1 : Math.min(listDate.size(), expectedParallelismNumber);
+ logger.info("In parallel mode, current expectedParallelismNumber:{}", expectedParallelismNumber);
+
+ int average = listDate.size() / effectThreadsCount;
+ int slice = listDate.size() % effectThreadsCount == 0 ? average : average + 1;
+
+ Lists.partition(listDate, slice).stream().forEach(partition -> {
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(partition.get(0)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(partition.get(partition.size() - 1)));
Review comment:
@zhuangchong Sorry for the delay.
This part of the code is very complicated.
First of all, we must know that: **`CronUtils#getSelfFireDateList` will only execute with the time points that satisfy cronExpression in the open time interval (startTime, endTime) .**
Then there are currently two complements, all needs to be taken into consideration:
1. Serial complement, `CronUtils#getSelfFireDateList` will only be executed once in `MasterExecThread#executeComplementProcess`.
2. Parallel complement, `CronUtils#getSelfFireDateList` will be executed twice: `ExecutorServiceImpl#createCommand` and `MasterExecThread#executeComplementProcess`, **executed twice will cause the `startTime` and `endTime` of each prepared-execute timeInterval in custom-parallel mode to be discarded**.
**I dare not modify `CronUtils.getSelfFireDateList`, because modifying it will also affect the serial complement case.**
**In order to be compatible with all scenarios at the same time, I modified the time range split in parallel mode, such as the prepared-execute timeInterval is [A,B], but we would split it into [A-1, B+1]. In this way, after executing `CronUtils.getSelfFireDateList` twice, we would finally got the correct interval.**
Now I've implemented it as I described above, and the manual test works well.
Do you have any ieda?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #5912: [NEW FEATURE][FIX-4385]Complement add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888815778
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list) [84.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kyoty commented on a change in pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
kyoty commented on a change in pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#discussion_r689212950
##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
##########
@@ -553,14 +555,21 @@ private int createCommand(CommandType commandType, int processDefineId,
}
}
if (!CollectionUtils.isEmpty(listDate)) {
- // loop by schedule date
- for (Date date : listDate) {
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(date));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(date));
+
+ int effectThreadsCount = expectedParallelismNumber == null ? 1 : Math.min(listDate.size(), expectedParallelismNumber);
+ logger.info("In parallel mode, current expectedParallelismNumber:{}", expectedParallelismNumber);
+
+ int average = listDate.size() / effectThreadsCount;
+ int slice = listDate.size() % effectThreadsCount == 0 ? average : average + 1;
+
+ Lists.partition(listDate, slice).stream().forEach(partition -> {
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(partition.get(0)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(partition.get(partition.size() - 1)));
Review comment:
@zhuangchong Sorry for the delay.
This part of the code is very complicated.
First of all, we must know that: **`CronUtils#getSelfFireDateList` will only execute with the time points that satisfy cronExpression in the open time interval (startTime, endTime) .**
Then there are currently two complements, all needs to be taken into consideration:
1. Serial complement, `CronUtils#getSelfFireDateList` will only be executed once in `MasterExecThread#executeComplementProcess`.
2. Parallel complement, `CronUtils#getSelfFireDateList` will be executed twice: `ExecutorServiceImpl#createCommand` and `MasterExecThread#executeComplementProcess`, **executed twice will cause the `startTime` and `endTime` of each prepared-execute timeInterval in custom-parallel mode to be discarded**.
**I dare not modify `CronUtils.getSelfFireDateList`, because modifying it will also affect the serial complement case.**
**In order to be compatible with all scenarios at the same time, I modified the time range split in parallel mode, such as the prepared-execute timeInterval is [A,B], but we would split it into [A-1, B+1]. In this way, after executing `CronUtils.getSelfFireDateList` twice, we would finally got the correct interval.**
Now I've implemented it as I described above, and the manual test works well.
Do you have any idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #5912: [NEW FEATURE][FIX-4385]Complement add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888815778
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list) [84.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-899178097
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list) [95.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kyoty commented on a change in pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
kyoty commented on a change in pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#discussion_r689212950
##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
##########
@@ -553,14 +555,21 @@ private int createCommand(CommandType commandType, int processDefineId,
}
}
if (!CollectionUtils.isEmpty(listDate)) {
- // loop by schedule date
- for (Date date : listDate) {
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(date));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(date));
+
+ int effectThreadsCount = expectedParallelismNumber == null ? 1 : Math.min(listDate.size(), expectedParallelismNumber);
+ logger.info("In parallel mode, current expectedParallelismNumber:{}", expectedParallelismNumber);
+
+ int average = listDate.size() / effectThreadsCount;
+ int slice = listDate.size() % effectThreadsCount == 0 ? average : average + 1;
+
+ Lists.partition(listDate, slice).stream().forEach(partition -> {
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(partition.get(0)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(partition.get(partition.size() - 1)));
Review comment:
@zhuangchong Sorry for the delay.
This part of the code is a little Confusing.
First of all, we must know that: **`CronUtils#getSelfFireDateList` will only execute with the time points that satisfy cronExpression in the open time interval (startTime, endTime) .**
Then there are currently two complements, all needs to be taken into consideration:
1. Serial complement, `CronUtils#getSelfFireDateList` will only be executed once in `MasterExecThread#executeComplementProcess`.
2. Parallel complement, `CronUtils#getSelfFireDateList` will be executed twice: `ExecutorServiceImpl#createCommand` and `MasterExecThread#executeComplementProcess`, **executed twice will cause the `startTime` and `endTime` of each prepared-execute timeInterval in custom-parallel mode to be discarded**.
**I dare not modify `CronUtils.getSelfFireDateList`, because modifying it will also affect the serial complement case.**
**In order to be compatible with all scenarios at the same time, I modified the time range split in parallel mode, such as the prepared-execute timeInterval is [A,B], but we would split it into [A-1, B+1]. In this way, after executing `CronUtils.getSelfFireDateList` twice, we would finally got the correct interval.**
Now I've implemented it as I described above, and the manual test works well.
Do you have any idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhuangchong commented on a change in pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
zhuangchong commented on a change in pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#discussion_r684865451
##########
File path: dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
##########
@@ -553,14 +555,21 @@ private int createCommand(CommandType commandType, int processDefineId,
}
}
if (!CollectionUtils.isEmpty(listDate)) {
- // loop by schedule date
- for (Date date : listDate) {
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(date));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(date));
+
+ int effectThreadsCount = expectedParallelismNumber == null ? 1 : Math.min(listDate.size(), expectedParallelismNumber);
+ logger.info("In parallel mode, current expectedParallelismNumber:{}", expectedParallelismNumber);
+
+ int average = listDate.size() / effectThreadsCount;
+ int slice = listDate.size() % effectThreadsCount == 0 ? average : average + 1;
+
+ Lists.partition(listDate, slice).stream().forEach(partition -> {
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(partition.get(0)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(partition.get(partition.size() - 1)));
Review comment:
@kyoty
I feel that we are not talking about a part.
What you describe is the judgment operation of the first complement generation task
What I am expressing is the execution operation after generating the complement task,
In the complement parallel task, multiple serial tasks are generated according to the custom parallelism. Each serial task has a start time and an end time (the start time of the nth time). In this serial task, when the loop is the nth time If the start time is equal to the end time, it will not be executed this time.
My understanding may be wrong ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #5912: [NEW FEATURE][FIX-4385] compensation task add the ability to configure parallelism
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #5912:
URL: https://github.com/apache/dolphinscheduler/pull/5912#issuecomment-888819590
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=5912&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list) [84.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=5912&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org