You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/06/08 16:41:52 UTC
[GitHub] [dolphinscheduler] hstdream opened a new pull request, #10376: [Improvement] [Api & Master] Complement function transformation
hstdream opened a new pull request, #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376
#10370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894116015
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -18,13 +18,9 @@
package org.apache.dolphinscheduler.api.service.impl;
import static org.apache.dolphinscheduler.api.constants.ApiFuncIdentificationConstant.WORKFLOW_START;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_END_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_START_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_START_NODES;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_START_PARAMS;
-import static org.apache.dolphinscheduler.common.Constants.MAX_TASK_TIMEOUT;
+import static org.apache.dolphinscheduler.common.Constants.*;
Review Comment:
OK，Modified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894184453
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/corn/CronUtils.java:
##########
@@ -35,13 +34,9 @@
import org.apache.commons.collections.CollectionUtils;
import java.text.ParseException;
-import java.util.ArrayList;
-import java.util.Calendar;
-import java.util.Collections;
-import java.util.Date;
-import java.util.GregorianCalendar;
-import java.util.List;
+import java.util.*;
Review Comment:
This has not been addressed.
##########
dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/WorkflowE2ETest.java:
##########
@@ -162,56 +162,56 @@ void testCreateSubWorkflow() {
workflowDefinitionPage.publish(workflow);
}
- @Test
- @Order(30)
- void testRunWorkflow() {
- final String workflow = "test-workflow-1";
- final ProjectDetailPage projectPage =
- new ProjectPage(browser)
- .goToNav(ProjectPage.class)
- .goTo(project);
-
- projectPage
- .goToTab(WorkflowInstanceTab.class)
- .deleteAll();
- projectPage
- .goToTab(WorkflowDefinitionTab.class)
- .run(workflow)
- .submit();
-
- await().untilAsserted(() -> {
- browser.navigate().refresh();
-
- final Row row = projectPage
- .goToTab(WorkflowInstanceTab.class)
- .instances()
- .iterator()
- .next();
-
- assertThat(row.isSuccess()).isTrue();
- assertThat(row.executionTime()).isEqualTo(1);
- });
- // Test rerun
- projectPage
- .goToTab(WorkflowInstanceTab.class)
- .instances()
- .stream()
- .filter(it -> it.rerunButton().isDisplayed())
- .iterator()
- .next()
- .rerun();
-
- await().untilAsserted(() -> {
- browser.navigate().refresh();
-
- final Row row = projectPage
- .goToTab(WorkflowInstanceTab.class)
- .instances()
- .iterator()
- .next();
-
- assertThat(row.isSuccess()).isTrue();
- assertThat(row.executionTime()).isEqualTo(2);
- });
- }
+// @Test
+// @Order(30)
+// void testRunWorkflow() {
+// final String workflow = "test-workflow-1";
+// final ProjectDetailPage projectPage =
+// new ProjectPage(browser)
+// .goToNav(ProjectPage.class)
+// .goTo(project);
+//
+// projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .deleteAll();
+// projectPage
+// .goToTab(WorkflowDefinitionTab.class)
+// .run(workflow)
+// .submit();
+//
+// await().untilAsserted(() -> {
+// browser.navigate().refresh();
+//
+// final Row row = projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .instances()
+// .iterator()
+// .next();
+//
+// assertThat(row.isSuccess()).isTrue();
+// assertThat(row.executionTime()).isEqualTo(1);
+// });
+// // Test rerun
+// projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .instances()
+// .stream()
+// .filter(it -> it.rerunButton().isDisplayed())
+// .iterator()
+// .next()
+// .rerun();
+//
+// await().untilAsserted(() -> {
+// browser.navigate().refresh();
+//
+// final Row row = projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .instances()
+// .iterator()
+// .next();
+//
+// assertThat(row.isSuccess()).isTrue();
+// assertThat(row.executionTime()).isEqualTo(2);
+// });
+// }
Review Comment:
Why did you comment out this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152376548
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r893037788
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java:
##########
@@ -943,15 +938,12 @@ private void initTaskQueue() {
if (processInstance.isComplementData() && complementListDate.size() == 0) {
Map<String, String> cmdParam = JSONUtils.toMap(processInstance.getCommandParam());
- if (cmdParam != null && cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)) {
+ if (cmdParam != null && cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE)) {
// reset global params while there are start parameters
setGlobalParamIfCommanded(processDefinition, cmdParam);
- Date start = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE));
- Date end = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE));
- List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(processInstance.getProcessDefinitionCode());
if (complementListDate.size() == 0 && needComplementProcess()) {
- complementListDate = CronUtils.getSelfFireDateList(start, end, schedules);
Review Comment:
As we discuss in wechat. I do not think this is a good idea to remove such a big feature of complement data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894116132
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -740,45 +729,24 @@ protected int createComplementCommandList(Date start, Date end, RunMode runMode,
}
}
logger.info("In parallel mode, current expectedParallelismNumber:{}", createCount);
-
// Distribute the number of tasks equally to each command.
// The last command with insufficient quantity will be assigned to the remaining tasks.
- int itemsPerCommand = (listDateSize / createCount);
- int remainingItems = (listDateSize % createCount);
- int startDateIndex = 0;
- int endDateIndex = 0;
-
- for (int i = 1; i <= createCount; i++) {
- int extra = (i <= remainingItems) ? 1 : 0;
- int singleCommandItems = (itemsPerCommand + extra);
-
- if (i == 1) {
- endDateIndex += singleCommandItems - 1;
- } else {
- startDateIndex = endDateIndex + 1;
- endDateIndex += singleCommandItems;
+ for(List<String> stringDate : Lists.partition(listDate,createCount)){
+ String tempDate = "";
+ for(String date : stringDate){
+ tempDate += date +",";
}
-
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(listDate.get(startDateIndex)));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(listDate.get(endDateIndex)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE, tempDate.substring(0,tempDate.length() - 1));
command.setCommandParam(JSONUtils.toJsonString(cmdParam));
processService.createCommand(command);
-
- if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
- logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
- + "dependent complement data", command.getProcessDefinitionCode());
- } else {
- dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
- }
Review Comment:
OK，Modified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152479511
> LGTM overall. But we still have discussion in [#10376 (comment)](https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894520852)
Hi @SbloodyS you should give a approve because the change requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152890998
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [37.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152325520
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r893015000
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java:
##########
@@ -17,12 +17,7 @@
package org.apache.dolphinscheduler.server.master.runner;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_END_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_START_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_RECOVERY_START_NODE_STRING;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_START_NODES;
-import static org.apache.dolphinscheduler.common.Constants.DEFAULT_WORKER_GROUP;
+import static org.apache.dolphinscheduler.common.Constants.*;
Review Comment:
Same here.
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -75,13 +71,7 @@
import org.apache.commons.collections.MapUtils;
import org.apache.commons.lang3.StringUtils;
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
Review Comment:
Same here.
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -740,45 +729,24 @@ protected int createComplementCommandList(Date start, Date end, RunMode runMode,
}
}
logger.info("In parallel mode, current expectedParallelismNumber:{}", createCount);
-
// Distribute the number of tasks equally to each command.
// The last command with insufficient quantity will be assigned to the remaining tasks.
- int itemsPerCommand = (listDateSize / createCount);
- int remainingItems = (listDateSize % createCount);
- int startDateIndex = 0;
- int endDateIndex = 0;
-
- for (int i = 1; i <= createCount; i++) {
- int extra = (i <= remainingItems) ? 1 : 0;
- int singleCommandItems = (itemsPerCommand + extra);
-
- if (i == 1) {
- endDateIndex += singleCommandItems - 1;
- } else {
- startDateIndex = endDateIndex + 1;
- endDateIndex += singleCommandItems;
+ for(List<String> stringDate : Lists.partition(listDate,createCount)){
+ String tempDate = "";
+ for(String date : stringDate){
+ tempDate += date +",";
Review Comment:
It's recommanded to format this line.
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -18,13 +18,9 @@
package org.apache.dolphinscheduler.api.service.impl;
import static org.apache.dolphinscheduler.api.constants.ApiFuncIdentificationConstant.WORKFLOW_START;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_END_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_START_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_START_NODES;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_START_PARAMS;
-import static org.apache.dolphinscheduler.common.Constants.MAX_TASK_TIMEOUT;
+import static org.apache.dolphinscheduler.common.Constants.*;
Review Comment:
Please avoid import *
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -740,45 +729,24 @@ protected int createComplementCommandList(Date start, Date end, RunMode runMode,
}
}
logger.info("In parallel mode, current expectedParallelismNumber:{}", createCount);
-
// Distribute the number of tasks equally to each command.
// The last command with insufficient quantity will be assigned to the remaining tasks.
- int itemsPerCommand = (listDateSize / createCount);
- int remainingItems = (listDateSize % createCount);
- int startDateIndex = 0;
- int endDateIndex = 0;
-
- for (int i = 1; i <= createCount; i++) {
- int extra = (i <= remainingItems) ? 1 : 0;
- int singleCommandItems = (itemsPerCommand + extra);
-
- if (i == 1) {
- endDateIndex += singleCommandItems - 1;
- } else {
- startDateIndex = endDateIndex + 1;
- endDateIndex += singleCommandItems;
+ for(List<String> stringDate : Lists.partition(listDate,createCount)){
+ String tempDate = "";
+ for(String date : stringDate){
+ tempDate += date +",";
}
-
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(listDate.get(startDateIndex)));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(listDate.get(endDateIndex)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE, tempDate.substring(0,tempDate.length() - 1));
command.setCommandParam(JSONUtils.toJsonString(cmdParam));
processService.createCommand(command);
-
- if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
- logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
- + "dependent complement data", command.getProcessDefinitionCode());
- } else {
- dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
- }
Review Comment:
Why did you remove this? It may cause complement dependent process error.
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/corn/CronUtils.java:
##########
@@ -35,13 +34,9 @@
import org.apache.commons.collections.CollectionUtils;
import java.text.ParseException;
-import java.util.ArrayList;
-import java.util.Calendar;
-import java.util.Collections;
-import java.util.Date;
-import java.util.GregorianCalendar;
-import java.util.List;
+import java.util.*;
Review Comment:
Same here.
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -740,45 +729,24 @@ protected int createComplementCommandList(Date start, Date end, RunMode runMode,
}
}
logger.info("In parallel mode, current expectedParallelismNumber:{}", createCount);
-
// Distribute the number of tasks equally to each command.
// The last command with insufficient quantity will be assigned to the remaining tasks.
- int itemsPerCommand = (listDateSize / createCount);
- int remainingItems = (listDateSize % createCount);
- int startDateIndex = 0;
- int endDateIndex = 0;
-
- for (int i = 1; i <= createCount; i++) {
- int extra = (i <= remainingItems) ? 1 : 0;
- int singleCommandItems = (itemsPerCommand + extra);
-
- if (i == 1) {
- endDateIndex += singleCommandItems - 1;
- } else {
- startDateIndex = endDateIndex + 1;
- endDateIndex += singleCommandItems;
+ for(List<String> stringDate : Lists.partition(listDate,createCount)){
+ String tempDate = "";
+ for(String date : stringDate){
+ tempDate += date +",";
}
-
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(listDate.get(startDateIndex)));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(listDate.get(endDateIndex)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE, tempDate.substring(0,tempDate.length() - 1));
Review Comment:
I think using it's a little confusing using ```substring``` to remove the last comma. Is it more elegant to use ```String.join(",", tmpDateList)``` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1150168268
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [3.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1151761085
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10376?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10376](https://codecov.io/gh/apache/dolphinscheduler/pull/10376?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d695694) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/9782fe4e7cd0c9313c748bb45b5b1f2d1ddd3367?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9782fe4) will **decrease** coverage by `0.03%`.
> The diff coverage is `32.09%`.
```diff
@@ Coverage Diff @@
## dev #10376 +/- ##
============================================
- Coverage 40.52% 40.48% -0.04%
- Complexity 4759 4762 +3
============================================
Files 877 877
Lines 35607 35676 +69
Branches 3945 3970 +25
============================================
+ Hits 14430 14444 +14
- Misses 19744 19786 +42
- Partials 1433 1446 +13
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10376?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...inscheduler/api/controller/ExecutorController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRXhlY3V0b3JDb250cm9sbGVyLmphdmE=) | `28.30% <ø> (ø)` | |
| [.../org/apache/dolphinscheduler/common/Constants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL0NvbnN0YW50cy5qYXZh) | `80.95% <ø> (ø)` | |
| [.../server/master/runner/WorkflowExecuteRunnable.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvV29ya2Zsb3dFeGVjdXRlUnVubmFibGUuamF2YQ==) | `7.75% <0.00%> (-0.07%)` | :arrow_down: |
| [...pache/dolphinscheduler/service/corn/CronUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2UvY29ybi9Dcm9uVXRpbHMuamF2YQ==) | `64.77% <0.00%> (-5.60%)` | :arrow_down: |
| [...cheduler/api/service/impl/ExecutorServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9FeGVjdXRvclNlcnZpY2VJbXBsLmphdmE=) | `40.10% <48.54%> (-2.58%)` | :arrow_down: |
| [.../org/apache/dolphinscheduler/api/enums/Status.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2VudW1zL1N0YXR1cy5qYXZh) | `100.00% <100.00%> (ø)` | |
| [...er/master/dispatch/host/assign/RandomSelector.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9ob3N0L2Fzc2lnbi9SYW5kb21TZWxlY3Rvci5qYXZh) | `77.77% <0.00%> (-5.56%)` | :arrow_down: |
| [...r/plugin/task/sqoop/parameter/SqoopParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Fvb3Avc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3Fvb3AvcGFyYW1ldGVyL1Nxb29wUGFyYW1ldGVycy5qYXZh) | `53.33% <0.00%> (ø)` | |
| [...dolphinscheduler/remote/future/ResponseFuture.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL2Z1dHVyZS9SZXNwb25zZUZ1dHVyZS5qYXZh) | `83.60% <0.00%> (+1.63%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10376?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10376?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [9782fe4...d695694](https://codecov.io/gh/apache/dolphinscheduler/pull/10376?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894407131
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java:
##########
@@ -88,6 +88,9 @@ public class ExecutorController extends BaseController {
* @param projectCode project code
* @param processDefinitionCode process definition code
* @param scheduleTime schedule time
+ * when CommandType is COMPLEMENT_DATA there are two ways to transfer parameters
Review Comment:
If the description is long, do you need to wrap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152846342
> > > LGTM overall. But we still have discussion in [#10376 (comment)](https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894520852)
> >
> >
> > Hi @SbloodyS you should give a approve because the change requested.
>
> Sorry. I don't have strong opinion here. I just not quite sure about whether should give a approve since we still have unresolved discussions.
>
> I gave approved. Do you have better suggestions? @zhongjiajie @caishunfeng
Sorry, I got it wrong, thanks for pointing it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1153052767
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [37.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894516480
##########
docs/docs/en/about/glossary.md:
##########
@@ -45,7 +45,7 @@ provided. **Continue** refers to regardless of the status of the task running in
failure. **End** means that once a failed task is found, Kill will also run the parallel task at the same time, and the
process fails and ends
-**Complement**: Supplement historical data，Supports **interval parallel and serial** two complement methods
+**Complement**: Supplement historical data，supports **interval parallel and serial** two complement methods, and two types of date selection which include **date range** and **date enumeration **.
Review Comment:
```suggestion
**Complement**: Supplement historical data，supports **interval parallel and serial** two complement methods, and two types of date selection which include **date range** and **date enumeration**.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894581117
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -75,13 +71,7 @@
import org.apache.commons.collections.MapUtils;
import org.apache.commons.lang3.StringUtils;
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
Review Comment:
> why our codesytle check can not catch this?
I'm also confusing about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r893041423
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java:
##########
@@ -407,6 +407,8 @@ public enum Status {
NO_CURRENT_OPERATING_PERMISSION(1400001, "The current user does not have this permission.", "当前用户无此权限"),
FUNCTION_DISABLED(1400002, "The current feature is disabled.", "当前功能已被禁用"),
+ SCHEDULE_TIME_NUMBER(1400003, "The number of complement dates exceed 100.", "补数日期个数超过100"),
Review Comment:
I think we should not limit the date range of user complement. In the actual production environment, it is very common to supplement the time range within one or two years.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894186901
##########
dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/WorkflowE2ETest.java:
##########
@@ -162,56 +162,56 @@ void testCreateSubWorkflow() {
workflowDefinitionPage.publish(workflow);
}
- @Test
- @Order(30)
- void testRunWorkflow() {
- final String workflow = "test-workflow-1";
- final ProjectDetailPage projectPage =
- new ProjectPage(browser)
- .goToNav(ProjectPage.class)
- .goTo(project);
-
- projectPage
- .goToTab(WorkflowInstanceTab.class)
- .deleteAll();
- projectPage
- .goToTab(WorkflowDefinitionTab.class)
- .run(workflow)
- .submit();
-
- await().untilAsserted(() -> {
- browser.navigate().refresh();
-
- final Row row = projectPage
- .goToTab(WorkflowInstanceTab.class)
- .instances()
- .iterator()
- .next();
-
- assertThat(row.isSuccess()).isTrue();
- assertThat(row.executionTime()).isEqualTo(1);
- });
- // Test rerun
- projectPage
- .goToTab(WorkflowInstanceTab.class)
- .instances()
- .stream()
- .filter(it -> it.rerunButton().isDisplayed())
- .iterator()
- .next()
- .rerun();
-
- await().untilAsserted(() -> {
- browser.navigate().refresh();
-
- final Row row = projectPage
- .goToTab(WorkflowInstanceTab.class)
- .instances()
- .iterator()
- .next();
-
- assertThat(row.isSuccess()).isTrue();
- assertThat(row.executionTime()).isEqualTo(2);
- });
- }
+// @Test
+// @Order(30)
+// void testRunWorkflow() {
+// final String workflow = "test-workflow-1";
+// final ProjectDetailPage projectPage =
+// new ProjectPage(browser)
+// .goToNav(ProjectPage.class)
+// .goTo(project);
+//
+// projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .deleteAll();
+// projectPage
+// .goToTab(WorkflowDefinitionTab.class)
+// .run(workflow)
+// .submit();
+//
+// await().untilAsserted(() -> {
+// browser.navigate().refresh();
+//
+// final Row row = projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .instances()
+// .iterator()
+// .next();
+//
+// assertThat(row.isSuccess()).isTrue();
+// assertThat(row.executionTime()).isEqualTo(1);
+// });
+// // Test rerun
+// projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .instances()
+// .stream()
+// .filter(it -> it.rerunButton().isDisplayed())
+// .iterator()
+// .next()
+// .rerun();
+//
+// await().untilAsserted(() -> {
+// browser.navigate().refresh();
+//
+// final Row row = projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .instances()
+// .iterator()
+// .next();
+//
+// assertThat(row.isSuccess()).isTrue();
+// assertThat(row.executionTime()).isEqualTo(2);
+// });
+// }
Review Comment:
Because the input parameters of runworkflow are modified, the E2E is shielded for the time being. After the front-end code is merged, it is opened again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894187995
##########
dolphinscheduler-e2e/dolphinscheduler-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/WorkflowE2ETest.java:
##########
@@ -162,56 +162,56 @@ void testCreateSubWorkflow() {
workflowDefinitionPage.publish(workflow);
}
- @Test
- @Order(30)
- void testRunWorkflow() {
- final String workflow = "test-workflow-1";
- final ProjectDetailPage projectPage =
- new ProjectPage(browser)
- .goToNav(ProjectPage.class)
- .goTo(project);
-
- projectPage
- .goToTab(WorkflowInstanceTab.class)
- .deleteAll();
- projectPage
- .goToTab(WorkflowDefinitionTab.class)
- .run(workflow)
- .submit();
-
- await().untilAsserted(() -> {
- browser.navigate().refresh();
-
- final Row row = projectPage
- .goToTab(WorkflowInstanceTab.class)
- .instances()
- .iterator()
- .next();
-
- assertThat(row.isSuccess()).isTrue();
- assertThat(row.executionTime()).isEqualTo(1);
- });
- // Test rerun
- projectPage
- .goToTab(WorkflowInstanceTab.class)
- .instances()
- .stream()
- .filter(it -> it.rerunButton().isDisplayed())
- .iterator()
- .next()
- .rerun();
-
- await().untilAsserted(() -> {
- browser.navigate().refresh();
-
- final Row row = projectPage
- .goToTab(WorkflowInstanceTab.class)
- .instances()
- .iterator()
- .next();
-
- assertThat(row.isSuccess()).isTrue();
- assertThat(row.executionTime()).isEqualTo(2);
- });
- }
+// @Test
+// @Order(30)
+// void testRunWorkflow() {
+// final String workflow = "test-workflow-1";
+// final ProjectDetailPage projectPage =
+// new ProjectPage(browser)
+// .goToNav(ProjectPage.class)
+// .goTo(project);
+//
+// projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .deleteAll();
+// projectPage
+// .goToTab(WorkflowDefinitionTab.class)
+// .run(workflow)
+// .submit();
+//
+// await().untilAsserted(() -> {
+// browser.navigate().refresh();
+//
+// final Row row = projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .instances()
+// .iterator()
+// .next();
+//
+// assertThat(row.isSuccess()).isTrue();
+// assertThat(row.executionTime()).isEqualTo(1);
+// });
+// // Test rerun
+// projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .instances()
+// .stream()
+// .filter(it -> it.rerunButton().isDisplayed())
+// .iterator()
+// .next()
+// .rerun();
+//
+// await().untilAsserted(() -> {
+// browser.navigate().refresh();
+//
+// final Row row = projectPage
+// .goToTab(WorkflowInstanceTab.class)
+// .instances()
+// .iterator()
+// .next();
+//
+// assertThat(row.isSuccess()).isTrue();
+// assertThat(row.executionTime()).isEqualTo(2);
+// });
+// }
Review Comment:
Currently our E2E test does not cover the timing module and the complement module. I think we should not change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1151706839
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [62.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894390336
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -228,6 +233,54 @@ private boolean checkMasterExists(Map<String, Object> result) {
return true;
}
+ /**
+ *
+ * @param complementData
+ * @param cronTime
+ * @return CommandType is COMPLEMENT_DATA and cronTime's number is not greater than return true , otherwise return false
Review Comment:
Missing greater than 100.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894514205
##########
docs/docs/zh/about/glossary.md:
##########
@@ -30,7 +30,7 @@
**失败策略**：对于并行运行的任务，如果有任务失败，提供两种失败策略处理方式，**继续**是指不管并行运行任务的状态，直到流程失败结束。**结束**是指一旦发现失败任务，则同时Kill掉正在运行的并行任务，流程失败结束
-**补数**：补历史数据，支持**区间并行和串行**两种补数方式
+**补数**：补历史数据，支持**区间并行和串行**两种补数方式，并且支持**选择日期范围和手动输入日期**两种日期方式
Review Comment:
```suggestion
**补数**：补历史数据，支持**区间并行和串行**两种补数方式，其日期选择方式包括**日期范围**和**日期枚举**两种
```
##########
docs/docs/en/about/glossary.md:
##########
@@ -45,7 +45,7 @@ provided. **Continue** refers to regardless of the status of the task running in
failure. **End** means that once a failed task is found, Kill will also run the parallel task at the same time, and the
process fails and ends
-**Complement**: Supplement historical data，Supports **interval parallel and serial** two complement methods
+**Complement**: Supplement historical data，Supports **interval parallel and serial** two complement methods, and supports **select date range and manually enter date** two date methods
Review Comment:
```suggestion
**Complement**: Supplement historical data，supports **interval parallel and serial** two complement methods, and two types of date selection which include **date range** and **date enumeration **.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152387591
docs/docs/zh/guide/project/workflow-definition.md this docs supplement after the front-end code is updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894116483
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -740,45 +729,24 @@ protected int createComplementCommandList(Date start, Date end, RunMode runMode,
}
}
logger.info("In parallel mode, current expectedParallelismNumber:{}", createCount);
-
// Distribute the number of tasks equally to each command.
// The last command with insufficient quantity will be assigned to the remaining tasks.
- int itemsPerCommand = (listDateSize / createCount);
- int remainingItems = (listDateSize % createCount);
- int startDateIndex = 0;
- int endDateIndex = 0;
-
- for (int i = 1; i <= createCount; i++) {
- int extra = (i <= remainingItems) ? 1 : 0;
- int singleCommandItems = (itemsPerCommand + extra);
-
- if (i == 1) {
- endDateIndex += singleCommandItems - 1;
- } else {
- startDateIndex = endDateIndex + 1;
- endDateIndex += singleCommandItems;
+ for(List<String> stringDate : Lists.partition(listDate,createCount)){
+ String tempDate = "";
+ for(String date : stringDate){
+ tempDate += date +",";
}
-
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(listDate.get(startDateIndex)));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(listDate.get(endDateIndex)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE, tempDate.substring(0,tempDate.length() - 1));
Review Comment:
OK，Modified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152084776
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [36.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152198505
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [42.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894444784
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -187,11 +183,20 @@ public Map<String, Object> execProcessInstance(User loginUser, long projectCode,
return result;
}
+ if(!checkScheduleTimeNum(commandType,cronTime)){
+ putMsg(result, Status.SCHEDULE_TIME_NUMBER);
+ return result;
+ }
+
+ if(!checkScheduleTimeRepeat(commandType,cronTime)){
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie merged pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
zhongjiajie merged PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152822213
> > LGTM overall. But we still have discussion in [#10376 (comment)](https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894520852)
>
> Hi @SbloodyS you should give a approve because the change requested.
Sorry. I don't have strong opinion here. I just not quite sure about whether should give a approve since we still have unresolved discussions.
I gave approved. Do you have better suggestions? @zhongjiajie @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152062245
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [36.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894518349
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java:
##########
@@ -809,4 +814,10 @@ private Constants() {
* tenant
*/
public static final int TENANT_FULL_NAME_MAX_LENGTH = 30;
+
+ /**
+ * schedule time
+ */
+ public static final int SCHEDULE_TIME_MAX_LENGTH = 100;
Review Comment:
Maybe we should add some docs about why we should set this value to `100` instead of `200` or other values
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -774,7 +775,12 @@ private int createComplementDataCommand(Date scheduleDate) {
if (cmdParam.containsKey(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING)) {
cmdParam.remove(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING);
}
- cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.format(scheduleDate, "yyyy-MM-dd HH:mm:ss", null));
+ if(cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST, cmdParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST).substring(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST).indexOf(",")+1));
+ }
+ if(cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)){
+ cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.format(scheduleDate, "yyyy-MM-dd HH:mm:ss", null));
Review Comment:
I think we have `Constants` value with both `","` and `"yyyy-MM-dd HH:mm:ss"`, should avoid using bare common stirng in our code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152348469
PTAL at this PR if you have time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894115913
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java:
##########
@@ -361,6 +361,11 @@ private Constants() {
*/
public static final String CMDPARAM_COMPLEMENT_DATA_END_DATE = "complementEndDate";
+ /**
+ * complement data Schedule date
+ */
+ public static final String CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE = "complementScheduleDate";
Review Comment:
Modified
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/corn/CronUtils.java:
##########
@@ -35,13 +34,9 @@
import org.apache.commons.collections.CollectionUtils;
import java.text.ParseException;
-import java.util.ArrayList;
-import java.util.Calendar;
-import java.util.Collections;
-import java.util.Date;
-import java.util.GregorianCalendar;
-import java.util.List;
+import java.util.*;
Review Comment:
Modified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894384470
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -854,4 +923,42 @@ private List<DependentProcessDefinition> checkDependentProcessDefinitionValid(Li
return validDependentProcessDefinitionList;
}
+
+ /**
+ *
+ * @param schedule
+ * @return check error return 0 otherwish 1
+ */
+ private int checkScheduleTime(String schedule){
+ Date start = null;
+ Date end = null;
+ Map<String,String> scheduleResult = JSONUtils.toMap(schedule);
+ if(scheduleResult == null){
+ return 0;
+ }
+ if(scheduleResult.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ if(scheduleResult.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST) == null){
+ return 0;
+ }
+ }
+ if(scheduleResult.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)){
+ String startDate = scheduleResult.get(CMDPARAM_COMPLEMENT_DATA_START_DATE);
+ String endDate = scheduleResult.get(CMDPARAM_COMPLEMENT_DATA_END_DATE);
+ if (startDate == null || endDate == null) {
+ return 0;
+ }
+ start = DateUtils.getScheduleDate(startDate);
+ end = DateUtils.getScheduleDate(endDate);
+ if(start == null || end == null){
+ return 0;
+ }
+ if (start.after(end)) {
+ logger.info("complement data error, wrong date start:{} and end date:{} ",
Review Comment:
This should be ```logger.error```
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -686,89 +728,116 @@ private int createCommand(CommandType commandType, long processDefineCode,
* create complement command
* close left and close right
*
- * @param start
- * @param end
+ * @param scheduleTimeParam
* @param runMode
* @return
*/
- protected int createComplementCommandList(Date start, Date end, RunMode runMode, Command command,
+ protected int createComplementCommandList(String scheduleTimeParam, RunMode runMode, Command command,
Integer expectedParallelismNumber, ComplementDependentMode complementDependentMode) {
int createCount = 0;
+ String startDate = null;
+ String endDate = null;
+ String dateList = null;
int dependentProcessDefinitionCreateCount = 0;
-
runMode = (runMode == null) ? RunMode.RUN_MODE_SERIAL : runMode;
Map<String, String> cmdParam = JSONUtils.toMap(command.getCommandParam());
+ Map<String, String> scheduleParam = JSONUtils.toMap(scheduleTimeParam);
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ dateList = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST);
+ }
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE) && scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_END_DATE)){
+ startDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE);
+ endDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE);
+ }
switch (runMode) {
case RUN_MODE_SERIAL: {
- if (start.after(end)) {
- logger.warn("The startDate {} is later than the endDate {}", start, end);
- break;
+ if(StringUtils.isNotEmpty(dateList) || dateList != null){
Review Comment:
I think ```StringUtils.isNotEmpty(null)``` would return false. So it's duplicated.
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -75,13 +71,7 @@
import org.apache.commons.collections.MapUtils;
import org.apache.commons.lang3.StringUtils;
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
Review Comment:
This has not been addressed.
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -686,89 +728,116 @@ private int createCommand(CommandType commandType, long processDefineCode,
* create complement command
* close left and close right
*
- * @param start
- * @param end
+ * @param scheduleTimeParam
* @param runMode
* @return
*/
- protected int createComplementCommandList(Date start, Date end, RunMode runMode, Command command,
+ protected int createComplementCommandList(String scheduleTimeParam, RunMode runMode, Command command,
Integer expectedParallelismNumber, ComplementDependentMode complementDependentMode) {
int createCount = 0;
+ String startDate = null;
+ String endDate = null;
+ String dateList = null;
int dependentProcessDefinitionCreateCount = 0;
-
runMode = (runMode == null) ? RunMode.RUN_MODE_SERIAL : runMode;
Map<String, String> cmdParam = JSONUtils.toMap(command.getCommandParam());
+ Map<String, String> scheduleParam = JSONUtils.toMap(scheduleTimeParam);
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ dateList = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST);
+ }
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE) && scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_END_DATE)){
+ startDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE);
+ endDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE);
+ }
switch (runMode) {
case RUN_MODE_SERIAL: {
- if (start.after(end)) {
- logger.warn("The startDate {} is later than the endDate {}", start, end);
- break;
+ if(StringUtils.isNotEmpty(dateList) || dateList != null){
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST,dateList);
Review Comment:
This line should be formated.
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -187,11 +183,20 @@ public Map<String, Object> execProcessInstance(User loginUser, long projectCode,
return result;
}
+ if(!checkScheduleTimeNum(commandType,cronTime)){
+ putMsg(result, Status.SCHEDULE_TIME_NUMBER);
+ return result;
+ }
+
+ if(!checkScheduleTimeRepeat(commandType,cronTime)){
Review Comment:
I think we should delete duplicate data here instead of return error. This will improve the user experience.
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -686,89 +728,116 @@ private int createCommand(CommandType commandType, long processDefineCode,
* create complement command
* close left and close right
*
- * @param start
- * @param end
+ * @param scheduleTimeParam
* @param runMode
* @return
*/
- protected int createComplementCommandList(Date start, Date end, RunMode runMode, Command command,
+ protected int createComplementCommandList(String scheduleTimeParam, RunMode runMode, Command command,
Integer expectedParallelismNumber, ComplementDependentMode complementDependentMode) {
int createCount = 0;
+ String startDate = null;
+ String endDate = null;
+ String dateList = null;
int dependentProcessDefinitionCreateCount = 0;
-
runMode = (runMode == null) ? RunMode.RUN_MODE_SERIAL : runMode;
Map<String, String> cmdParam = JSONUtils.toMap(command.getCommandParam());
+ Map<String, String> scheduleParam = JSONUtils.toMap(scheduleTimeParam);
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ dateList = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST);
+ }
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE) && scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_END_DATE)){
+ startDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE);
+ endDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE);
+ }
switch (runMode) {
case RUN_MODE_SERIAL: {
- if (start.after(end)) {
- logger.warn("The startDate {} is later than the endDate {}", start, end);
- break;
+ if(StringUtils.isNotEmpty(dateList) || dateList != null){
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST,dateList);
+ command.setCommandParam(JSONUtils.toJsonString(cmdParam));
+ createCount = processService.createCommand(command);
}
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(start));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(end));
- command.setCommandParam(JSONUtils.toJsonString(cmdParam));
- createCount = processService.createCommand(command);
-
- // dependent process definition
- List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
-
- if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
- logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
- + "dependent complement data", command.getProcessDefinitionCode());
- } else {
- dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
+ if(startDate != null && endDate != null){
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, startDate);
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, endDate);
+ command.setCommandParam(JSONUtils.toJsonString(cmdParam));
+ createCount = processService.createCommand(command);
+
+ // dependent process definition
+ List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
+
+ if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
+ logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
+ + "dependent complement data", command.getProcessDefinitionCode());
+ } else {
+ dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
+ }
}
-
break;
}
case RUN_MODE_PARALLEL: {
- if (start.after(end)) {
- logger.warn("The startDate {} is later than the endDate {}", start, end);
- break;
- }
-
- List<Date> listDate = new ArrayList<>();
- List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
- listDate.addAll(CronUtils.getSelfFireDateList(start, end, schedules));
- int listDateSize = listDate.size();
- createCount = listDate.size();
- if (!CollectionUtils.isEmpty(listDate)) {
- if (expectedParallelismNumber != null && expectedParallelismNumber != 0) {
- createCount = Math.min(listDate.size(), expectedParallelismNumber);
- if (listDateSize < createCount) {
- createCount = listDateSize;
+ if(startDate != null && endDate != null){
+ List<Date> listDate = new ArrayList<>();
+ List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
+ listDate.addAll(CronUtils.getSelfFireDateList(DateUtils.getScheduleDate(startDate), DateUtils.getScheduleDate(endDate), schedules));
+ int listDateSize = listDate.size();
+ createCount = listDate.size();
+ if (!CollectionUtils.isEmpty(listDate)) {
+ if (expectedParallelismNumber != null && expectedParallelismNumber != 0) {
+ createCount = Math.min(listDate.size(), expectedParallelismNumber);
+ if (listDateSize < createCount) {
+ createCount = listDateSize;
+ }
+ }
+ logger.info("In parallel mode, current expectedParallelismNumber:{}", createCount);
+
+ // Distribute the number of tasks equally to each command.
+ // The last command with insufficient quantity will be assigned to the remaining tasks.
+ int itemsPerCommand = (listDateSize / createCount);
+ int remainingItems = (listDateSize % createCount);
+ int startDateIndex = 0;
+ int endDateIndex = 0;
+
+ for (int i = 1; i <= createCount; i++) {
+ int extra = (i <= remainingItems) ? 1 : 0;
+ int singleCommandItems = (itemsPerCommand + extra);
+
+ if (i == 1) {
+ endDateIndex += singleCommandItems - 1;
+ } else {
+ startDateIndex = endDateIndex + 1;
+ endDateIndex += singleCommandItems;
+ }
+
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(listDate.get(startDateIndex)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(listDate.get(endDateIndex)));
+ command.setCommandParam(JSONUtils.toJsonString(cmdParam));
+ processService.createCommand(command);
+
+ if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
+ logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
+ + "dependent complement data", command.getProcessDefinitionCode());
+ } else {
+ dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
+ }
}
}
- logger.info("In parallel mode, current expectedParallelismNumber:{}", createCount);
-
- // Distribute the number of tasks equally to each command.
- // The last command with insufficient quantity will be assigned to the remaining tasks.
- int itemsPerCommand = (listDateSize / createCount);
- int remainingItems = (listDateSize % createCount);
- int startDateIndex = 0;
- int endDateIndex = 0;
-
- for (int i = 1; i <= createCount; i++) {
- int extra = (i <= remainingItems) ? 1 : 0;
- int singleCommandItems = (itemsPerCommand + extra);
-
- if (i == 1) {
- endDateIndex += singleCommandItems - 1;
- } else {
- startDateIndex = endDateIndex + 1;
- endDateIndex += singleCommandItems;
+ }
+ if(StringUtils.isNotEmpty(dateList) || dateList != null){
Review Comment:
Same as 754 line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894529999
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java:
##########
@@ -809,4 +814,10 @@ private Constants() {
* tenant
*/
public static final int TENANT_FULL_NAME_MAX_LENGTH = 30;
+
+ /**
+ * schedule time
+ */
+ public static final int SCHEDULE_TIME_MAX_LENGTH = 100;
Review Comment:
This figure is neither too much nor too little. It is tentatively set at 100 at present
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152864572
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [37.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894990748
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -774,7 +775,12 @@ private int createComplementDataCommand(Date scheduleDate) {
if (cmdParam.containsKey(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING)) {
cmdParam.remove(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING);
}
- cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.format(scheduleDate, "yyyy-MM-dd HH:mm:ss", null));
+ if(cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST, cmdParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST).substring(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST).indexOf(",")+1));
+ }
+ if(cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)){
+ cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.format(scheduleDate, "yyyy-MM-dd HH:mm:ss", null));
Review Comment:
it seem that you still not change this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r893043579
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -740,45 +729,24 @@ protected int createComplementCommandList(Date start, Date end, RunMode runMode,
}
}
logger.info("In parallel mode, current expectedParallelismNumber:{}", createCount);
-
// Distribute the number of tasks equally to each command.
// The last command with insufficient quantity will be assigned to the remaining tasks.
- int itemsPerCommand = (listDateSize / createCount);
- int remainingItems = (listDateSize % createCount);
- int startDateIndex = 0;
- int endDateIndex = 0;
-
- for (int i = 1; i <= createCount; i++) {
- int extra = (i <= remainingItems) ? 1 : 0;
- int singleCommandItems = (itemsPerCommand + extra);
-
- if (i == 1) {
- endDateIndex += singleCommandItems - 1;
- } else {
- startDateIndex = endDateIndex + 1;
- endDateIndex += singleCommandItems;
+ for(List<String> stringDate : Lists.partition(listDate,createCount)){
+ String tempDate = "";
+ for(String date : stringDate){
+ tempDate += date +",";
}
-
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(listDate.get(startDateIndex)));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(listDate.get(endDateIndex)));
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE, tempDate.substring(0,tempDate.length() - 1));
command.setCommandParam(JSONUtils.toJsonString(cmdParam));
processService.createCommand(command);
-
- if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
- logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
- + "dependent complement data", command.getProcessDefinitionCode());
- } else {
- dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
- }
Review Comment:
+1, I had the same question here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1151816538
@SbloodyS @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894533401
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -774,7 +775,12 @@ private int createComplementDataCommand(Date scheduleDate) {
if (cmdParam.containsKey(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING)) {
cmdParam.remove(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING);
}
- cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.format(scheduleDate, "yyyy-MM-dd HH:mm:ss", null));
+ if(cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST, cmdParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST).substring(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST).indexOf(",")+1));
+ }
+ if(cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)){
+ cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.format(scheduleDate, "yyyy-MM-dd HH:mm:ss", null));
Review Comment:
Thanks,I also think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894658515
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -686,89 +705,117 @@ private int createCommand(CommandType commandType, long processDefineCode,
* create complement command
* close left and close right
*
- * @param start
- * @param end
+ * @param scheduleTimeParam
* @param runMode
* @return
*/
- protected int createComplementCommandList(Date start, Date end, RunMode runMode, Command command,
+ protected int createComplementCommandList(String scheduleTimeParam, RunMode runMode, Command command,
Integer expectedParallelismNumber, ComplementDependentMode complementDependentMode) {
int createCount = 0;
+ String startDate = null;
+ String endDate = null;
+ String dateList = null;
int dependentProcessDefinitionCreateCount = 0;
-
runMode = (runMode == null) ? RunMode.RUN_MODE_SERIAL : runMode;
Map<String, String> cmdParam = JSONUtils.toMap(command.getCommandParam());
+ Map<String, String> scheduleParam = JSONUtils.toMap(scheduleTimeParam);
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ dateList = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST);
+ dateList = removeDuplicates(dateList);
+ }
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE) && scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_END_DATE)){
+ startDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE);
+ endDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE);
+ }
switch (runMode) {
case RUN_MODE_SERIAL: {
- if (start.after(end)) {
- logger.warn("The startDate {} is later than the endDate {}", start, end);
- break;
+ if(StringUtils.isNotEmpty(dateList)){
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST, dateList);
+ command.setCommandParam(JSONUtils.toJsonString(cmdParam));
+ createCount = processService.createCommand(command);
}
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(start));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(end));
- command.setCommandParam(JSONUtils.toJsonString(cmdParam));
- createCount = processService.createCommand(command);
-
- // dependent process definition
- List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
-
- if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
- logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
- + "dependent complement data", command.getProcessDefinitionCode());
- } else {
- dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
+ if(startDate != null && endDate != null){
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, startDate);
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, endDate);
+ command.setCommandParam(JSONUtils.toJsonString(cmdParam));
+ createCount = processService.createCommand(command);
+
+ // dependent process definition
+ List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
+
+ if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
+ logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
+ + "dependent complement data", command.getProcessDefinitionCode());
+ } else {
+ dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
+ }
}
-
break;
}
case RUN_MODE_PARALLEL: {
- if (start.after(end)) {
- logger.warn("The startDate {} is later than the endDate {}", start, end);
- break;
- }
-
- List<Date> listDate = new ArrayList<>();
- List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
- listDate.addAll(CronUtils.getSelfFireDateList(start, end, schedules));
- int listDateSize = listDate.size();
- createCount = listDate.size();
- if (!CollectionUtils.isEmpty(listDate)) {
- if (expectedParallelismNumber != null && expectedParallelismNumber != 0) {
- createCount = Math.min(listDate.size(), expectedParallelismNumber);
- if (listDateSize < createCount) {
- createCount = listDateSize;
+ if(startDate != null && endDate != null){
Review Comment:
Thanks,here I will consider how to improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894969678
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -774,7 +775,12 @@ private int createComplementDataCommand(Date scheduleDate) {
if (cmdParam.containsKey(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING)) {
cmdParam.remove(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING);
}
- cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.format(scheduleDate, "yyyy-MM-dd HH:mm:ss", null));
+ if(cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST, cmdParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST).substring(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST).indexOf(",")+1));
+ }
+ if(cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)){
+ cmdParam.replace(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.format(scheduleDate, "yyyy-MM-dd HH:mm:ss", null));
Review Comment:
This discussion is not resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152883253
> > > > LGTM overall. But we still have discussion in [#10376 (comment)](https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894520852)
> > >
> > >
> > > Hi @SbloodyS you should give a approve because the change requested.
> >
> >
> > Sorry. I don't have strong opinion here. I just not quite sure about whether should give a approve since we still have unresolved discussions.
> > I gave approved. Do you have better suggestions? @zhongjiajie @caishunfeng
>
> i think we better add some java doc on that constants value about why we set it to 100 instead of other value
At that time, it was considered that the amount of data was too large, and then set the value of 100.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1153066992
I restart the failing `E2E/fileManage`CI, and why this test failed often
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lenboo commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
lenboo commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r893012213
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java:
##########
@@ -361,6 +361,11 @@ private Constants() {
*/
public static final String CMDPARAM_COMPLEMENT_DATA_END_DATE = "complementEndDate";
+ /**
+ * complement data Schedule date
+ */
+ public static final String CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE = "complementScheduleDate";
Review Comment:
It would be a list string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1151764865
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [34.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r893693459
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/enums/Status.java:
##########
@@ -407,6 +407,8 @@ public enum Status {
NO_CURRENT_OPERATING_PERMISSION(1400001, "The current user does not have this permission.", "当前用户无此权限"),
FUNCTION_DISABLED(1400002, "The current feature is disabled.", "当前功能已被禁用"),
+ SCHEDULE_TIME_NUMBER(1400003, "The number of complement dates exceed 100.", "补数日期个数超过100"),
Review Comment:
The current solution is to manually enter and use the selected date. Manual input requires limiting the number and length of the entire set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894117181
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java:
##########
@@ -943,15 +938,12 @@ private void initTaskQueue() {
if (processInstance.isComplementData() && complementListDate.size() == 0) {
Map<String, String> cmdParam = JSONUtils.toMap(processInstance.getCommandParam());
- if (cmdParam != null && cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)) {
+ if (cmdParam != null && cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE)) {
// reset global params while there are start parameters
setGlobalParamIfCommanded(processDefinition, cmdParam);
- Date start = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE));
- Date end = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE));
- List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(processInstance.getProcessDefinitionCode());
if (complementListDate.size() == 0 && needComplementProcess()) {
- complementListDate = CronUtils.getSelfFireDateList(start, end, schedules);
Review Comment:
At present, it has been modified to manual input and automatic selection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894392519
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java:
##########
@@ -88,6 +88,9 @@ public class ExecutorController extends BaseController {
* @param projectCode project code
* @param processDefinitionCode process definition code
* @param scheduleTime schedule time
+ * when CommandType is COMPLEMENT_DATA there are two ways to transfer parameters
Review Comment:
It's recommand to comment before of after the @param instead of between them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894404583
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -854,4 +923,42 @@ private List<DependentProcessDefinition> checkDependentProcessDefinitionValid(Li
return validDependentProcessDefinitionList;
}
+
+ /**
+ *
+ * @param schedule
+ * @return check error return 0 otherwish 1
+ */
+ private int checkScheduleTime(String schedule){
+ Date start = null;
+ Date end = null;
+ Map<String,String> scheduleResult = JSONUtils.toMap(schedule);
+ if(scheduleResult == null){
+ return 0;
+ }
+ if(scheduleResult.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ if(scheduleResult.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST) == null){
+ return 0;
+ }
+ }
+ if(scheduleResult.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)){
+ String startDate = scheduleResult.get(CMDPARAM_COMPLEMENT_DATA_START_DATE);
+ String endDate = scheduleResult.get(CMDPARAM_COMPLEMENT_DATA_END_DATE);
+ if (startDate == null || endDate == null) {
+ return 0;
+ }
+ start = DateUtils.getScheduleDate(startDate);
+ end = DateUtils.getScheduleDate(endDate);
+ if(start == null || end == null){
+ return 0;
+ }
+ if (start.after(end)) {
+ logger.info("complement data error, wrong date start:{} and end date:{} ",
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894197071
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/corn/CronUtils.java:
##########
@@ -16,14 +16,13 @@
*/
package org.apache.dolphinscheduler.service.corn;
-
+import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST;
Review Comment:
There needs to be a blank line in front of this according to checkstyle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894197703
##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/corn/CronUtils.java:
##########
@@ -16,14 +16,13 @@
*/
package org.apache.dolphinscheduler.service.corn;
-
+import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST;
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894647361
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -686,89 +705,117 @@ private int createCommand(CommandType commandType, long processDefineCode,
* create complement command
* close left and close right
*
- * @param start
- * @param end
+ * @param scheduleTimeParam
* @param runMode
* @return
*/
- protected int createComplementCommandList(Date start, Date end, RunMode runMode, Command command,
+ protected int createComplementCommandList(String scheduleTimeParam, RunMode runMode, Command command,
Integer expectedParallelismNumber, ComplementDependentMode complementDependentMode) {
int createCount = 0;
+ String startDate = null;
+ String endDate = null;
+ String dateList = null;
int dependentProcessDefinitionCreateCount = 0;
-
runMode = (runMode == null) ? RunMode.RUN_MODE_SERIAL : runMode;
Map<String, String> cmdParam = JSONUtils.toMap(command.getCommandParam());
+ Map<String, String> scheduleParam = JSONUtils.toMap(scheduleTimeParam);
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ dateList = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST);
+ dateList = removeDuplicates(dateList);
+ }
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE) && scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_END_DATE)){
+ startDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE);
+ endDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE);
+ }
switch (runMode) {
case RUN_MODE_SERIAL: {
- if (start.after(end)) {
- logger.warn("The startDate {} is later than the endDate {}", start, end);
- break;
+ if(StringUtils.isNotEmpty(dateList)){
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST, dateList);
+ command.setCommandParam(JSONUtils.toJsonString(cmdParam));
+ createCount = processService.createCommand(command);
}
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, DateUtils.dateToString(start));
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, DateUtils.dateToString(end));
- command.setCommandParam(JSONUtils.toJsonString(cmdParam));
- createCount = processService.createCommand(command);
-
- // dependent process definition
- List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
-
- if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
- logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
- + "dependent complement data", command.getProcessDefinitionCode());
- } else {
- dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
+ if(startDate != null && endDate != null){
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, startDate);
+ cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, endDate);
+ command.setCommandParam(JSONUtils.toJsonString(cmdParam));
+ createCount = processService.createCommand(command);
+
+ // dependent process definition
+ List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
+
+ if (schedules.isEmpty() || complementDependentMode == ComplementDependentMode.OFF_MODE) {
+ logger.info("process code: {} complement dependent in off mode or schedule's size is 0, skip "
+ + "dependent complement data", command.getProcessDefinitionCode());
+ } else {
+ dependentProcessDefinitionCreateCount += createComplementDependentCommand(schedules, command);
+ }
}
-
break;
}
case RUN_MODE_PARALLEL: {
- if (start.after(end)) {
- logger.warn("The startDate {} is later than the endDate {}", start, end);
- break;
- }
-
- List<Date> listDate = new ArrayList<>();
- List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
- listDate.addAll(CronUtils.getSelfFireDateList(start, end, schedules));
- int listDateSize = listDate.size();
- createCount = listDate.size();
- if (!CollectionUtils.isEmpty(listDate)) {
- if (expectedParallelismNumber != null && expectedParallelismNumber != 0) {
- createCount = Math.min(listDate.size(), expectedParallelismNumber);
- if (listDateSize < createCount) {
- createCount = listDateSize;
+ if(startDate != null && endDate != null){
Review Comment:
It's better to separate these out into a method like: createComplementCommandByCron and createComplementCommandByDateList, just a code improvement suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152306340
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [38.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894507725
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -75,13 +71,7 @@
import org.apache.commons.collections.MapUtils;
import org.apache.commons.lang3.StringUtils;
-import java.util.ArrayList;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;
Review Comment:
why our codesytle check can not catch this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1152874927
> > > LGTM overall. But we still have discussion in [#10376 (comment)](https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894520852)
> >
> > Hi @SbloodyS you should give a approve because the change requested.
>
> Sorry. I don't have strong opinion here. I just not quite sure about whether should give a approve since we still have unresolved discussions.
>
> I gave approved. Do you have better suggestions? @zhongjiajie @caishunfeng
i think we better add some java doc on that constants value about why we set it to 100 instead of other value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r893694879
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -740,45 +729,24 @@ protected int createComplementCommandList(Date start, Date end, RunMode runMode,
}
}
logger.info("In parallel mode, current expectedParallelismNumber:{}", createCount);
-
// Distribute the number of tasks equally to each command.
// The last command with insufficient quantity will be assigned to the remaining tasks.
- int itemsPerCommand = (listDateSize / createCount);
- int remainingItems = (listDateSize % createCount);
- int startDateIndex = 0;
- int endDateIndex = 0;
-
- for (int i = 1; i <= createCount; i++) {
- int extra = (i <= remainingItems) ? 1 : 0;
- int singleCommandItems = (itemsPerCommand + extra);
-
- if (i == 1) {
- endDateIndex += singleCommandItems - 1;
- } else {
- startDateIndex = endDateIndex + 1;
- endDateIndex += singleCommandItems;
+ for(List<String> stringDate : Lists.partition(listDate,createCount)){
+ String tempDate = "";
+ for(String date : stringDate){
+ tempDate += date +",";
Review Comment:
1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1151815491
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [34.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lenboo commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
lenboo commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1150994311
I think we can keep two complement data way in the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894117086
##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteThread.java:
##########
@@ -943,15 +938,12 @@ private void initTaskQueue() {
if (processInstance.isComplementData() && complementListDate.size() == 0) {
Map<String, String> cmdParam = JSONUtils.toMap(processInstance.getCommandParam());
- if (cmdParam != null && cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)) {
+ if (cmdParam != null && cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE)) {
// reset global params while there are start parameters
setGlobalParamIfCommanded(processDefinition, cmdParam);
- Date start = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE));
- Date end = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE));
- List<Schedule> schedules = processService.queryReleaseSchedulerListByProcessDefinitionCode(processInstance.getProcessDefinitionCode());
if (complementListDate.size() == 0 && needComplementProcess()) {
- complementListDate = CronUtils.getSelfFireDateList(start, end, schedules);
Review Comment:
目前已经修改成手动输入和自动选择两种方式
At present, it has been modified to manual input and automatic selection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
hstdream commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894402415
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java:
##########
@@ -686,89 +728,116 @@ private int createCommand(CommandType commandType, long processDefineCode,
* create complement command
* close left and close right
*
- * @param start
- * @param end
+ * @param scheduleTimeParam
* @param runMode
* @return
*/
- protected int createComplementCommandList(Date start, Date end, RunMode runMode, Command command,
+ protected int createComplementCommandList(String scheduleTimeParam, RunMode runMode, Command command,
Integer expectedParallelismNumber, ComplementDependentMode complementDependentMode) {
int createCount = 0;
+ String startDate = null;
+ String endDate = null;
+ String dateList = null;
int dependentProcessDefinitionCreateCount = 0;
-
runMode = (runMode == null) ? RunMode.RUN_MODE_SERIAL : runMode;
Map<String, String> cmdParam = JSONUtils.toMap(command.getCommandParam());
+ Map<String, String> scheduleParam = JSONUtils.toMap(scheduleTimeParam);
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST)){
+ dateList = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_SCHEDULE_DATE_LIST);
+ }
+ if(scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE) && scheduleParam.containsKey(CMDPARAM_COMPLEMENT_DATA_END_DATE)){
+ startDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE);
+ endDate = scheduleParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE);
+ }
switch (runMode) {
case RUN_MODE_SERIAL: {
- if (start.after(end)) {
- logger.warn("The startDate {} is later than the endDate {}", start, end);
- break;
+ if(StringUtils.isNotEmpty(dateList) || dateList != null){
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#discussion_r894392519
##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java:
##########
@@ -88,6 +88,9 @@ public class ExecutorController extends BaseController {
* @param projectCode project code
* @param processDefinitionCode process definition code
* @param scheduleTime schedule time
+ * when CommandType is COMPLEMENT_DATA there are two ways to transfer parameters
Review Comment:
It's recommand to comment before of after the `@param` instead of between them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10376: [Improvement] [Api & Master] Complement function transformation
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10376:
URL: https://github.com/apache/dolphinscheduler/pull/10376#issuecomment-1153061073
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10376)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10376&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list) [37.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10376&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org