You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/04/21 06:32:04 UTC
[GitHub] [dolphinscheduler] Hou-Shuaishuai opened a new issue, #9641: [Feature][API] Self dependencies are in a permanent wait state.
Hou-Shuaishuai opened a new issue, #9641:
URL: https://github.com/apache/dolphinscheduler/issues/9641
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
If a task is configured with self-dependencies, the task needs to wait for the self-dependencies to complete the first time it is executed
### Use case
Configuring self-dependency for the first time should not require waiting for the self-dependency to complete
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Hou-Shuaishuai commented on issue #9641: [Feature][API] Self dependencies are in a permanent wait state.
Posted by GitBox <gi...@apache.org>.
Hou-Shuaishuai commented on issue #9641:
URL: https://github.com/apache/dolphinscheduler/issues/9641#issuecomment-1109450412
1. Update update_time in t_ds_schedules both online and offline
2. When judging the dependency status, if it is found to be self-dependent, it will judge whether the scheduling execution time is the first execution time after the scheduling update, and return the dependency successfully completed state if it is equal.
Logic code：org.apache.dolphinscheduler.server.utils.DependentExecute.getDependentResultForItem Method
//isSelfDep: Determine depending on projectCode, definitionCode, and depTaskCode
//getScheduleFirstFireTime: according to t_ds_schedules table update_time crontab and it is concluded that the execution time for the first time
private DependResult getDependentResultForItem(DependentItem dependentItem, Date currentTime) {
if (isSelfDep() && getScheduleFirstFireTime() == processInstance.getSchedule_time()){
return DependResult.SUCCESS;
}
List<DateInterval> dateIntervals = DependentUtils.getDateIntervalList(currentTime, dependentItem.getDateValue());
return calculateResultForTasks(dependentItem, dateIntervals);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #9641: [Feature][API] Self dependencies are in a permanent wait state.
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #9641:
URL: https://github.com/apache/dolphinscheduler/issues/9641#issuecomment-1104844707
Then how can the task finish its first time task run success?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Hou-Shuaishuai commented on issue #9641: [Feature][API] Self dependencies are in a permanent wait state.
Posted by GitBox <gi...@apache.org>.
Hou-Shuaishuai commented on issue #9641:
URL: https://github.com/apache/dolphinscheduler/issues/9641#issuecomment-1109704629
Since the dependency has an option like the previous 7 days, the logic of the dependency executed in the previous 7 days should be different from the logic executed after 7 days.
eg：
On day 1, the judgment dependency returns success
On day 2, only the dependencies executed on the previous 1 day are judged
On day 3, only the dependencies executed in the previous 2 days are judged
...
On day 7, only the dependencies executed in the previous 6 days are judged
On day 8, only the dependencies executed in the previous 7 days are judged
On day 9, only the dependencies executed in the previous 7 days are judged
...
To sum up, only the dependence after the execution time of the first scheduling is judged
1. Update update_time in t_ds_schedules both online and offline
2. When determining the dependency status, if it is found to be 'self-dependency', the first scheduling execution time after scheduling update is obtained: 'firstScheduleFireTime'. Then go through all the dependent dates you need to determine: 'dateIntervals', and delete the dependent dates whose 'startTime' is smaller than 'firstScheduleFireTime'
3. If dateIntervals is empty that is the first execution, return success
Logic code：org.apache.dolphinscheduler.server.utils.DependentExecute.getDependentResultForItem Method
```java
private DependResult getDependentResultForItem(DependentItem dependentItem, Date currentTime) {
List<DateInterval> dateIntervals = DependentUtils.getDateIntervalList(currentTime, dependentItem.getDateValue());
if (isSelfDep()){
Date firstScheduleFireTime = getScheduleFirstFireTime();
removeFirstScheduleFireTimeBreforeInterval(dateIntervals, firstScheduleFireTime);
if(dateIntervals.isEmpty()){
return DependResult.SUCCESS;
}
}
return calculateResultForTasks(dependentItem, dateIntervals);
}
//TODO delete the dependent dates whose 'startTime'is smaller than 'firstScheduleFireTime'
private void removeFirstScheduleFireTimeBreforeInterval(List<DateInterval> dateIntervals, Date firstScheduleFireTime) {
Iterator<DateInterval> iterator = dateIntervals.iterator();
while (iterator.hasNext()) {
if (iterator.next().getStartTime().after(firstScheduleFireTime) {
iterator.remove();
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #9641: [Feature][API] Self dependencies are in a permanent wait state.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #9641:
URL: https://github.com/apache/dolphinscheduler/issues/9641#issuecomment-1104767904
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org