You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/24 15:26:15 UTC
[GitHub] [dolphinscheduler] sketchmind opened a new issue, #11631: [Improvement][Task] Dependent task nodes support setting dependencies based on offsets
sketchmind opened a new issue, #11631:
URL: https://github.com/apache/dolphinscheduler/issues/11631
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
The current dependence task node can support very limited scenarios, if it is changed to offset, it will be more flexible to support more scenarios.
Volcengine has a very good implementation, like the following, which can achieve more dependency scenarios:
># Dependency settings
>
>## Task dependencies
>
>The essence of timed tasks is data processing. In the process of data processing, two processes of data input and data output may be >involved. When tasks read data, they will form dependencies between tasks, which are essentially dependencies between tables. Data >development uses task dependencies to form the upstream and downstream relationships of data workflows.
>
>## Cross-Cycle Self-Dependence
>
>* Monthly tasks: The execution of this month's tasks depends on the running results of the previous month.
>
>* Weekly tasks: The execution of this week's tasks depends on the results of the previous week.
>
>* Day tasks: The execution of today's tasks depends on the results of yesterday's operation.
>
>* Hourly task: The task execution of the current hour depends on the running result of the previous hour.
>
>* Minute task: The execution of the task instance in the current minute depends on the running result of the previous minute task >instance.
>
>## Dependency configuration between tasks
>
>Description: M-month execution frequency, W-week execution frequency, D-day execution frequency, H-hour execution frequency
| type | actual case | offset settings (interval/collection offset offset units) |
|----------------|---------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------|
| D depends on D | Depends on the instance of the current day of the upstream task | Collection 0 days |
| D depends on D | Depends on the instance from the day before the upstream task | Collection - 1 day |
| D depends on D | Instances that depend on the upstream task in the last 3 days | interval [-2, 0] days |
| D depends on H | Instances that depend on upstream tasks for the same business date for 24 hours | interval [-23, 0] hours |
| D depends on H | Instances that depend on the same business date of the upstream task and the business date is 0:00, 8:00, and 16:00 | set [-23, -15, -7] hours |
| D depends on W | Depends on the instance of the upstream task last week | Collection - 1 week |
| D depends on M | Depends on the instance of the upstream task last month | Collection - January |
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Hxssssss commented on issue #11631: [Improvement][Task] Dependent task nodes support setting dependencies based on offsets
Posted by GitBox <gi...@apache.org>.
Hxssssss commented on issue #11631:
URL: https://github.com/apache/dolphinscheduler/issues/11631#issuecomment-1226820556
I want to participate in the development of this piece of content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Task] Dependent task nodes support setting dependencies based on offsets [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11631:
URL: https://github.com/apache/dolphinscheduler/issues/11631#issuecomment-1778268623
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11631: [Improvement][Task] Dependent task nodes support setting dependencies based on offsets
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11631:
URL: https://github.com/apache/dolphinscheduler/issues/11631#issuecomment-1279859484
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sketchmind commented on issue #11631: [Improvement][Task] Dependent task nodes support setting dependencies based on offsets
Posted by GitBox <gi...@apache.org>.
sketchmind commented on issue #11631:
URL: https://github.com/apache/dolphinscheduler/issues/11631#issuecomment-1279925073
keep-alivekeep-alive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] YuanGunGunO commented on issue #11631: [Improvement][Task] Dependent task nodes support setting dependencies based on offsets
Posted by "YuanGunGunO (via GitHub)" <gi...@apache.org>.
YuanGunGunO commented on issue #11631:
URL: https://github.com/apache/dolphinscheduler/issues/11631#issuecomment-1722133956
it seems like self dependencies can solve your probleam but can deal with cross cycle dependencies problems
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11631: [Improvement][Task] Dependent task nodes support setting dependencies based on offsets
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11631:
URL: https://github.com/apache/dolphinscheduler/issues/11631#issuecomment-1225918379
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Task] Dependent task nodes support setting dependencies based on offsets [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11631: [Improvement][Task] Dependent task nodes support setting dependencies based on offsets
URL: https://github.com/apache/dolphinscheduler/issues/11631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][Task] Dependent task nodes support setting dependencies based on offsets [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11631:
URL: https://github.com/apache/dolphinscheduler/issues/11631#issuecomment-1765463841
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org