You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/02/14 09:57:44 UTC
[GitHub] [dolphinscheduler] caishunfeng opened a new issue #8373: [Feature][MasterServer] Dependent tasks can re-run automatically in the case of complement
caishunfeng opened a new issue #8373:
URL: https://github.com/apache/dolphinscheduler/issues/8373
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
- Feature Scenario:
When an exception occurs in the upstream task and needs to re-run complement, hope that can selete to re-run the dependent task automatically when cross DAG.
- Analysis:
Now in DS, there are two dependent task types: DEPENDENT and SUB_PROCESS. SUB_PROCESS task can re-run when parent workflow execute, so what we need to support is the DEPENDENT task type.
- Main idea:
- How to get the workflow relationship? Now there is no relation table of workflow, but you can use the `WorkFlowLineage`, see `WorkFlowLineageServiceImpl.queryWorkFlowLineage`
- The same time frequency, for example, DAG A1 schedule each hour, and the DAG B1 which use a DEPENDENT task B1.a1 to check A1.a1, the time frequency for the DEPENDENT task B1.a1 to check must be every hour. (If B1.a1 check each day, auto re-run will create many useless B1 process instances; But if you have a good idea, please say it)
- Dependent task should be the start task to re-run, if it has some front task, all these front tasks should be skip.
Chinese Translation:
- 功能场景：
当上游任务发生异常需要重新补数时，希望跨DAG的情况下可以选择自动触发下游任务补数。
- 分析：
现在在DS 中，有两种依赖任务类型：DEPENDENT 和 SUB_PROCESS。 SUB_PROCESS 任务可以在父工作流执行时重新运行，所以我们需要支持的是 DEPENDENT 任务类型。
- 大致思路：
- 如何获得工作流关系？现在DS没有工作流的关系表，但是你可以借鉴`WorkFlowLineage`，见`WorkFlowLineageServiceImpl.queryWorkFlowLineage`
- 相同的时间频率，例如DAG A1调度每小时一次，DAG B1使用DEPENDENT任务B1.a1检查A1.a1，DEPENDENT任务B1.a1检查的时间频率必须是每小时。 （如果 B1.a1 每天检查一次，则自动触发下游补数会创建很多无用的 B1 流程实例；如果您有好的想法，请告诉我）
- 依赖任务应该是自动触发下游补数的开始节点，如果它有一些前置任务，则所有这些前置任务都应该跳过。
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #8373: [Feature][MasterServer] Dependent tasks can re-run automatically in the case of complement
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #8373:
URL: https://github.com/apache/dolphinscheduler/issues/8373#issuecomment-1046859951
I would like to develop this feature. Please assign to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #8373: [Feature][MasterServer] Dependent tasks can re-run automatically in the case of complement
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #8373:
URL: https://github.com/apache/dolphinscheduler/issues/8373#issuecomment-1038882819
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #8373: [Feature][MasterServer] Dependent tasks can re-run automatically in the case of complement
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #8373:
URL: https://github.com/apache/dolphinscheduler/issues/8373#issuecomment-1038882651
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
- Feature Scenario:
When an exception occurs in the upstream task and needs to re-run complement, hope that can selete to re-run the dependent task automatically when cross DAG.
- Analysis:
Now in DS, there are two dependent task types: DEPENDENT and SUB_PROCESS. SUB_PROCESS task can re-run when parent workflow execute, so what we need to support is the DEPENDENT task type.
- Main idea:
- How to get the workflow relationship? Now there is no relation table of workflow, but you can use the `WorkFlowLineage`, see `WorkFlowLineageServiceImpl.queryWorkFlowLineage`
- The same time frequency, for example, DAG A1 schedule each hour, and the DAG B1 which use a DEPENDENT task B1.a1 to check A1.a1, the time frequency for the DEPENDENT task B1.a1 to check must be every hour. (If B1.a1 check each day, auto re-run will create many useless B1 process instances; But if you have a good idea, please say it)
- Dependent task should be the start task to re-run, if it has some front task, all these front tasks should be skip.
Chinese Translation:
- 功能场景：
当上游任务发生异常需要重新补数时，希望跨DAG的情况下可以选择自动触发下游任务补数。
- 分析：
现在在DS 中，有两种依赖任务类型：DEPENDENT 和 SUB_PROCESS。 SUB_PROCESS 任务可以在父工作流执行时重新运行，所以我们需要支持的是 DEPENDENT 任务类型。
- 大致思路：
- 如何获得工作流关系？现在DS没有工作流的关系表，但是你可以借鉴`WorkFlowLineage`，见`WorkFlowLineageServiceImpl.queryWorkFlowLineage`
- 相同的时间频率，例如DAG A1调度每小时一次，DAG B1使用DEPENDENT任务B1.a1检查A1.a1，DEPENDENT任务B1.a1检查的时间频率必须是每小时。 （如果 B1.a1 每天检查一次，则自动触发下游补数会创建很多无用的 B1 流程实例；如果您有好的想法，请告诉我）
- 依赖任务应该是自动触发下游补数的开始节点，如果它有一些前置任务，则所有这些前置任务都应该跳过。
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng closed issue #8373: [Feature][MasterServer] Dependent tasks can re-run automatically in the case of complement
Posted by GitBox <gi...@apache.org>.
caishunfeng closed issue #8373:
URL: https://github.com/apache/dolphinscheduler/issues/8373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org