You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "det101 (via GitHub)" <gi...@apache.org> on 2025/11/27 03:54:50 UTC
[I] [DSIP-95][API] Complete the function to enable dependency mode. [dolphinscheduler]
det101 opened a new issue, #17748:
URL: https://github.com/apache/dolphinscheduler/issues/17748
### Search before asking
- [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.
### Motivation
Current dependency mode configuration in frontend cannot connect the backfill dependency enabling action with downstream workflows. We want to improve the rerun workflow so that it correctly triggers the downstream workflows after backfill, supporting two dependency modes.
### Design Detail
- Add implementation for frontend dependency mode switching
- Support both Basic Mode (trigger only direct downstream workflows) and All-Level Mode (recursively trigger all dependency levels).
- Implement traversal logic (BFS/DFS) for recursive triggering, with protection against cycles and duplicate triggering.
- Update UI to reflect mode and ensure users can select preferred behavior.
- Add tests for both triggering modes.
### Compatibility, Deprecation, and Migration Plan
This improvement does not introduce breaking changes to existing workflows, but recursive triggering should be idempotent to prevent duplicate executions. Cycle detection will be implemented to protect against circular dependencies.
### Test Plan
- Verify direct and recursive triggering work correctly for various workflow dependency graphs.
- Add automated tests on workflow DAGs for both dependency modes.
- Manual test for large-scale workflow with complex dependencies to ensure reliability and performance.
- Test cycle detection and duplicate triggering prevention.
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [DSIP-95][API] Complete the functionality of using dependencies in the complement data [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17748:
URL: https://github.com/apache/dolphinscheduler/issues/17748#issuecomment-3599792067
Does `recursive mode with user-defined depth` is really useful? This might be difficult to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [DSIP-95][API] Complete the function to enable dependency mode. [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17748:
URL: https://github.com/apache/dolphinscheduler/issues/17748#issuecomment-3588052902
> We want to improve the rerun workflow so that it correctly triggers the downstream workflows after backfill, supporting two dependency modes.
I'm a little confused. Do you want to optimize the complement process or rerun the workflow process?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [DSIP-95][API] Complete the functionality of using dependencies in the complement data [dolphinscheduler]
Posted by "det101 (via GitHub)" <gi...@apache.org>.
det101 commented on issue #17748:
URL: https://github.com/apache/dolphinscheduler/issues/17748#issuecomment-3601140512
> Does `recursive mode with user-defined depth` is really useful? This might be difficult to use.
After discussion and careful consideration, we found that in complex dependency scenarios, specifying a custom level may trigger dependency chains that end up waiting on one another. Supporting this feature would likely require recursively checking whether upstream nodes are part of the dependency chain, which is significantly more complex than initially anticipated. I recommend implementing the existing functionality first and discussing the custom-level feature separately. @ruanwenjun @SbloodyS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [DSIP-95][API] Complete the function to enable dependency mode. [dolphinscheduler]
Posted by "det101 (via GitHub)" <gi...@apache.org>.
det101 commented on issue #17748:
URL: https://github.com/apache/dolphinscheduler/issues/17748#issuecomment-3588088722
> > We want to improve the rerun workflow so that it correctly triggers the downstream workflows after backfill, supporting two dependency modes.
>
> I'm a little confused. Do you want to optimize the complement process or rerun the workflow process?
It should be "improve the complement data process," the wording was incorrect and has been corrected. @SbloodyS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [DSIP-95][API] Complete the functionality of using dependencies in the complement data [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS closed issue #17748: [DSIP-95][API] Complete the functionality of using dependencies in the complement data
URL: https://github.com/apache/dolphinscheduler/issues/17748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [DSIP-95][API] Complete the functionality of using dependencies in the complement data [dolphinscheduler]
Posted by "det101 (via GitHub)" <gi...@apache.org>.
det101 commented on issue #17748:
URL: https://github.com/apache/dolphinscheduler/issues/17748#issuecomment-3588344075
> > It should be "improve the complement data process," the wording was incorrect and has been corrected. [@SbloodyS](https://github.com/SbloodyS)
>
> Thanks for clarification. Before PR, there are some details needs to be discuss.
>
> 1. Is it better to use provide the user with the specified level in the basic mode and set 1 as the default value? [@det101](https://github.com/det101)
Sounds good. I’ll update the design so users can specify the recursion depth and we’ll keep the default at 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [DSIP-95][API] Complete the functionality of using dependencies in the complement data [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17748:
URL: https://github.com/apache/dolphinscheduler/issues/17748#issuecomment-3594642419
I have no more questions. Let's see if others have other opinions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [DSIP-95][API] Complete the functionality of using dependencies in the complement data [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17748:
URL: https://github.com/apache/dolphinscheduler/issues/17748#issuecomment-3588140701
> It should be "improve the complement data process," the wording was incorrect and has been corrected. @SbloodyS
Thanks for clarification. Before PR, there are some details needs to be discuss.
1. Is it better to use provide the user with the specified level in the basic mode and set 1 as the default value? @det101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org