You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/07/15 07:54:56 UTC
[GitHub] [dolphinscheduler] jieguangzhou opened a new issue, #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
jieguangzhou opened a new issue, #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
Can we support read yaml to create and run workflow？
like this :https://github.com/ajbosco/dag-factory
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1189073944
> Good idea, it's better to add some design details.
Ok, I will add details here in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1189070710
Can we parse the data directly and store it in the database? or do we have to do it in other ways?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1185422835
@caishunfeng @ruanwenjun @EricGao888 @zhongjiajie PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1185584172
> Hi @jieguangzhou, this looks good to me. For some users who are not willing to spend some time picking up python but do have tons of workflows to generate, yaml is a better approach than dragging and connecting tasks through the UI. Just curious, why do you want this feature? Are you a `yaml lover`? 😄
If we can use YAML to manage workflows, it might make it easier for engineers to manage workflows and do version management. Git can be used to manage and share workflows.
In addition, after communicating with some overseas engineers, I found that some engineers prefer to use YAML to manage programs, which I think is also suitable for me.
I think `yaml` is better than `TOML` or `JSON` for this scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1185285531
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1188827659
Good idea, it's better to add some design details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1189077996
> It's OK for me to define a DSL to create a workflow, some geeks may like this way, but this can not work well in big workflow.
> It's OK for me to define a DSL to create a workflow, some geeks may like this way, but this can not work well in big workflow.
Thank you for your suggestions. I will think again about how to handle the large workflow scenario, and if not, let the user use the original way to support the large workflow for the time being
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1188893517
It's OK for me to define a DSL to create a workflow, some geeks may like this way, but this can not work well in big workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1195353667
Hi @jieguangzhou I add `DSIP` to this issue, because it add new mechanism to Python API, please follow the https://dolphinscheduler.apache.org/en-us/community/DSIP.html to create a new DSIP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1195354264
BTW, I think it is a good adding for python api
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10995: [Feature][pydolphinscheduler] Dynamically generate workflows from YAML configuration files
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10995:
URL: https://github.com/apache/dolphinscheduler/issues/10995#issuecomment-1185518708
Hi @jieguangzhou, this looks good to me. For some users who are not willing to spend some time picking up python but do have tons of workflows to generate, yaml is a better approach than dragging and connecting tasks through the UI. Just curious, why do you want this feature? Are you a `yaml lover`? 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie closed issue #10995: [DSIP-11][Feature][python] Use the pydolphinscheduler to dynamically generate workflows from the configuration file
Posted by GitBox <gi...@apache.org>.
zhongjiajie closed issue #10995: [DSIP-11][Feature][python] Use the pydolphinscheduler to dynamically generate workflows from the configuration file
URL: https://github.com/apache/dolphinscheduler/issues/10995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org