[DISCUSS][DSIP-11] Use the pydolphinscheduler to dynamically generate workflows from the configuration file

[Jieguang Zhou <ji...@163.com>]

Hi community,

I think we can use the pydolphinscheduler to dynamically generate workflows from the configuration file.

Dynamically generate workflows from YAML configuration files, aka workflows-as-code. 

Now we can use python-dolphinscheduler to build workflow easily. Workflows-as-code would be easier to use if we supported configuration file definition workflows. It will make the workflow clearer, shareable, and reviewable.

Then we can upload our project as a git repository. This can help us do richer operations, such as CICD, etc

We can create a workload by defining the following fields, all the field definitions are derived from [PydolphinScheduler](https://dolphinscheduler.apache.org/python/concept.html#).

A YAML file defines a workflow :

```yaml
# Define the process
Process:
  #process(workflow) name
  name: prepare_datas

  # Parameters for process creation
  param:
      project: /data/project

  # Whether to run the workflow after the creation is complete
  run: True

# Define the tasks under the process
Tasks:
  -
    # task type: Shell, python, SubProcess, Spark, etc
    TaskType: Shell

    # Upstream Task List
    dependencies: [xxxx]

    # Parameters for task creation
    params:
      name: task1
      other parameters: ...

  -  
    TaskType: Python
    params:
      name: task2
  - ...
  - ... 
```

I already add a GitHub Issue for my proposal, which you could see in https://github.com/apache/dolphinscheduler/issues/10995

Looking forward any feedback for this thread.

Re: [DISCUSS][DSIP-11] Use the pydolphinscheduler to dynamically generate workflows from the configuration file

[Chufeng Gao <ch...@gmail.com>]

Hi Jieguang,

Generating workflows from `yaml` template is a better way for users who
need to create tons of workflows at a time.
Furthermore, if users have little knowledge in python, `yaml` helps them
get hands dirty more easily if they want to try `workflow as code`.
BTW, if users want to migrate from other big data orchestration tools such
as Airflow, Azkaban, Oozie, etc. to DS, converting workflows from previous
systems into `yaml` files helps ease the pain.

Above all, I'm +1 to this and looking forward to a detailed design.

Thanks.

*Best Regards,*

*Chufeng (Eric) Gao*



Jieguang Zhou <ji...@163.com> 于2022年7月29日周五 15:20写道：

> Hi community,
>
> I think we can use the pydolphinscheduler to dynamically generate
> workflows from the configuration file.
>
> Dynamically generate workflows from YAML configuration files, aka
> workflows-as-code.
>
> Now we can use python-dolphinscheduler to build workflow easily.
> Workflows-as-code would be easier to use if we supported configuration file
> definition workflows. It will make the workflow clearer, shareable, and
> reviewable.
>
> Then we can upload our project as a git repository. This can help us do
> richer operations, such as CICD, etc
>
> We can create a workload by defining the following fields, all the field
> definitions are derived from [PydolphinScheduler](
> https://dolphinscheduler.apache.org/python/concept.html#).
>
> A YAML file defines a workflow :
>
> ```yaml
> # Define the process
> Process:
>   #process(workflow) name
>   name: prepare_datas
>
>   # Parameters for process creation
>   param:
>       project: /data/project
>
>   # Whether to run the workflow after the creation is complete
>   run: True
>
> # Define the tasks under the process
> Tasks:
>   -
>     # task type: Shell, python, SubProcess, Spark, etc
>     TaskType: Shell
>
>     # Upstream Task List
>     dependencies: [xxxx]
>
>     # Parameters for task creation
>     params:
>       name: task1
>       other parameters: ...
>
>   -
>     TaskType: Python
>     params:
>       name: task2
>   - ...
>   - ...
> ```
>
> I already add a GitHub Issue for my proposal, which you could see in
> https://github.com/apache/dolphinscheduler/issues/10995
>
> Looking forward any feedback for this thread.