[DISCUSS][Feature] Organize workflows and tasks with directories

[Rick Cheng <ri...@gmail.com>]

Hi, Community

Currently, it is not possible to create a directory in the workflow
definition list to customize the organization of the workflow. When there
are a large number of workflows, it will appear very messy.

This feature hopes to organize workflows through directories, and since
tasks belong to a certain workflow, task definitions will naturally be
organized by directories.

Related issue: https://github.com/apache/dolphinscheduler/issues/14333

# UI
## Process Definition Page

* Show subcategories and subprocesses
* Support to create directories
* The operations supported by the directory include modification and
deletion

## Task Definition Page

* Show subcategories and subtasks
* Directory does not support any operations

# Metadata
* Add column of `t_ds_process_definition`
  * `parent_directory_code`: the code of the parent directory or 0 if the
parent directory is the root of the project
* Add `t_ds_process_definition_directory`
  * id
  * name
  * code
  * parent_directory_code:  the code of the parent directory or 0 if the
parent directory is the root of the project
  * project_code
  * user_id
  * ...

The current solution is to add a new table
`t_ds_process_definition_directory`  for storing directoy entities, and add
a column in `t_ds_process_definition` to store directory codes.

If we do not add a new table (for example, just add a tag in the process
definition), we cannot save the directory created by the user. And it will
also become very complicated when querying workflows and subdirectories
under a certain directory.

# RESTful API
## New API
* createProcessDefinitionDirectory
* updateProcessDefinitionDirectory
* deleteProcessDefinitionDirectory

## Modify API
### createProcessDefinition
* Add `parent_directory_code`, which means creating a workflow in a certain
directory
### queryProcessDefinitionListPaging
* Add `parent_directory_code` to return all process definition and
subdirectories under a certain directory
* Add an `isDirectory` field to the returned result entity, indicating
whether it is a directory
### queryTaskDefinitionListPaging
* A directory of a task is consistent with the workflow it belongs to
* Add `parent_directory_code` to return all task definition and
subdirectories under a certain directory


I gave some design in this issue. If you are interested, any comments or
suggestions are welcome.

Best Regards,
Rick Cheng

Re: [DISCUSS][Feature] Organize workflows and tasks with directories

[Jay Chung <zh...@gmail.com>]

Hey Rick,

Thanks for bringing this up, I also have a question about users prefer
to use a tag or directory to organize the workflow in the same
project. I personally think using tag is good(can reduce new table
add), but do not have strong opinion, if others thing using a
directory is a better option, is also good to me.

On Fri, Jul 14, 2023 at 3:36 PM Rick Cheng <ri...@gmail.com> wrote:
>
> Hi, Community
>
> Currently, it is not possible to create a directory in the workflow
> definition list to customize the organization of the workflow. When there
> are a large number of workflows, it will appear very messy.
>
> This feature hopes to organize workflows through directories, and since
> tasks belong to a certain workflow, task definitions will naturally be
> organized by directories.
>
> Related issue: https://github.com/apache/dolphinscheduler/issues/14333
>
> # UI
> ## Process Definition Page
>
> * Show subcategories and subprocesses
> * Support to create directories
> * The operations supported by the directory include modification and
> deletion
>
> ## Task Definition Page
>
> * Show subcategories and subtasks
> * Directory does not support any operations
>
> # Metadata
> * Add column of `t_ds_process_definition`
>   * `parent_directory_code`: the code of the parent directory or 0 if the
> parent directory is the root of the project
> * Add `t_ds_process_definition_directory`
>   * id
>   * name
>   * code
>   * parent_directory_code:  the code of the parent directory or 0 if the
> parent directory is the root of the project
>   * project_code
>   * user_id
>   * ...
>
> The current solution is to add a new table
> `t_ds_process_definition_directory`  for storing directoy entities, and add
> a column in `t_ds_process_definition` to store directory codes.
>
> If we do not add a new table (for example, just add a tag in the process
> definition), we cannot save the directory created by the user. And it will
> also become very complicated when querying workflows and subdirectories
> under a certain directory.
>
> # RESTful API
> ## New API
> * createProcessDefinitionDirectory
> * updateProcessDefinitionDirectory
> * deleteProcessDefinitionDirectory
>
> ## Modify API
> ### createProcessDefinition
> * Add `parent_directory_code`, which means creating a workflow in a certain
> directory
> ### queryProcessDefinitionListPaging
> * Add `parent_directory_code` to return all process definition and
> subdirectories under a certain directory
> * Add an `isDirectory` field to the returned result entity, indicating
> whether it is a directory
> ### queryTaskDefinitionListPaging
> * A directory of a task is consistent with the workflow it belongs to
> * Add `parent_directory_code` to return all task definition and
> subdirectories under a certain directory
>
>
> I gave some design in this issue. If you are interested, any comments or
> suggestions are welcome.
>
> Best Regards,
> Rick Cheng



-- 
Best Wish
—Jay Chung