You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by 3656562 <36...@qq.com> on 2020/12/09 11:17:12 UTC
回复：[DISCUSS]Code solution for project/workflow/task
Schema 3 + 1
原始邮件
发件人:leon baoleonbao@apache.org
收件人:devdev@dolphinscheduler.apache.org
发送时间:2020年12月7日(周一) 23:56
主题:[DISCUSS]Code solution for project/workflow/task
##1. Background *It is suggested to take a look at the design scheme of splitting workflow JSON* the mail link about json splitting: https://lists.apache.org/thread.html/r84ed046de63899476f067dd207496428812a8687045bb4485072ee2d%40%3Cdev.dolphinscheduler.apache.org%3E The current business coding is proposed to be designed on the basis of splitting workflow JSON, with three purposes: -The references among workflow, relationship and job, and the external reference of dependent job and sub workflow job are described by project code, workflow code and business code -It supports the version management of workflow, job and resource, and provides a stable business code -Solve the problems of workflow, job copy, migration, export and import scene ID auto increment, conflict and so on ##2. Coding design ###2.1 scheme I Project table, workflow table and job table introduce a new business code field User manual input / automatically generated by the system. Here, the UI prompts the user to input the code. Internal reference is established through the code. Subsequent modification costs are high Project code is globally unique; workflow code and job code are project level unique Because the possibility of business code modification is very low, all codes in the current design do not support modification. The purpose of code modification can be achieved through the function of export and import When the workflow and job are copied, the user needs to input the new business code, and the system will check the conflict When the workflow and job are exported and imported, the business code (which can be edited) is displayed on the UI, and the user can choose whether to change the business code; when exporting, the UI adds the option of whether to export dependent jobs at the same time When the workflow and job are migrated, the business code (which can be edited) is displayed on the UI, and the user can choose whether to change the business code; the project code can be directly updated after the migration ###2.2 scheme 2 On the basis of scheme 1, there will be three description words: code, name, description. Code and name are likely to be repeatedly entered in practical application, while description is rarely used. Therefore, the current scheme is to replace code with name without adding a code field. As for the input of English name or Chinese name, it is up to the user to decide whether to input English name or Chinese name, and whether to use description for specific function description field ###2.3 scheme 3 Project code, workflow code and job code are globally unique. They are only responsible for the uniqueness of identification data and are not related to business. They are automatically generated by the system and do not provide page modification methods. 1. Workflow, job create new workflow (task) / copy workflow from template: generate new definition_ code/task_ code 2. Export: export exports all the contents of the current project / workflow / task as a JSON file 3. Import: check the definition_code/task_code exist -If it is a duplicate of definition_code/task_code in another project , you will be prompted and the import failed. -If it is the same as the definition_code/task_code in this project , you will be prompted to override -If there is no duplicate, all codes remain unchanged ##Design relation 3 The project code field is maintained in the workflow table to identify which project the workflow belongs to The relation table maintains the item code and workflow code fields, and identifies the workflow of which project the relationship belongs The relation table maintains the project code and job code of the reference node (supports jobs that reference other projects) The item code is maintained in the operation table to identify which project the job belongs to The referenced workflow / job is determined by project code, workflow code and job code The referenced workflow is determined by item code and workflow code in sub workflow-- ==================================================================== ## 1.背景 *建议先看一下拆分工作流json的设计方案* 关联邮件地址： https://lists.apache.org/thread.html/r84ed046de63899476f067dd207496428812a8687045bb4485072ee2d%40%3Cdev.dolphinscheduler.apache.org%3E 当前业务编码是建议在拆分工作流json基础上设计的，目的有3个： - 工作流、关系、作业之间的引用，以及依赖作业、子工作流作业对外部的引用通过项目编码、工作流编码、业务编码进行描述 - 支持工作流、作业、资源的版本管理，提供一个稳定不变的业务编码 - 解决工作流、作业的复制、迁移、导出导入场景的ID自增、冲突等问题 ## 2.编码设计 ### 2.1 方案一 项目表、工作流表、作业表引入一个新的业务编码字段 用户手工输入/系统自动生成，这里UI提示建议用户输入编码，内部通过编码建立引用，后续修改成本高 项目编码全局唯一；工作流编码、作业编码项目级唯一 因为业务编码修改的可能性非常低，所以当前设计所有编码都不支持修改，可以通过导出导入功能达到修改编码的目的 工作流、作业复制时，需要用户输入新业务编码，系统做冲突检查提醒 工作流、作业导出导入时，UI上显示业务编码（可编辑），用户可以选择是否变更业务编码；导出时UI增加是否同时导出依赖作业选项 工作流、作业迁移时，UI上显示业务编码（可编辑），用户可以选择是否变更业务编码；迁移直接更新项目编码就可以了 ### 2.2 方案二 在方案一基础上，会有三个描述字：code、name、description，code和name在实际应用中很可能是重复输入的，而description又很少应用，所以当前方案是用name替换code，不需要新增code字段，至于输入英文name还是中文name由用户自己决定，段具体的功能描述还是用description字段 ### 2.3 方案三 项目编码、工作流编码、作业编码全局唯一，只负责标识数据的唯一，不和业务相关，由系统自动生成，不提供页面修改方式。 1. 工作流、作业从模板创建新工作流（任务）/复制工作流: 生成新的definition_code/task_code 2. 导出： 导出将当前项目/工作流/任务所有内容导出为json文件 3. 导入： 检查definition_code/task_code是否存在 - 如果和别的项目的definition_code/task_code重复，则提示并导入失败。 - 如果和本项目内definition_code/task_code重复，则提示覆盖 - 如果没有重复的，则所有code不变导入 ## 3.引用关系设计 工作流表维护项目编码字段，标识工作流归属于哪个项目 关系表维护项目编码、工作流编码字段，标识关系归属于哪个项目的工作流 关系表维护引用节点的项目编码、作业编码（支持引用其他项目的作业） 作业表维护项目编码，标识作业归属于哪个项目 依赖作业内部通过项目编码、工作流编码、作业编码确定引用的工作流/作业 子工作流内部通过项目编码、工作流编码确定引用的工作流 DolphinScheduler(Incubator) PPMC BaoLiang 鲍亮 leonbao@apache.org
回复： [DISCUSS]Code solution for project/workflow/task
Posted by yongqing <59...@qq.com>.
+1
------------------ 原始邮件 ------------------
发件人: "dev" <dailidong66@gmail.com>;
发送时间: 2020年12月15日(星期二) 下午3:53
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re: [DISCUSS]Code solution for project/workflow/task
hi developers，
8 days have passed. we have reached a conclusion that Schema 3 will be
our final solution for the project/workflow/task.
thanks everybody
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
On Wed, Dec 9, 2020 at 7:18 PM 3656562 <3656562@qq.com> wrote:
> Schema 3 + 1
>
>
> 原始邮件
> 发件人:leon baoleonbao@apache.org
> 收件人:devdev@dolphinscheduler.apache.org
> 发送时间:2020年12月7日(周一) 23:56
> 主题:[DISCUSS]Code solution for project/workflow/task
>
>
> ##1. Background *It is suggested to take a look at the design scheme of
> splitting workflow JSON* the mail link about json splitting:
> https://lists.apache.org/thread.html/r84ed046de63899476f067dd207496428812a8687045bb4485072ee2d%40%3Cdev.dolphinscheduler.apache.org%3E
> The current business coding is proposed to be designed on the basis of
> splitting workflow JSON, with three purposes: -The references among
> workflow, relationship and job, and the external reference of dependent job
> and sub workflow job are described by project code, workflow code and
> business code -It supports the version management of workflow, job and
> resource, and provides a stable business code -Solve the problems of
> workflow, job copy, migration, export and import scene ID auto increment,
> conflict and so on ##2. Coding design ###2.1 scheme I Project table,
> workflow table and job table introduce a new business code field User
> manual input / automatically generated by the system. Here, the UI prompts
> the user to input the code. Internal reference is established through the
> code. Subsequent modification costs are high Project code is globally
> unique; workflow code and job code are project level unique Because the
> possibility of business code modification is very low, all codes in the
> current design do not support modification. The purpose of code
> modification can be achieved through the function of export and import When
> the workflow and job are copied, the user needs to input the new business
> code, and the system will check the conflict When the workflow and job are
> exported and imported, the business code (which can be edited) is displayed
> on the UI, and the user can choose whether to change the business code;
> when exporting, the UI adds the option of whether to export dependent jobs
> at the same time When the workflow and job are migrated, the business code
> (which can be edited) is displayed on the UI, and the user can choose
> whether to change the business code; the project code can be directly
> updated after the migration ###2.2 scheme 2 On the basis of scheme 1, there
> will be three description words: code, name, description. Code and name are
> likely to be repeatedly entered in practical application, while description
> is rarely used. Therefore, the current scheme is to replace code with name
> without adding a code field. As for the input of English name or Chinese
> name, it is up to the user to decide whether to input English name or
> Chinese name, and whether to use description for specific function
> description field ###2.3 scheme 3 Project code, workflow code and job code
> are globally unique. They are only responsible for the uniqueness of
> identification data and are not related to business. They are automatically
> generated by the system and do not provide page modification methods. 1.
> Workflow, job create new workflow (task) / copy workflow from template:
> generate new definition_ code/task_ code 2. Export: export exports all the
> contents of the current project / workflow / task as a JSON file 3. Import:
> check the definition_code/task_code exist -If it is a duplicate of
> definition_code/task_code in another project , you will be prompted and the
> import failed. -If it is the same as the definition_code/task_code in this
> project , you will be prompted to override -If there is no duplicate, all
> codes remain unchanged ##Design relation 3 The project code field is
> maintained in the workflow table to identify which project the workflow
> belongs to The relation table maintains the item code and workflow code
> fields, and identifies the workflow of which project the relationship
> belongs The relation table maintains the project code and job code of the
> reference node (supports jobs that reference other projects) The item code
> is maintained in the operation table to identify which project the job
> belongs to The referenced workflow / job is determined by project code,
> workflow code and job code The referenced workflow is determined by item
> code and workflow code in sub workflow--
> ==================================================================== ##
> 1.背景 *建议先看一下拆分工作流json的设计方案* 关联邮件地址：
> https://lists.apache.org/thread.html/r84ed046de63899476f067dd207496428812a8687045bb4485072ee2d%40%3Cdev.dolphinscheduler.apache.org%3E
> 当前业务编码是建议在拆分工作流json基础上设计的，目的有3个： -
> 工作流、关系、作业之间的引用，以及依赖作业、子工作流作业对外部的引用通过项目编码、工作流编码、业务编码进行描述 -
> 支持工作流、作业、资源的版本管理，提供一个稳定不变的业务编码 - 解决工作流、作业的复制、迁移、导出导入场景的ID自增、冲突等问题 ## 2.编码设计
> ### 2.1 方案一 项目表、工作流表、作业表引入一个新的业务编码字段
> 用户手工输入/系统自动生成，这里UI提示建议用户输入编码，内部通过编码建立引用，后续修改成本高 项目编码全局唯一；工作流编码、作业编码项目级唯一
> 因为业务编码修改的可能性非常低，所以当前设计所有编码都不支持修改，可以通过导出导入功能达到修改编码的目的
> 工作流、作业复制时，需要用户输入新业务编码，系统做冲突检查提醒
> 工作流、作业导出导入时，UI上显示业务编码（可编辑），用户可以选择是否变更业务编码；导出时UI增加是否同时导出依赖作业选项
> 工作流、作业迁移时，UI上显示业务编码（可编辑），用户可以选择是否变更业务编码；迁移直接更新项目编码就可以了 ### 2.2 方案二
> 在方案一基础上，会有三个描述字：code、name、description，code和name在实际应用中很可能是重复输入的，而description又很少应用，所以当前方案是用name替换code，不需要新增code字段，至于输入英文name还是中文name由用户自己决定，段具体的功能描述还是用description字段
> ### 2.3 方案三 项目编码、工作流编码、作业编码全局唯一，只负责标识数据的唯一，不和业务相关，由系统自动生成，不提供页面修改方式。 1.
> 工作流、作业从模板创建新工作流（任务）/复制工作流: 生成新的definition_code/task_code 2. 导出：
> 导出将当前项目/工作流/任务所有内容导出为json文件 3. 导入： 检查definition_code/task_code是否存在 -
> 如果和别的项目的definition_code/task_code重复，则提示并导入失败。 -
> 如果和本项目内definition_code/task_code重复，则提示覆盖 - 如果没有重复的，则所有code不变导入 ## 3.引用关系设计
> 工作流表维护项目编码字段，标识工作流归属于哪个项目 关系表维护项目编码、工作流编码字段，标识关系归属于哪个项目的工作流
> 关系表维护引用节点的项目编码、作业编码（支持引用其他项目的作业） 作业表维护项目编码，标识作业归属于哪个项目
> 依赖作业内部通过项目编码、工作流编码、作业编码确定引用的工作流/作业 子工作流内部通过项目编码、工作流编码确定引用的工作流
> DolphinScheduler(Incubator) PPMC BaoLiang 鲍亮 leonbao@apache.org
Re: [DISCUSS]Code solution for project/workflow/task
Posted by lidong dai <da...@gmail.com>.
hi developers，
8 days have passed. we have reached a conclusion that Schema 3 will be
our final solution for the project/workflow/task.
thanks everybody
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
On Wed, Dec 9, 2020 at 7:18 PM 3656562 <36...@qq.com> wrote:
> Schema 3 + 1
>
>
> 原始邮件
> 发件人:leon baoleonbao@apache.org
> 收件人:devdev@dolphinscheduler.apache.org
> 发送时间:2020年12月7日(周一) 23:56
> 主题:[DISCUSS]Code solution for project/workflow/task
>
>
> ##1. Background *It is suggested to take a look at the design scheme of
> splitting workflow JSON* the mail link about json splitting:
> https://lists.apache.org/thread.html/r84ed046de63899476f067dd207496428812a8687045bb4485072ee2d%40%3Cdev.dolphinscheduler.apache.org%3E
> The current business coding is proposed to be designed on the basis of
> splitting workflow JSON, with three purposes: -The references among
> workflow, relationship and job, and the external reference of dependent job
> and sub workflow job are described by project code, workflow code and
> business code -It supports the version management of workflow, job and
> resource, and provides a stable business code -Solve the problems of
> workflow, job copy, migration, export and import scene ID auto increment,
> conflict and so on ##2. Coding design ###2.1 scheme I Project table,
> workflow table and job table introduce a new business code field User
> manual input / automatically generated by the system. Here, the UI prompts
> the user to input the code. Internal reference is established through the
> code. Subsequent modification costs are high Project code is globally
> unique; workflow code and job code are project level unique Because the
> possibility of business code modification is very low, all codes in the
> current design do not support modification. The purpose of code
> modification can be achieved through the function of export and import When
> the workflow and job are copied, the user needs to input the new business
> code, and the system will check the conflict When the workflow and job are
> exported and imported, the business code (which can be edited) is displayed
> on the UI, and the user can choose whether to change the business code;
> when exporting, the UI adds the option of whether to export dependent jobs
> at the same time When the workflow and job are migrated, the business code
> (which can be edited) is displayed on the UI, and the user can choose
> whether to change the business code; the project code can be directly
> updated after the migration ###2.2 scheme 2 On the basis of scheme 1, there
> will be three description words: code, name, description. Code and name are
> likely to be repeatedly entered in practical application, while description
> is rarely used. Therefore, the current scheme is to replace code with name
> without adding a code field. As for the input of English name or Chinese
> name, it is up to the user to decide whether to input English name or
> Chinese name, and whether to use description for specific function
> description field ###2.3 scheme 3 Project code, workflow code and job code
> are globally unique. They are only responsible for the uniqueness of
> identification data and are not related to business. They are automatically
> generated by the system and do not provide page modification methods. 1.
> Workflow, job create new workflow (task) / copy workflow from template:
> generate new definition_ code/task_ code 2. Export: export exports all the
> contents of the current project / workflow / task as a JSON file 3. Import:
> check the definition_code/task_code exist -If it is a duplicate of
> definition_code/task_code in another project , you will be prompted and the
> import failed. -If it is the same as the definition_code/task_code in this
> project , you will be prompted to override -If there is no duplicate, all
> codes remain unchanged ##Design relation 3 The project code field is
> maintained in the workflow table to identify which project the workflow
> belongs to The relation table maintains the item code and workflow code
> fields, and identifies the workflow of which project the relationship
> belongs The relation table maintains the project code and job code of the
> reference node (supports jobs that reference other projects) The item code
> is maintained in the operation table to identify which project the job
> belongs to The referenced workflow / job is determined by project code,
> workflow code and job code The referenced workflow is determined by item
> code and workflow code in sub workflow--
> ==================================================================== ##
> 1.背景 *建议先看一下拆分工作流json的设计方案* 关联邮件地址：
> https://lists.apache.org/thread.html/r84ed046de63899476f067dd207496428812a8687045bb4485072ee2d%40%3Cdev.dolphinscheduler.apache.org%3E
> 当前业务编码是建议在拆分工作流json基础上设计的，目的有3个： -
> 工作流、关系、作业之间的引用，以及依赖作业、子工作流作业对外部的引用通过项目编码、工作流编码、业务编码进行描述 -
> 支持工作流、作业、资源的版本管理，提供一个稳定不变的业务编码 - 解决工作流、作业的复制、迁移、导出导入场景的ID自增、冲突等问题 ## 2.编码设计
> ### 2.1 方案一 项目表、工作流表、作业表引入一个新的业务编码字段
> 用户手工输入/系统自动生成，这里UI提示建议用户输入编码，内部通过编码建立引用，后续修改成本高 项目编码全局唯一；工作流编码、作业编码项目级唯一
> 因为业务编码修改的可能性非常低，所以当前设计所有编码都不支持修改，可以通过导出导入功能达到修改编码的目的
> 工作流、作业复制时，需要用户输入新业务编码，系统做冲突检查提醒
> 工作流、作业导出导入时，UI上显示业务编码（可编辑），用户可以选择是否变更业务编码；导出时UI增加是否同时导出依赖作业选项
> 工作流、作业迁移时，UI上显示业务编码（可编辑），用户可以选择是否变更业务编码；迁移直接更新项目编码就可以了 ### 2.2 方案二
> 在方案一基础上，会有三个描述字：code、name、description，code和name在实际应用中很可能是重复输入的，而description又很少应用，所以当前方案是用name替换code，不需要新增code字段，至于输入英文name还是中文name由用户自己决定，段具体的功能描述还是用description字段
> ### 2.3 方案三 项目编码、工作流编码、作业编码全局唯一，只负责标识数据的唯一，不和业务相关，由系统自动生成，不提供页面修改方式。 1.
> 工作流、作业从模板创建新工作流（任务）/复制工作流: 生成新的definition_code/task_code 2. 导出：
> 导出将当前项目/工作流/任务所有内容导出为json文件 3. 导入： 检查definition_code/task_code是否存在 -
> 如果和别的项目的definition_code/task_code重复，则提示并导入失败。 -
> 如果和本项目内definition_code/task_code重复，则提示覆盖 - 如果没有重复的，则所有code不变导入 ## 3.引用关系设计
> 工作流表维护项目编码字段，标识工作流归属于哪个项目 关系表维护项目编码、工作流编码字段，标识关系归属于哪个项目的工作流
> 关系表维护引用节点的项目编码、作业编码（支持引用其他项目的作业） 作业表维护项目编码，标识作业归属于哪个项目
> 依赖作业内部通过项目编码、工作流编码、作业编码确定引用的工作流/作业 子工作流内部通过项目编码、工作流编码确定引用的工作流
> DolphinScheduler(Incubator) PPMC BaoLiang 鲍亮 leonbao@apache.org