Re:[DISCUSS]The global params that Task can change

[CalvinKirs <ac...@163.com>]

I think it's good.


Looking forward to your PR submission.


Best  wishes！
CalvinKirs


On 01/8/2021 10:55，Xingjie Wang(联通集团联通大数据有限公司本部)<wa...@chinaunicom.cn> wrote：

Hi Dev Team

 

The scheme of The global params that Task can change as blow。

Here is the ISSUE about this DISCUSS。

https://github.com/apache/incubator-dolphinscheduler/issues/3805

 

Discussion on the implementation scheme of global params (parameter transfer between nodes)

 

At present, we have implemented the global params function

1. The params takes effect in the process instance, each node can accept the variable, SQL and SHELL nodes can modify the params value, and subsequent nodes use the params as the modified value. (requirement 1)

2. The subprocess node passes in variables, and the subprocess can receive params values (requirement 2)

 

The detail of the scheme  as follows

 

a： requirement 1

1. When saving workflow, users will add global params definition (global definition) in the pop-up box after clicking save button, and save the defined params name and default value (if there is an initial value) to t_ds_process_definition.global_Params field (using the existing function). When saving, it will verify that the params names used by the node are all from global definition.

2. When the workflow is called up for instantiation, get the value of globalparams from definition and initialize the params value to t_ds_process_instance.global_Params field. (at this time, the initial value of the workflow instance is the default value defined globally)

3. About input parameters: when creating TaskInstance, take variables from localparam, and set the value from t_ds_process_instance.global_params to localparam. After storing the task instance, the task instance is saved in the database and distributed to the worker, and the subsequent parameter processing follows the original part.

4. About output parameters: after the task execution of SQL node and SHELL node is finished(SQL query results, only support one field one value, if multiple fields or rows of data will be formatted as JSON;The SHELL uses the output of the last line as the result), write the result of task execution into TaskExecuteResponseCommand, return to master together with the worker execution result information, and the master will update the output t_ds_process_instance.global_params and update it to the database.

 

b： requirement 2

1. About subprocess node: judge whether there is parameter passing when generating subprocess command. If there is parameter passing, the parameter will be written into the command table.

2. When the subprocess is instantiated, judge that the command contains the parameter passed down by the main process, and update the value to t_ds_process_instance.global_params field. The subsequent part is consistent with requirement 1

To implement requirement 2, you need to add COLUMN son_Params in t_ds_command table.

 

c： requirement 3

In addition, in order to realize the function of modifying parameters easily when the workflow is executed manually

1. When you click the [run] button, the window to modify the global params value will pop up, and save it into command.son_Params . The follow-up process is consistent with requirement 2

 

 

The above three demand plans are submitted in two versions. The first version completes demand 1, and the second version completes demand 2 and 3

 

关于全局变量(节点间参数传递)实现方案的讨论：
目前我们已经实现了全局变量功能，具体为：
1.变量在流程实例内生效，每个节点可接受变量，sql和shell节点可修改变量值，后续节点在使用该变量时为修改后的值。（功能1）
2.子流程节点传入变量，子流程可以接受到变量值（功能2）
实现方案为：

一：功能1

1.        用户在保存工作流时，会在点击保存按钮后的弹出框内添加全局变量定义（全局定义），将定义的变量名和默认值（如果有初始值的话）保存到t_ds_process_definition.global_params字段（沿用已有功能），保存时会校验节点所使用的的变量名全部来自【全局定义】。

2.        在工作流被调起实例化的时候，从definition中获取globalParams的值，并将变量值初始化到t_ds_process_instance.global_params字段中。（此时工作流实例的初始值为全局定义的默认值）

3.        关于入参：在创建taskInstance时，从localParam中取出变量，并将t_ds_process_instance.global_params中最新的值更新至localParam中。存入taskInstance，后续将taskInstance保存数据库，并分发给worker，后续参数处理沿用原有部分。

4.        关于出参：在sql节点和shell节点任务执行结束后，将任务执行的结果（sql查询的结果，只支持一个字段一个值，如果是多个字段或多行数据会格式化为json；shell使用最后一行的输出作为结果）写入TaskExecuteResponseCommand，随worker执行结果信息一起返回master，master将出参更新t_ds_process_instance.global_params，并将其更新至数据库。

二：功能2

1.      关于子流程节点：生成子流程的command时判断是否有参数传递，如果有参数传递，会将参数写入到command表中。

2.      在子流程实例化时判断command中包含主流程向下传递的参数，将该值更新至t_ds_process_instance.global_params字段中。后续部分与需求1一致

实现功能2需要在command中增加son­_params字段。

三：另外为了实现在手动执行工作流时方便修改参数设计功能3：

1．在点击【运行】按钮时，弹出修改全局变量值的窗口，保存进command. son­_params中。后续流程与需求2一致



以上三个需求计划两个版本提交，第一个版本完成需求1，第二个版本完成需求2、3

 

 

以下为该方案流程图：

 

附件为该流程图的Visio版。

Welcome more discussion and suggestions.

 

如果您错误接收了该邮件，请通过电子邮件立即通知我们。请回复邮件到 hqs-spmc@chinaunicom.cn，即可以退订此邮件。我们将立即将您的信息从我们的发送目录中删除。 If you have received this email in error please notify us immediately by e-mail. Please reply to hqs-spmc@chinaunicom.cn ,you can unsubscribe from this mail. We will immediately remove your information from send catalogue of our.

Re: [DISCUSS]The global params that Task can change

[lidong dai <da...@gmail.com>]

hi
  Looking forward to your implemention.


Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------


On Fri, Jan 8, 2021 at 2:31 PM CalvinKirs <ac...@163.com> wrote:

> I think it's good.
>
>
> Looking forward to your PR submission.
>
>
> Best  wishes！
> CalvinKirs
>
>
> On 01/8/2021 10:55，Xingjie Wang(联通集团联通大数据有限公司本部)<wa...@chinaunicom.cn>
> wrote：
>
> Hi Dev Team
>
>
>
> The scheme of The global params that Task can change as blow。
>
> Here is the ISSUE about this DISCUSS。
>
> https://github.com/apache/incubator-dolphinscheduler/issues/3805
>
>
>
> Discussion on the implementation scheme of global params (parameter
> transfer between nodes)
>
>
>
> At present, we have implemented the global params function
>
> 1. The params takes effect in the process instance, each node can accept
> the variable, SQL and SHELL nodes can modify the params value, and
> subsequent nodes use the params as the modified value. (requirement 1)
>
> 2. The subprocess node passes in variables, and the subprocess can receive
> params values (requirement 2)
>
>
>
> The detail of the scheme  as follows
>
>
>
> a： requirement 1
>
> 1. When saving workflow, users will add global params definition (global
> definition) in the pop-up box after clicking save button, and save the
> defined params name and default value (if there is an initial value) to
> t_ds_process_definition.global_Params field (using the existing function).
> When saving, it will verify that the params names used by the node are all
> from global definition.
>
> 2. When the workflow is called up for instantiation, get the value of
> globalparams from definition and initialize the params value to
> t_ds_process_instance.global_Params field. (at this time, the initial value
> of the workflow instance is the default value defined globally)
>
> 3. About input parameters: when creating TaskInstance, take variables from
> localparam, and set the value from t_ds_process_instance.global_params to
> localparam. After storing the task instance, the task instance is saved in
> the database and distributed to the worker, and the subsequent parameter
> processing follows the original part.
>
> 4. About output parameters: after the task execution of SQL node and SHELL
> node is finished(SQL query results, only support one field one value, if
> multiple fields or rows of data will be formatted as JSON;The SHELL uses
> the output of the last line as the result), write the result of task
> execution into TaskExecuteResponseCommand, return to master together with
> the worker execution result information, and the master will update the
> output t_ds_process_instance.global_params and update it to the database.
>
>
>
> b： requirement 2
>
> 1. About subprocess node: judge whether there is parameter passing when
> generating subprocess command. If there is parameter passing, the parameter
> will be written into the command table.
>
> 2. When the subprocess is instantiated, judge that the command contains
> the parameter passed down by the main process, and update the value to
> t_ds_process_instance.global_params field. The subsequent part is
> consistent with requirement 1
>
> To implement requirement 2, you need to add COLUMN son_Params in
> t_ds_command table.
>
>
>
> c： requirement 3
>
> In addition, in order to realize the function of modifying parameters
> easily when the workflow is executed manually
>
> 1. When you click the [run] button, the window to modify the global params
> value will pop up, and save it into command.son_Params . The follow-up
> process is consistent with requirement 2
>
>
>
>
>
> The above three demand plans are submitted in two versions. The first
> version completes demand 1, and the second version completes demand 2 and 3
>
>
>
> 关于全局变量(节点间参数传递)实现方案的讨论：
> 目前我们已经实现了全局变量功能，具体为：
> 1.变量在流程实例内生效，每个节点可接受变量，sql和shell节点可修改变量值，后续节点在使用该变量时为修改后的值。（功能1）
> 2.子流程节点传入变量，子流程可以接受到变量值（功能2）
> 实现方案为：
>
> 一：功能1
>
> 1.
> 用户在保存工作流时，会在点击保存按钮后的弹出框内添加全局变量定义（全局定义），将定义的变量名和默认值（如果有初始值的话）保存到t_ds_process_definition.global_params字段（沿用已有功能），保存时会校验节点所使用的的变量名全部来自【全局定义】。
>
> 2.
> 在工作流被调起实例化的时候，从definition中获取globalParams的值，并将变量值初始化到t_ds_process_instance.global_params字段中。（此时工作流实例的初始值为全局定义的默认值）
>
> 3.
> 关于入参：在创建taskInstance时，从localParam中取出变量，并将t_ds_process_instance.global_params中最新的值更新至localParam中。存入taskInstance，后续将taskInstance保存数据库，并分发给worker，后续参数处理沿用原有部分。
>
> 4.
> 关于出参：在sql节点和shell节点任务执行结束后，将任务执行的结果（sql查询的结果，只支持一个字段一个值，如果是多个字段或多行数据会格式化为json；shell使用最后一行的输出作为结果）写入TaskExecuteResponseCommand，随worker执行结果信息一起返回master，master将出参更新t_ds_process_instance.global_params，并将其更新至数据库。
>
> 二：功能2
>
> 1.      关于子流程节点：生成子流程的command时判断是否有参数传递，如果有参数传递，会将参数写入到command表中。
>
> 2.
> 在子流程实例化时判断command中包含主流程向下传递的参数，将该值更新至t_ds_process_instance.global_params字段中。后续部分与需求1一致
>
> 实现功能2需要在command中增加son­_params字段。
>
> 三：另外为了实现在手动执行工作流时方便修改参数设计功能3：
>
> 1．在点击【运行】按钮时，弹出修改全局变量值的窗口，保存进command. son­_params中。后续流程与需求2一致
>
>
>
> 以上三个需求计划两个版本提交，第一个版本完成需求1，第二个版本完成需求2、3
>
>
>
>
>
> 以下为该方案流程图：
>
>
>
> 附件为该流程图的Visio版。
>
> Welcome more discussion and suggestions.
>
>
>
> 如果您错误接收了该邮件，请通过电子邮件立即通知我们。请回复邮件到 hqs-spmc@chinaunicom.cn，即可以退订此邮件。我们将立即将您的信息从我们的发送目录中删除。
> If you have received this email in error please notify us immediately by
> e-mail. Please reply to hqs-spmc@chinaunicom.cn ,you can unsubscribe from
> this mail. We will immediately remove your information from send catalogue
> of our.