serial processInstance execution

["Xingjie Wang (联通集团联通数字科技有限公司)" <wa...@chinaunicom.cn>]

Hi community:


The current workflow instances are scheduled to run in parallel, plan to submit workflow serial functionality.

The functions include:
1. ProcessInstance of processDefinition that defines a serial plan will execute sequentially, that is, there will be at most one processInstance running.
2. Serial execution is divided into three policies (the processInstance described below belongs to the same processDefinition and defines the corresponding serial rules)

(1) Serial waiting: When a new processInstance is generated, it is executed in sequence according to the order of the progress queue.
(2) Serial discard: When a new processInstance is generated, if a processInstance (under the same processDefinition) is found , abandon thisprocessInstance , the state is stopped.
(3) Serial priority: When a new processInstance is generated, if a running instance is found, stop this running processInstance.

Implementation:

Serial wait:
1. After the master reads the command, it generates a processInstance according to the existing logic,
2. Before saving, determine whether the serial plan is defined in the processDefinition, if not defined, continue to execute according to the original logic. If defined: Determine whether the processInstance under the processDefinition has a processInstance with the status of [Running: 1] or [Waiting Queue: 14] (newly added), and if there is, update the status to [Waiting Queue].
3. Save the processInstanceId to the processInstance where the nextProcessInstanceId field (newly added) is empty (here, if the processInstance just ends running, there will be problems, so before the end of the processInstance, judge if the serial plan is defined and the nextProcessInstanceId is empty, then Find the processInstance with the status of [Waiting Queue: 14] under the processDefinition and start it).
4. After the processInstance is executed, it is judged that if nextProcessInstanceId is not empty, a command is generated to resume the operation of the processInstance.
Note: The operation of Part 3 will have concurrency after 2.0, so the database optimistic lock is used to solve the concurrency problem when saving the nextProcessInstanceId field.

Serial discard:
1. After the master reads the command, it generates a processInstance according to the existing logic,
2. Before saving, determine whether the serial plan is defined in the processDefinition, if not defined, continue execution according to the original logic. If defined: Check whether there is a running process, and if there is a current process state, set it to stop.

Serial priority:
1. After the master reads the command, it generates a processInstance according to the existing logic,
2. Before saving, determine whether the serial plan is defined in the processDefinition, if not defined, continue execution according to the original logic. If defined: Check whether there is a running process, and if there is, set the running process status to stop.

Best Wishes！

--------------------------------------------------------------------------------------------------------------------------------

目前工作流实例的运行计划均为并行，计划提交工作流串行功能

功能主要包括
1.定义了串行计划的processDefinition的processInstance会顺序执行，即最多只会有一个正在运行的processInstance。
2.串行执行分为三种策略（以下所描述的processInstance在属于同一processDefinition，并且该processDefinition定义了对应串行规则）
（1）串行等待：生成新的processInstance时，按照进度队列的先后顺序依次执行。
（2）串行抛弃：生成新的processInstance时，如果发现有正在运行的工作流实例（同一个processDefinition下）放弃运行，状态为停止。
（3）串行优先：生成新的processInstance时，如果发现有正在运行的实例，停止该processInstance的运行。

实现方案：
串行等待：
1.master在读取command后，按照现有逻辑生成processInstance，
2.在保存之前判断该processDefinition中是否定义了串行计划，如果没有定义，按照原有逻辑继续执行。如果定义：判断该processDefinition下的processInstance是否有状态为【正在运行：1】或【等待队列：14】（新增）的processInstance，如果有将状态更新为【等待队列】。
3.将该processInstanceId保存到nextProcessInstanceId字段（新增）为空的processInstance中（此处如果该processInstance刚好结束运行会出现问题，所以在processInstance结束之前判断如果定义了串行计划，并且nextProcessInstanceId为空，则查找该processDefinition下状态为【等待队列：14】的processInstance并且启动）。
4.processInstance执行结束后判断nextProcessInstanceId如果不为空则生成command恢复该processInstance的运行。
注：第3部的操作在2.0后会出现并发情况，所以在保存nextProcessInstanceId字段时利用数据库乐观锁解决并发问题。

串行抛弃：
1.master在读取command后，按照现有逻辑生成processInstance，
2.在保存之前判断该processDefinition中是否定义了串行计划，如果没有定义，按照原有逻辑继续执行。如果定义：检查是否有正在运行的流程，如果有当前流程状态置为停止。

串行优先：
1.master在读取command后，按照现有逻辑生成processInstance，
2.在保存之前判断该processDefinition中是否定义了串行计划，如果没有定义，按照原有逻辑继续执行。如果定义：检查是否有正在运行的流程，如果有将正在运行的流程状态置为停止。

附件为处理串行部分的流程，如果您有更好的解决方案，或感觉本次方案存在问题请随时联系我。

祝好！
如果您错误接收了该邮件，请通过电子邮件立即通知我们。请回复邮件到 hqs-spmc@chinaunicom.cn，即可以退订此邮件。我们将立即将您的信息从我们的发送目录中删除。 If you have received this email in error please notify us immediately by e-mail. Please reply to hqs-spmc@chinaunicom.cn ,you can unsubscribe from this mail. We will immediately remove your information from send catalogue of our.

Re: serial processInstance execution

[David Dai <da...@gmail.com>]

good job
I think what you have said is very comprehensive, you can start with this
proposal


Best Regards



---------------
Apache DolphinScheduler PMC Chair
David Dai
lidongdai@apache.org
Linkedin: https://www.linkedin.com/in/dailidong
Twitter: @WorkflowEasy <https://twitter.com/WorkflowEasy>
---------------


On Thu, Sep 2, 2021 at 3:48 PM Xingjie Wang(联通集团联通数字科技有限公司) <
wangxj959@chinaunicom.cn> wrote:

> Hi community:
>
>
> The current workflow instances are scheduled to run in parallel, plan to
> submit workflow serial functionality.
>
> The functions include:
> 1. ProcessInstance of processDefinition that defines a serial plan will
> execute sequentially, that is, there will be at most one processInstance
> running.
> 2. Serial execution is divided into three policies (the processInstance
> described below belongs to the same processDefinition and defines the
> corresponding serial rules)
>
> (1) Serial waiting: When a new processInstance is generated, it is
> executed in sequence according to the order of the progress queue.
> (2) Serial discard: When a new processInstance is generated, if a
> processInstance (under the same processDefinition) is found , abandon
> thisprocessInstance , the state is stopped.
> (3) Serial priority: When a new processInstance is generated, if a running
> instance is found, stop this running processInstance.
>
> Implementation:
>
> Serial wait:
> 1. After the master reads the command, it generates a processInstance
> according to the existing logic,
> 2. Before saving, determine whether the serial plan is defined in the
> processDefinition, if not defined, continue to execute according to the
> original logic. If defined: Determine whether the processInstance under the
> processDefinition has a processInstance with the status of [Running: 1] or
> [Waiting Queue: 14] (newly added), and if there is, update the status to
> [Waiting Queue].
> 3. Save the processInstanceId to the processInstance where the
> nextProcessInstanceId field (newly added) is empty (here, if the
> processInstance just ends running, there will be problems, so before the
> end of the processInstance, judge if the serial plan is defined and the
> nextProcessInstanceId is empty, then Find the processInstance with the
> status of [Waiting Queue: 14] under the processDefinition and start it).
> 4. After the processInstance is executed, it is judged that if
> nextProcessInstanceId is not empty, a command is generated to resume the
> operation of the processInstance.
> Note: The operation of Part 3 will have concurrency after 2.0, so the
> database optimistic lock is used to solve the concurrency problem when
> saving the nextProcessInstanceId field.
>
> Serial discard:
> 1. After the master reads the command, it generates a processInstance
> according to the existing logic,
> 2. Before saving, determine whether the serial plan is defined in the
> processDefinition, if not defined, continue execution according to the
> original logic. If defined: Check whether there is a running process, and
> if there is a current process state, set it to stop.
>
> Serial priority:
> 1. After the master reads the command, it generates a processInstance
> according to the existing logic,
> 2. Before saving, determine whether the serial plan is defined in the
> processDefinition, if not defined, continue execution according to the
> original logic. If defined: Check whether there is a running process, and
> if there is, set the running process status to stop.
>
> Best Wishes！
>
>
> --------------------------------------------------------------------------------------------------------------------------------
>
> 目前工作流实例的运行计划均为并行，计划提交工作流串行功能
>
> 功能主要包括
>
> 1.定义了串行计划的processDefinition的processInstance会顺序执行，即最多只会有一个正在运行的processInstance。
>
> 2.串行执行分为三种策略（以下所描述的processInstance在属于同一processDefinition，并且该processDefinition定义了对应串行规则）
> （1）串行等待：生成新的processInstance时，按照进度队列的先后顺序依次执行。
>
> （2）串行抛弃：生成新的processInstance时，如果发现有正在运行的工作流实例（同一个processDefinition下）放弃运行，状态为停止。
> （3）串行优先：生成新的processInstance时，如果发现有正在运行的实例，停止该processInstance的运行。
>
> 实现方案：
> 串行等待：
> 1.master在读取command后，按照现有逻辑生成processInstance，
>
> 2.在保存之前判断该processDefinition中是否定义了串行计划，如果没有定义，按照原有逻辑继续执行。如果定义：判断该processDefinition下的processInstance是否有状态为【正在运行：1】或【等待队列：14】（新增）的processInstance，如果有将状态更新为【等待队列】。
>
> 3.将该processInstanceId保存到nextProcessInstanceId字段（新增）为空的processInstance中（此处如果该processInstance刚好结束运行会出现问题，所以在processInstance结束之前判断如果定义了串行计划，并且nextProcessInstanceId为空，则查找该processDefinition下状态为【等待队列：14】的processInstance并且启动）。
>
> 4.processInstance执行结束后判断nextProcessInstanceId如果不为空则生成command恢复该processInstance的运行。
> 注：第3部的操作在2.0后会出现并发情况，所以在保存nextProcessInstanceId字段时利用数据库乐观锁解决并发问题。
>
> 串行抛弃：
> 1.master在读取command后，按照现有逻辑生成processInstance，
>
> 2.在保存之前判断该processDefinition中是否定义了串行计划，如果没有定义，按照原有逻辑继续执行。如果定义：检查是否有正在运行的流程，如果有当前流程状态置为停止。
>
> 串行优先：
> 1.master在读取command后，按照现有逻辑生成processInstance，
>
> 2.在保存之前判断该processDefinition中是否定义了串行计划，如果没有定义，按照原有逻辑继续执行。如果定义：检查是否有正在运行的流程，如果有将正在运行的流程状态置为停止。
>
> 附件为处理串行部分的流程，如果您有更好的解决方案，或感觉本次方案存在问题请随时联系我。
>
> 祝好！
> 如果您错误接收了该邮件，请通过电子邮件立即通知我们。请回复邮件到 hqs-spmc@chinaunicom.cn，即可以退订此邮件。我们将立即将您的信息从我们的发送目录中删除。
> If you have received this email in error please notify us immediately by
> e-mail. Please reply to hqs-spmc@chinaunicom.cn ,you can unsubscribe from
> this mail. We will immediately remove your information from send catalogue
> of our.
>