You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "CloudSen (via GitHub)" <gi...@apache.org> on 2024/07/25 06:25:44 UTC
[I] [Bug] [MASTER] When workflow are frequently scheduled, the state synchronization mechanism has critical errors. [dolphinscheduler]
CloudSen opened a new issue, #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
状态的转换是一个过程，但是现在流程之间的通知机制依赖瞬时状态。
State transition is a process, but currently, the notification mechanism between processes relies on instantaneous states.
在此，举一个例子，如果工作流也有超时配置，这个例子里，工作流超时后，下一个串行等待的工作流可能会卡住。
For example, let's imagine if the workflows also had timeout configurations，in this scenario, if a workflow times out, the next serially waiting workflow might get stuck.
```
@startuml
StateWheelExecuteThread --> WorkflowTimeoutStateEventHandler: send process A PROCESS_TIMEOUT event
WorkflowTimeoutStateEventHandler -> WorkflowExecuteRunnable: processTimeout
WorkflowExecuteRunnable --> WorkflowStateEventHandler: send process A STOP event
WorkflowStateEventHandler -> WorkflowExecuteRunnable: endProcess
WorkflowExecuteRunnable -> WorkflowExecuteRunnable:checkSerialProcess
WorkflowExecuteRunnable --> ProcessServiceImpl: send process B RECOVER_SERIAL_WAIT command
ProcessServiceImpl -> ProcessServiceImpl: STEP 1： handleCommand(if state of process A is RUNNING_PROCESS_STATE, state of proces B will change back to SERIAL_WAIT)
WorkflowStateEventHandler -> WorkflowStateEventHandler: STEP 2: update process A to STOP
@enduml
```
由于STEP 1和STEP 2的顺序无法保证，会导致后续所有实例都卡在“串行等待”状态。
Since the order of STEP 1 and STEP 2 cannot be guaranteed now, all subsequent instances might get stuck in the "serial wait" state.
### What you expected to happen
恢复后续实例前，需要保证自己的状态更新完毕
Before resuming subsequent instances, you need to ensure that your own state has been fully updated.
### How to reproduce
- Create Workflow A and Workflow B, both scheduled to run every minute.
- In Workflow A, create a SUB_PROCESS task node that references Workflow B.
- Online both Workflow A and Workflow B, and online the schedules.
- Observe the state changes.
### Anything else
_No response_
### Version
dev
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2445563175
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2708583049
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "CloudSen (via GitHub)" <gi...@apache.org>.
CloudSen commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2295682973
> Right now the serial wait implementation is really unstable, there are a lot of case will cause it doesn't work well. e.g.
>
> 1. Concurrent trigger will cause multiple workflow instance running which should in serial wait.
> 2. Notify failed might cause the origin workflow instance cannot finish.
> 3. The workflow should deal with the notify logic, this make the workflow instance state transition more complex.
>
> It's better to refactor this, use a global SerialWaitCoordinator to notify the serial wait workflow instance, the origin workflow instance don't need to care whether it need to do notification.
Agree with your opinion, I have thought about a coordinator similar to SerialWaitCoordinator instead of notifying tasks themselves.
- Is there a refactoring plan for this?
- How should this bug in 3.1.x be fixed? As I said, currently I can temporarily solve most problems by re-sending the RECOVER_SERIAL_WAIT command
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2639325992
This issue doesn's fixed, I reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "ZhaoRuidong (via GitHub)" <gi...@apache.org>.
ZhaoRuidong commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2272524983
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "Gallardot (via GitHub)" <gi...@apache.org>.
Gallardot commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2363776089
In version 3.2.1, I fixed a similar issue, which might be helpful for the current issue. You can refer to it.
#15270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2362473876
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "CloudSen (via GitHub)" <gi...@apache.org>.
CloudSen commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2366189262
@Gallardot Yes, I’ve already applied patch #15270. But as I mentioned earlier, the state change is always a process. Opening a new transaction only speeds up the state update, but the same issue still exists in concurrent scenarios. A synchronizer is still needed for serially running workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2727092091
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2295289850
Right now the serial wait implementation is really unstable, there are a lot of case will cause it doesn't work well.
e.g.
1. Concurrent trigger will cause multiple workflow instance running which should in serial wait.
2. Notify failed might cause the origin workflow instance cannot finish.
3. The workflow should deal with the notify logic, this make the workflow instance state transition more complex.
It's better to refactor this, use a global SerialWaitCoordinator to notify the serial wait workflow instance, the origin workflow instance don't need to care whether it need to do notification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "CloudSen (via GitHub)" <gi...@apache.org>.
CloudSen commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2366306798
@Gallardot Here’s another example for `saveSerialProcess`: In concurrent scenarios (multiple parent workflows with the same scheduling cycle reference the same sub-workflow), multiple start_workflow commands will be consumed by different masters at the same time. Then, in `saveSerialProcess`, each will detect that there are no running instances simultaneously(transaction have not been committed yet), and multiple start events for the workflow will be triggered at the same time. This ultimately leads to multiple workflow instances running concurrently, which is incorrect when they are supposed to be processed serially.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "CloudSen (via GitHub)" <gi...@apache.org>.
CloudSen commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2277155765
Currently, I resolved this issue by retrying the RECOVER_SERIAL_WAIT command when the previous workflow was not fully updated, and it works well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16369: [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck
URL: https://github.com/apache/dolphinscheduler/issues/16369
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2430555650
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16369: [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck
URL: https://github.com/apache/dolphinscheduler/issues/16369
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When workflow are frequently scheduled, the state synchronization mechanism has critical errors. [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2249553292
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
State transition is a process, but now the notification mechanism between processes relies on transient state.
State transition is a process, but currently, the notification mechanism between processes relies on instantaneous states.
Here, take an example, if the workflow also has a timeout configuration, in this example, after the workflow times out, the next serially waiting workflow may be stuck.
For example, let's imagine if the workflows also had timeout configurations, in this scenario, if a workflow times out, the next serially waiting workflow might get stuck.
```
@startuml
StateWheelExecuteThread --> WorkflowTimeoutStateEventHandler: send process A PROCESS_TIMEOUT event
WorkflowTimeoutStateEventHandler -> WorkflowExecuteRunnable: processTimeout
WorkflowExecuteRunnable --> WorkflowStateEventHandler: send process A STOP event
WorkflowStateEventHandler -> WorkflowExecuteRunnable: endProcess
WorkflowExecuteRunnable -> WorkflowExecuteRunnable:checkSerialProcess
WorkflowExecuteRunnable --> ProcessServiceImpl: send process B RECOVER_SERIAL_WAIT command
ProcessServiceImpl -> ProcessServiceImpl: STEP 1: handleCommand(if state of process A is RUNNING_PROCESS_STATE, state of process B will change back to SERIAL_WAIT)
WorkflowStateEventHandler -> WorkflowStateEventHandler: STEP 2: update process A to STOP
@enduml
```
Since the order of STEP 1 and STEP 2 cannot be guaranteed, all subsequent instances will be stuck in the "serial waiting" state.
Since the order of STEP 1 and STEP 2 cannot be guaranteed now, all subsequent instances might get stuck in the "serial wait" state.
### What you expected to happen
Before restoring subsequent instances, you need to ensure that your status has been updated.
Before resuming subsequent instances, you need to ensure that your own state has been fully updated.
### How to reproduce
- Create Workflow A and Workflow B, both scheduled to run every minute.
- In Workflow A, create a SUB_PROCESS task node that references Workflow B.
- Online both Workflow A and Workflow B, and online the schedules.
- Observe the state changes.
### Anything else
_No response_
### Version
dev
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, the state synchronization mechanism has critical errors. [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2249604813
cc @ruanwenjun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, the state synchronization mechanism has critical errors. [dolphinscheduler]
Posted by "CloudSen (via GitHub)" <gi...@apache.org>.
CloudSen commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2251789482
Maybe we need a cache queue for the RECOVER_SERIAL_WAIT command? If the command execution finds that the previous workflow has not been updated to a completed state in time, it should be put back in the queue and checked again after a while.
When the previous workflow state has been fully updated, this command will successfully consume from the queue and transition from the SERIAL_WAIT state to the SUBMITTED_SUCCESS state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [MASTER] When Serial Wait workflow are frequently scheduled, serially waiting workflow might get stuck [dolphinscheduler]
Posted by "hujian0401 (via GitHub)" <gi...@apache.org>.
hujian0401 commented on issue #16369:
URL: https://github.com/apache/dolphinscheduler/issues/16369#issuecomment-2636218650
@CloudSen Do you already have any workaround to solve this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org