You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "crazychengmm (via GitHub)" <gi...@apache.org> on 2026/04/17 03:37:41 UTC

[I] [Bug] Potential Deadlock Risk in SubProcess Workflow: Resource Exhaustion when Worker Threads are Saturated [dolphinscheduler]

crazychengmm opened a new issue, #18176:
URL: https://github.com/apache/dolphinscheduler/issues/18176

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What happened
   
   In the current architecture of Apache DolphinScheduler, a SubProcess task behaves as a "monitor" that occupies a worker thread while waiting for the child workflow to complete.
   
   We have observed a significant stability risk in large-scale production environments:
   
   Resource Saturation: When the worker.exec.threads on all workers are fully utilized by parent workflow tasks (SubProcess nodes), no worker threads are available to pick up new tasks.
   The Deadlock: The parent workflow is stuck in the RUNNING state, waiting for the child workflow to finish. However, the child workflow is stuck in the SUBMITTED_SUCCESS or DELAY_EXECUTION state (in Master's queue) because there are no available worker threads to execute it.
   Impact: This creates a circular dependency where parent tasks hold resources needed by child tasks to release those very resources. Since many users do not configure task timeouts, this leads to permanent blockage of the entire workflow orchestration engine.
   Why this is a critical issue for Platform Providers
   As a platform provider, we support numerous business teams. It is difficult to enforce strict "Task Timeout" configurations across all user workflows. Therefore, we believe DolphinScheduler should have a robust mechanism to prevent such circular resource dependencies from affecting platform-wide availability.
   
   
   ### What you expected to happen
   
   We would appreciate the community's input on how to resolve this. Here are a few potential directions:
   
   Dedicated Thread Pool for Control Tasks: Implement a separate, dedicated thread pool for non-computing tasks (like SubProcess monitoring, Dependent tasks, etc.) that is independent of the main worker.exec.threads pool to ensure control signals are never blocked.
   Adaptive Resource Reservation: Allow the Master/Worker to reserve a small percentage of threads exclusively for "child" or "dependent" tasks to prevent total starvation.
   Automatic "Resource-Aware" Deadlock Detection: If the Master detects that a chain of parent-child workflows is stalled due to resource exhaustion, could the system automatically trigger a warning or provide a mechanism to temporarily bypass the concurrency limit for child workflows?
   Priority-based Preemption: Allow the system to prioritize the execution of child workflows over new parent workflows when the worker pool is saturated.
   
   ### How to reproduce
   
   This issue is becoming a major blocker for our cluster stability. We are looking for a sustainable architectural approach to handle high-concurrency nested workflows without requiring end-users to manually manage timeouts for every task.
   
   
   
   ### Anything else
   
   _No response_
   
   ### Version
   
   3.3.2
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Potential Deadlock Risk in SubProcess Workflow: Resource Exhaustion when Worker Threads are Saturated [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #18176:
URL: https://github.com/apache/dolphinscheduler/issues/18176#issuecomment-4272284315

   @crazychengmm Your second picture is correct. This feature was introduced in version 3.3.x.
   
   In earlier versions (3.0.x, and possibly 2.x), subprocess tasks were also executed on the master. However, separate thread pools for logic tasks were not configurable at that time. Although this did not result in deadlocks, it could still lead to other forms of blocking or execution stalls.
   
   Recommand to upgrade to latest 3.4.x version


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Potential Deadlock Risk in SubProcess Workflow: Resource Exhaustion when Worker Threads are Saturated [dolphinscheduler]

Posted by "crazychengmm (via GitHub)" <gi...@apache.org>.
crazychengmm commented on issue #18176:
URL: https://github.com/apache/dolphinscheduler/issues/18176#issuecomment-4266571067

   > `SubWorkflow` task use a separate thread pool at master, you can set the size, the default size is enouth, since it use async mode.
   > 
   > ```
   > logic-task-config:
   >    The number of threads used to execute logic task.
   >    task-executor-thread-count: 10
   > ```
   > 
   > Why this will related to worker threads? you meas the task define at sub workflow? this will not cause deadlock, one task finish will release the resource.
   
   Thank you for the quick response! I think there may be a misunderstanding about where the bottleneck occurs. Allow me to clarify.
   
   We are not referring to the Master-side logic-task-config thread pool.
   
   We fully understand that the SubWorkflow task itself is managed asynchronously on the Master side via a dedicated logic task thread pool. That part works fine.
   
   The deadlock we observed occurs at the Worker thread level, specifically caused by tasks defined inside the child workflow.
   
   Here is the exact scenario:
   
   <img width="900" height="890" alt="Image" src="https://github.com/user-attachments/assets/852af5f1-61d6-4432-9c51-3c7f7457a3a3" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Potential Deadlock Risk in SubProcess Workflow: Resource Exhaustion when Worker Threads are Saturated [dolphinscheduler]

Posted by "crazychengmm (via GitHub)" <gi...@apache.org>.
crazychengmm commented on issue #18176:
URL: https://github.com/apache/dolphinscheduler/issues/18176#issuecomment-4266754961

   Sorry, I just need one simple confirmation.
   
   Based on your description of logic-task-config, we believe the Logic Task architecture works as illustrated in the diagram above.
   
   If this is correct:
   
   SubWorkflow Task executes locally on Master via the Logic Task Pool
   Child workflow is triggered asynchronously, without occupying any Worker threads
   Worker threads are only used by the actual physical tasks inside the child workflow
   Worker threads are NEVER held by SubWorkflow itself
   Then we fully agree that the deadlock risk we described is completely eliminated in this architecture. ✅
   
   Could you simply confirm whether the diagram above correctly represents the current Logic Task implementation?
   
   And if so, could you let us know which version this architecture was introduced? This will help us plan our cluster upgrade. 🙏
   This is critical for us because we operate multiple clusters running different versions
   Some of our clusters are on older versions that may not have this architecture
   We need to identify which clusters are still at risk and prioritize their upgrades accordingly
   
   <img width="1296" height="882" alt="Image" src="https://github.com/user-attachments/assets/5da5b3f9-d093-44e6-b850-4bfd7a5e2ab6" />
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Potential Deadlock Risk in SubProcess Workflow: Resource Exhaustion when Worker Threads are Saturated [dolphinscheduler]

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #18176:
URL: https://github.com/apache/dolphinscheduler/issues/18176#issuecomment-4265779253

   cc @ruanwenjun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Potential Deadlock Risk in SubProcess Workflow: Resource Exhaustion when Worker Threads are Saturated [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #18176:
URL: https://github.com/apache/dolphinscheduler/issues/18176#issuecomment-4265969409

   `SubWorkflow` task use a separate thread pool at master, you can set the size, the default size is enouth, since it use async mode. 
   ```
   logic-task-config:
      The number of threads used to execute logic task.
      task-executor-thread-count: 10
   ```
   
   Why this will related to worker threads? you meas the task define at sub workflow? this will not cause deadlock.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Question] Potential Deadlock Risk in SubProcess Workflow: Resource Exhaustion when Worker Threads are Saturated [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun closed issue #18176: [Question] Potential Deadlock Risk in SubProcess Workflow: Resource Exhaustion when Worker Threads are Saturated
URL: https://github.com/apache/dolphinscheduler/issues/18176


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org