(yunikorn-site) branch master updated: [YUNIKORN-3348] Fix and extend the concurrency documentation (#571)

[ma...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

manirajv06 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git


The following commit(s) were added to refs/heads/master by this push:
     new b00884e6af [YUNIKORN-3348] Fix and extend the concurrency documentation (#571)
b00884e6af is described below

commit b00884e6afeb7d4dd33725695586360fcaa1e49b
Author: Tigerquoll <ti...@outlook.com>
AuthorDate: Thu Aug 13 16:30:27 2026 +0530

    [YUNIKORN-3348] Fix and extend the concurrency documentation (#571)
    
    Extended the concurrency documentaion, mainly focussing on deadlock detection and locking order.
    
    Fixed typo's as well
    
    Signed-off-by: Tigerquoll <ti...@outlook.com>
    
    Closes: #571
    
    Signed-off-by: Manikandan R <ma...@gmail.com>
---
 docs/design/cache_removal.md       |  8 +++++--
 docs/developer_guide/build.md      |  6 +++++
 docs/user_guide/troubleshooting.md | 45 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+), 2 deletions(-)

diff --git a/docs/design/cache_removal.md b/docs/design/cache_removal.md
index 91bd14e8e0..6a89a7db4f 100644
--- a/docs/design/cache_removal.md
+++ b/docs/design/cache_removal.md
@@ -423,7 +423,7 @@ A state change is thus immediate and this should prevent an issue like [YUNIKORN
 ### Direction of lock 
 It is possible to acquire another lock while holding a lock, but we need to make sure that we do not allow: 
 - Holding A.lock and acquire B's lock. 
-- Holding B.lock and acquire B's lock. 
+- Holding B.lock and acquire A's lock. 
 
 The current code in the scheduler takes a lock as late as possible and only for the time period needed.
 Some actions are not locked on the scheduler side just on the cache side as each object has its own lock.
@@ -453,4 +453,8 @@ The partition should be locked while retrieving for instance the node that needs
 
 This approach fits in with the current locking approach and will keep the locking changes to a minimum.
 Testing, specifically end-to-end testing, should catch these deadlocks. 
-There are no known tools that could be used to detect or describe lock order.
+
+Since this design was written, runtime detection has been added.
+The `pkg/locking` package wraps [go-deadlock](https://github.com/sasha-s/go-deadlock) and provides both lock order detection and lock wait timeout detection for all locks that use it.
+It is disabled by default and is turned on using the environment variables described in [deadlock detection](../user_guide/troubleshooting.md#deadlock-detection).
+The unit test run, `make test`, enables it in both the core and the k8shim.
diff --git a/docs/developer_guide/build.md b/docs/developer_guide/build.md
index 88cca29676..d916b0c283 100644
--- a/docs/developer_guide/build.md
+++ b/docs/developer_guide/build.md
@@ -163,6 +163,12 @@ To make sure that the local changes will not break other parts of the
 build you should run:
 - A full build `make` (build target depends on the repository)
 - A full unit test run `make test`
+  This runs the tests with the race detector (`-race`) and with deadlock detection
+  enabled: `DEADLOCK_DETECTION_ENABLED=true`, `DEADLOCK_TIMEOUT_SECONDS=10` and
+  `DEADLOCK_EXIT=true`. A run that ends with a `POTENTIAL DEADLOCK` report and a
+  non-zero exit code is a detected lock problem, not a flaky test. See
+  [deadlock detection](../user_guide/troubleshooting.md#deadlock-detection) for
+  the settings and how to read the report.
 - For diagnosing flaky tests, which are challenging due to their infrequent failures, use a looping command to repeatedly run the test. For instance, to diagnose `TestNoFillWithoutEventPluginRegistered` in `yunikorn-core/pkg/events/event_publisher_test.go`, you can use the following command:
 
   ```sh
diff --git a/docs/user_guide/troubleshooting.md b/docs/user_guide/troubleshooting.md
index 81b6fc1b83..5a410d7798 100644
--- a/docs/user_guide/troubleshooting.md
+++ b/docs/user_guide/troubleshooting.md
@@ -167,6 +167,51 @@ With the below scheduler REST API returns information about full state dump used
 
 For more details around the content of the state dump, please refer to the documentation on [retrieve-full-state-dump](api/system.md#retrieve-state-dump)
 
+## Deadlock detection
+
+If the scheduler stops making progress but the process is still running and responding to the
+REST API, a lock problem is one possible cause. YuniKorn can detect this at runtime.
+
+All locks in the scheduler core and the Kubernetes shim are created through the `pkg/locking`
+package, which wraps [go-deadlock](https://github.com/sasha-s/go-deadlock). The shim delegates
+its configuration to the core, so a single set of settings covers both. Detection is **disabled
+by default** and is turned on with environment variables on the scheduler container:
+
+| Variable | Default | Effect |
+|---|---|---|
+| `DEADLOCK_DETECTION_ENABLED` | `false` | Master switch. go-deadlock is always compiled in, this only enables it. |
+| `DEADLOCK_TIMEOUT_SECONDS` | `60` | How long a goroutine may wait for a lock before a potential deadlock is reported. |
+| `DEADLOCK_EXIT` | `false` | Terminate the process with exit code 1 when a potential deadlock is detected. |
+| `DEADLOCK_DISABLE_LOCK_ORDER` | `false` | Disable lock order (ABBA) detection and keep only the wait timeout check. |
+
+Two independent checks are performed. The **wait timeout** check reports a goroutine that has
+been waiting for a single lock for longer than `DEADLOCK_TIMEOUT_SECONDS`. The **lock order**
+check reports two locks that have been acquired in opposite orders by different goroutines,
+which can deadlock even if it has not happened yet.
+
+Reports are written to the log under the `core.diagnostics` subsystem at `ERROR` level, and
+begin with `POTENTIAL DEADLOCK`. Each report names the goroutine holding the lock and the
+goroutine waiting for it, with a stack trace for both.
+
+Whether detection is currently active can be read back from the scheduler:
+
+```shell script
+curl -X 'GET' http://localhost:9080/ws/v1/config -H 'accept: application/json'
+```
+
+The response includes `DeadlockDetectionEnabled` and `DeadlockTimeoutSeconds`.
+
+:::note
+`DEADLOCK_EXIT=true` will terminate a running scheduler as soon as a potential deadlock is
+seen. That is useful when reproducing a problem, because it captures the state at the point of
+detection instead of leaving a hung process, but consider the effect on a production cluster
+before enabling it there.
+:::
+
+Detection is enabled for unit tests in both repositories. The `make test` target sets
+`DEADLOCK_DETECTION_ENABLED=true`, `DEADLOCK_TIMEOUT_SECONDS=10` and `DEADLOCK_EXIT=true`, so a
+test run that ends with a `POTENTIAL DEADLOCK` report has found a real lock problem.
+
 ## Restart the scheduler
 
 :::note


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org