[jira] [Created] (UNOMI-967) Clustered scheduler locks and recovery are still unsafe under node races

["Serge Huber (Jira)" <ji...@apache.org>]

Serge Huber created UNOMI-967:
---------------------------------

             Summary: Clustered scheduler locks and recovery are still unsafe under node races
                 Key: UNOMI-967
                 URL: https://issues.apache.org/jira/browse/UNOMI-967
             Project: Apache Unomi
          Issue Type: Bug
          Components: unomi-3.1.0, unomi(-core)
    Affects Versions: unomi-3.1.0
            Reporter: Serge Huber
            Assignee: Serge Huber
             Fix For: unomi-3.1.0


h2. Summary

In a multi-node Unomi cluster, scheduled tasks can still lose locks incorrectly, run on the wrong node after a crash, or stay stuck after a node stops. Operators cannot rely on exclusive tasks and recovery without risking duplicate or stranded work.

h2. What is going wrong today

* A node can clear or overwrite another node's lock during release or completion races
* Crash recovery can dispatch work from non-executor nodes, or leave tasks stranded after shutdown
* Exclusive (non-parallel) tasks default and type-level checks are not strong enough for operators who expect one active run
* Lock verification false-negatives after CAS can leave locks held or drop a peer's claim
* Unit coverage around managers, multi-node isolation, and these races is too thin to catch regressions

h2. Impact

* Duplicate or missed task execution in production clusters
* Tasks stuck in RUNNING/CRASHED until manual intervention or lock timeout
* Hard-to-reproduce CI flakes when scheduler races are involved

h2. Who is affected

* Operators running Unomi with more than one scheduler/executor node
* Contributors changing the scheduler who lack regression tests for cluster races

h2. What good would look like

* Exclusive tasks do not steal or wipe a peer's valid lock
* Terminal and crash transitions only commit when the node still owns the work
* Non-executor nodes can mark crashes but do not dispatch recovery unless the task is meant to run on all nodes
* Shutdown after prepare does not leave orphan locks
* Solid unit tests cover lock, execution, recovery, provider shutdown, and multi-node isolation races
* Scheduler manual documents states, locks, and ES/OpenSearch OCC expectations clearly



--
This message was sent by Atlassian Jira
(v8.20.10#820010)