[jira] [Created] (RATIS-2314) Deadlock when appending large batches of entries

["Tian Jiang (Jira)" <ji...@apache.org>]

Tian Jiang created RATIS-2314:
---------------------------------

             Summary: Deadlock when appending large batches of entries
                 Key: RATIS-2314
                 URL: https://issues.apache.org/jira/browse/RATIS-2314
             Project: Ratis
          Issue Type: Bug
          Components: server
            Reporter: Tian Jiang
         Attachments: image-2025-06-30-13-25-54-574.png, image-2025-06-30-13-33-13-766.png, image-2025-06-30-13-35-09-832.png, image-2025-06-30-13-37-28-743.png

Greeting, I found a deadlock when I am using Ratis, which is shown in the following stack:

!image-2025-06-30-13-25-54-574.png!

The problem is that Ratis uses CompletableFuture to serialize the process of appending logs, and below is the timeline:
 # Thread "SegmentedRaftLogWorker" is working on some logs;
 # Thread "SomeClient" tries to append 5000 entries to the log;
 # Because "SegmentedRaftLogWorker" is working, "SomeClient" adds the entries to the CompletableFuture; !image-2025-06-30-13-33-13-766.png!
 # When "SegmentedRaftLogWorker" is done with its last task, it checks the CompletableFuture for the next task; !image-2025-06-30-13-35-09-832.png!
 # Of course, there is one from "SomeClient", ordering it to append another 5000 entries; therefore "SegmentedRaftLogWorker" continues to append the entries, and it comes to:  !image-2025-06-30-13-37-28-743.png!
 # However, the queue of SegmentedRaftLogWorker has a max capacity of 4096, and there are 5000 entries to be appended, so Thread "SegmentedRaftLogWorker" waits for the queue to be not full;
 # But who is supposed to consume from the queue? Thread "SegmentedRaftLogWorker"  itself! As a result, the wait shall never end.

I notice that Ratis is using many thread-steal tricks to improve thread locality, which is appreciated. However, it seems such a technique, in this scenario, creates deadlocks.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)