You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by "Tsz-wo Sze (Jira)" <ji...@apache.org> on 2025/10/13 18:44:00 UTC

[jira] [Commented] (RATIS-2345) Leader stepDown could cause a deadlock

    [ https://issues.apache.org/jira/browse/RATIS-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18029589#comment-18029589 ] 

Tsz-wo Sze commented on RATIS-2345:
-----------------------------------

The stack traces [^result-3-18450620115-split-2.zip] showing the deadlock:
{code}
2025-10-12 23:01:12,060 [Timer-1] INFO  netty.TestRaftAsyncWithNetty (RaftBasicTests.java:lambda$run$1(371)) - "s1@group-680D12652185-FollowerState" Id=170 BLOCKED on org.apache.ratis.server.impl.RaftServerImpl@23e662ef owned by "s1@group-680D12652185-LeaderStateImpl" Id=144
	at org.apache.ratis.server.impl.FollowerState.runImpl(FollowerState.java:160)
	-  blocked on org.apache.ratis.server.impl.RaftServerImpl@23e662ef
	at org.apache.ratis.server.impl.FollowerState.run(FollowerState.java:130)


2025-10-12 23:01:12,061 [Timer-1] INFO  netty.TestRaftAsyncWithNetty (RaftBasicTests.java:lambda$run$1(371)) - "s1@group-680D12652185->s0-LogAppenderDefault-LogAppenderDaemon" Id=145 BLOCKED on org.apache.ratis.server.impl.RaftServerImpl@23e662ef owned by "s1@group-680D12652185-LeaderStateImpl" Id=144
	at org.apache.ratis.server.leader.LogAppender.onFollowerTerm(LogAppender.java:213)
	-  blocked on org.apache.ratis.server.impl.RaftServerImpl@23e662ef
	at org.apache.ratis.server.leader.LogAppenderDefault.handleReply(LogAppenderDefault.java:197)
	at org.apache.ratis.server.leader.LogAppenderDefault.run(LogAppenderDefault.java:165)
	at org.apache.ratis.server.leader.LogAppenderDaemon.run(LogAppenderDaemon.java:80)
	at org.apache.ratis.server.leader.LogAppenderDaemon$$Lambda$1040/1375983778.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:750)


// holding 23e662ef
2025-10-12 23:01:12,061 [Timer-1] INFO  netty.TestRaftAsyncWithNetty (RaftBasicTests.java:lambda$run$1(371)) - "s1@group-680D12652185-LeaderStateImpl" Id=144 WAITING on java.util.concurrent.CompletableFuture$Signaller@49255ac3
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.CompletableFuture$Signaller@49255ac3
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
	at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
	at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
	at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
	at org.apache.ratis.server.impl.LeaderStateImpl.stepDown(LeaderStateImpl.java:706)
	at org.apache.ratis.server.impl.LeaderStateImpl.lambda$submitStepDownEvent$12(LeaderStateImpl.java:700)
{code}

> Leader stepDown could cause a deadlock
> --------------------------------------
>
>                 Key: RATIS-2345
>                 URL: https://issues.apache.org/jira/browse/RATIS-2345
>             Project: Ratis
>          Issue Type: Bug
>          Components: Leader
>            Reporter: Tsz-wo Sze
>            Assignee: Tsz-wo Sze
>            Priority: Major
>         Attachments: result-3-18450620115-split-2.zip
>
>
> Leader stepDown could cause a deadlock:
> - LeaderStateImpl.stepDown(..), who is holding the RaftServerImpl lock, joins the future returned from server.changeToFollowerAndPersistMetadata(..) 
> -- the future completes after RoleInfo.shutdownLeaderState(..),
> -- which calls LeaderStateImpl.stop()
> -- which waits for all LogAppender to stop.
> - However, LogAppender may waits for the RaftServerImpl lock in LogAppender.onFollowerTerm(..)
> -----
> (Original description)
> In the 10x10 run below, it has 3/100 failures. All failed with timeout.
> - https://github.com/apache/ratis/actions/runs/18450620115/job/52563900327



--
This message was sent by Atlassian Jira
(v8.20.10#820010)