[jira] [Resolved] (RATIS-2761) Fix incorrect locking in LeaderElection

["Tsz-wo Sze (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/RATIS-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz-wo Sze resolved RATIS-2761.
-------------------------------
    Fix Version/s: 3.4.0
       Resolution: Fixed

The pull request is now merged.  Thanks, [~slfan1989]!

> Fix incorrect locking in LeaderElection
> ---------------------------------------
>
>                 Key: RATIS-2761
>                 URL: https://issues.apache.org/jira/browse/RATIS-2761
>             Project: Ratis
>          Issue Type: Bug
>          Components: election
>            Reporter: Shilun Fan
>            Assignee: Shilun Fan
>            Priority: Major
>             Fix For: 3.4.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> TestGroupManagementWithGrpc.testMultiGroup9Nodes failed with:
> {code:java}
> GroupMismatchException: s0: Group group-82F96BE2938F not found. {code}
> CI failure:
> [https://github.com/apache/ratis/actions/runs/38951714056/job/117144457810?pr=1645]
>  
> Investigation of the unit-grpc artifact shows that GroupMismatchException is a secondary failure. The original failure is concurrent persistence of Raft metadata during leader election.
>  
> *Root Case*
>  
> LeaderElection synchronizes on its ServerInterface adapter, while ElectionRequestHandler synchronizes on the underlying RaftServerImpl. These are different objects, so the two paths are not mutually exclusive.
>  
> Consequently, the election thread and the requestVote RPC thread can concurrently write to the same raft-meta.tmp file. Once one thread renames the temporary file to raft-meta, the other thread fails with NoSuchFileException.
>  
> RaftStorageMetadataFileImpl uses AtomicReference.updateAndGet, but this does not serialize the file writes performed inside the update function.
>  
> *Evidence from the CI logs (UTC; paths abbreviated)*
>  
> {code:java}
> Evidence from the CI logs (UTC; paths abbreviated)
> 19:20:25.173 [s0@group-82F96BE2938F-LeaderElection15]Opened .../s0/.../current/raft-meta.tmp
> 19:20:25.261 [grpc-default-executor-1]Opened the same raft-meta.tmp file
> 19:20:25.261 [s0@group-82F96BE2938F-LeaderElection15]Successfully moved raft-
> meta.tmp to raft-meta
> 19:20:25.264 [grpc-default-executor-1]NoSuchFileException: .../raft-meta.tmp -> .../raft-meta
> 19:20:25.305 [s0@group-82F96BE2938F-close]shutdown this division due to a failure.
> At 19:20:32, the first group removal attempt removes the group fromthe server's map, but fails because the division's close futurecontains the earlier persistence exception. The client retriesthe same request (client-000000000026, cid=98), which then failswith GroupMismatchException. {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)