[jira] [Updated] (RATIS-2129) Low replication performance because of lock contention on RaftLog

["Duong (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/RATIS-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Duong updated RATIS-2129:
-------------------------
    Summary: Low replication performance because of lock contention on RaftLog  (was: Low replication performance because GrpcLogAppender is often blocked by RaftLog's readLock)

> Low replication performance because of lock contention on RaftLog
> -----------------------------------------------------------------
>
>                 Key: RATIS-2129
>                 URL: https://issues.apache.org/jira/browse/RATIS-2129
>             Project: Ratis
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.1.0
>            Reporter: Duong
>            Assignee: Tsz-wo Sze
>            Priority: Blocker
>              Labels: Performance, performance
>         Attachments: Screenshot 2024-07-22 at 4.40.07 PM-1.png, Screenshot 2024-07-22 at 4.40.07 PM.png, dn_echo_leader_profile.html, image-2024-07-22-15-25-46-155.png, ratis_ratfLog_lock_contention.png
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Today, the GrpcLogAppender thread makes a lot of calls that need RaftLog's readLock. In an active environment, RaftLog is always busy appending transactions from clients, thus writeLock is frequently busy. This makes the replication performance slow. 
> See the [^dn_echo_leader_profile.html], or in the picture below, the purple is the time taken to acquire readLock from RaftLog.
>  # !image-2024-07-22-15-25-46-155.png|width=854,height=425!
> So far, I'm not sure if this is a regression from a recent change in 3.1.0/3.0.0, or if it's been always the case. 
> A few early considerations:
>  # The rate of calling RaftLog per GrpcLogAppender seems to be too high. Instead of calling RaftLog multiple, maybe the log appended can call once to obtain all the required information?
>  # Can RaftLog expose those data without requiring a read lock? 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)