[jira] [Commented] (HDDS-14969) Bump Ratis to 3.2.2

["Priyesh K (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075753#comment-18075753 ] 

Priyesh K commented on HDDS-14969:
----------------------------------

[~adoroszlai] TestOzoneManagerHAWithStoppedNodes failures are because repeated leader election happening after {{testListVolumes test which}} slow test execution and random tests get timedout.

Three configurations interact to cause this issue
 # Aggressive log purging ({{{}LOG_PURGE_GAP = 50{}}}, {{{}SNAPSHOT_THRESHOLD = 50{}}})
A Ratis snapshot is taken every 50 transactions, and log entries older than {{snapshotIndex 50}} are permanently deleted. Creating 100 volumes generates 100+ Raft log entries, which pushes the cluster well past the purge boundary. After the purge, entries in the early part of the log are gone.
 # {{installSnapshotEnabled = false}}
Ratis's built-in gRPC snapshot transfer is disabled in OM. When the leader's LogAppender detects that a lagging follower needs log entries that have been purged, it cannot send them via {{{}AppendEntries{}}}. Instead it triggers {{{}notifyInstallSnapshotFromLeader(){}}}, which starts an async HTTP snapshot download. While the download is in progress, the LogAppender sends no heartbeats to that follower.

 # 1-second election timeout ({{{}RATIS_RPC_TIMEOUT = 1000ms{}}})
Without heartbeats, the follower's election timer fires every ~1 second. It repeatedly attempts {{{}PRE_VOTE{}}}, gets rejected instantly (valid leader exists), and resets its timer. This loop continues for the entire duration of the snapshot download, which scales with the size of the RocksDB state that is with how many volumes were created.

I figured out this problem when i changed total volume created to 10 and then compared logs with the existing one. It can be resolved by changing {{LOG_PURGE_GAP to higher value. Updated PR with these change.}}

> Bump Ratis to 3.2.2
> -------------------
>
>                 Key: HDDS-14969
>                 URL: https://issues.apache.org/jira/browse/HDDS-14969
>             Project: Apache Ozone
>          Issue Type: Task
>          Components: build
>            Reporter: Attila Doroszlai
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes-output.txt, org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes.txt
>
>
> Bump Ratis and its dependencies to:
> {code}
>     <!-- versions included in ratis-thirdparty, update in sync -->
>     <ratis-thirdparty.grpc.version>1.77.1</ratis-thirdparty.grpc.version>
>     <ratis-thirdparty.netty.version>4.1.130.Final</ratis-thirdparty.netty.version>
>     <ratis-thirdparty.protobuf.version>3.25.8</ratis-thirdparty.protobuf.version>
>     <ratis.thirdparty.version>1.0.11</ratis.thirdparty.version>
>     <ratis.version>3.2.2</ratis.version>
> {code}
> Fix failure in TestOzoneManagerHAWithStoppedNodes:
> {code}
> Tests run: 13, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 229.9 s <<< FAILURE! -- in org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes
> org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes.testRetryCacheWithDownedOM -- Time elapsed: 18.21 s <<< ERROR!
> org.apache.ratis.protocol.exceptions.TransferLeadershipException: omNode-3@group-523986131536: Failed to transfer leadership to omNode-1 (the current leader is omNode-3): TIMED_OUT(10s)
> 	at org.apache.ratis.server.impl.TransferLeadership$PendingRequest.complete(TransferLeadership.java:152)
> 	at org.apache.ratis.server.impl.TransferLeadership.lambda$complete$14(TransferLeadership.java:345)
> 	at java.base/java.util.Optional.ifPresent(Optional.java:178)
> 	at org.apache.ratis.server.impl.TransferLeadership.complete(TransferLeadership.java:345)
> 	at org.apache.ratis.server.impl.TransferLeadership.lambda$start$9(TransferLeadership.java:310)
> 	at org.apache.ratis.util.TimeoutTimer.lambda$onTimeout$2(TimeoutTimer.java:101)
> 	at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:38)
> 	at org.apache.ratis.util.LogUtils$1.run(LogUtils.java:78)
> 	at org.apache.ratis.util.TimeoutTimer$Task.run(TimeoutTimer.java:55)
> 	at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
> 	at java.base/java.util.TimerThread.run(Timer.java:516)
> org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes.testIncrementalWaitTimeWithSameNodeFailover -- Time elapsed: 8.006 s <<< ERROR!
> java.io.IOException: Could not determine or connect to OM Leader.
> 	at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:119)
> 	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:341)
> 	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createVolume(OzoneManagerProtocolClientSideTranslatorPB.java:363)
> 	at org.apache.hadoop.ozone.client.rpc.RpcClient.createVolume(RpcClient.java:466)
> 	at org.apache.hadoop.ozone.client.ObjectStore.createVolume(ObjectStore.java:122)
> 	at org.apache.hadoop.ozone.om.TestOzoneManagerHA.createKeyTest(TestOzoneManagerHA.java:387)
> 	at org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes.testIncrementalWaitTimeWithSameNodeFailover(TestOzoneManagerHAWithStoppedNodes.java:487)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org