[jira] [Created] (HDDS-16398) TestCommitWatcher#testReleaseBuffersOnException intermittently fails with AlreadyClosedException (NotLeaderException on freshly-allocated pipeline)

["Andrey Yarovoy (Jira)" <ji...@apache.org>]

Andrey Yarovoy created HDDS-16398:
-------------------------------------

             Summary: TestCommitWatcher#testReleaseBuffersOnException intermittently fails with AlreadyClosedException (NotLeaderException on freshly-allocated pipeline)
                 Key: HDDS-16398
                 URL: https://issues.apache.org/jira/browse/HDDS-16398
             Project: Apache Ozone
          Issue Type: Bug
          Components: SCM Client
            Reporter: Andrey Yarovoy


{{org.apache.hadoop.hdds.scm.storage.TestCommitWatcher#testReleaseBuffersOnException}} fails intermittently in CI. The failure occurs during test setup — while waiting on the _first_ {{putBlock}} response ({{{}future1.get(){}}} at {{{}TestCommitWatcher.java:262{}}}), before the test reaches the datanode shutdowns that exercise the behavior actually under test.

*Stack trace*

{code} java.util.concurrent.ExecutionException: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-B2BDCC4ECA19->RAFT is closed. at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2094) at org.apache.hadoop.hdds.scm.storage.TestCommitWatcher.testReleaseBuffersOnException(TestCommitWatcher.java:262) Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-... is closed. at org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:409) ... Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest: client-...->d62e25aa-...@group-..., cid=29, seq=1*, Watch(0), null for 3 attempts with RequestTypeDependentRetryPolicy\{ ... } at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:352) Caused by: org.apache.ratis.protocol.exceptions.NotLeaderException: Server d62e25aa-... is not the leader, suggested leader is: 59f5747a-...|127.0.0.1:15071 at org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:402) \{code}

*Root cause*

The test allocates a fresh RATIS THREE pipeline and immediately issues async write/putBlock commands:

{code} ContainerWithPipeline container = storageContainerLocationClient .allocateContainer(RATIS, THREE, OZONE); Pipeline pipeline = container.getPipeline(); ... ratisClient.sendCommandAsync(writeChunkRequest); XceiverClientReply reply = ratisClient.sendCommandAsync(putBlockRequest); \{code}

{{cluster.waitForClusterToBeReady()}} (called in {{{}@BeforeEach{}}}) waits for datanode registration and SCM readiness, but does *not* guarantee that the newly-allocated Ratis group has completed leader election. When the write lands on a node that is not yet the leader, Ratis returns {{{}NotLeaderException{}}}. Under the aggressive per-test retry policy (short watch timeout, limited attempts), the request exhausts its retries and the client is closed, surfacing as {{AlreadyClosedException}} from {{{}future1.get(){}}}.

This is a timing/leader-election race in test setup, not a product defect in the commit-watch path.

*Reproducibility*

Intermittent. Timing-dependent; does not reproduce reliably. This test has a long history of the same class of flakiness — see related issues below (HDDS-9766 reports the identical {{AlreadyClosedException}} signature).

*Suggested fix*

Wait for the allocated pipeline to have an elected leader before issuing writes — e.g., block on pipeline readiness / leader election ({{{}XceiverClientRatis.watchForCommit{}}} on the leader, or a {{{}waitForPipelineReady{}}}-style helper) after {{allocateContainer}} and before the first {{{}sendCommandAsync{}}}, so setup no longer races election. Apply the same guard to {{{}testReleaseBuffers{}}}, which shares the pattern.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org