[jira] [Created] (HDDS-16235) Intermittent failure in TestOzoneClientRetriesOnExceptions#testMaxRetriesByOzoneClient

["Meng-Shuan Tsai (Jira)" <ji...@apache.org>]

Meng-Shuan Tsai created HDDS-16235:
--------------------------------------

             Summary: Intermittent failure in TestOzoneClientRetriesOnExceptions#testMaxRetriesByOzoneClient
                 Key: HDDS-16235
                 URL: https://issues.apache.org/jira/browse/HDDS-16235
             Project: Apache Ozone
          Issue Type: Sub-task
            Reporter: Meng-Shuan Tsai
            Assignee: Meng-Shuan Tsai


testMaxRetriesByOzoneClient fails intermittently (~3% of runs) at TestOzoneClientRetriesOnExceptions.java:206 with:
{code:java}
java.io.IOException: Failed to execute command cmdType: CreateContainer containerID: 1 datanodeUuid: "..." createContainer { state: OPEN }
	at org.apache.hadoop.hdds.scm.XceiverClientSpi.getIOExceptionForSendCommand(XceiverClientSpi.java:178)
	at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:131)
	at org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnExceptions.testMaxRetriesByOzoneClient(TestOzoneClientRetriesOnExceptions.java:206)
Caused by: java.util.concurrent.ExecutionException: org.apache.ratis.protocol.exceptions.StateMachineException: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Container creation failed because key value container already exists
	at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:125)
Caused by: org.apache.ratis.protocol.exceptions.StateMachineException: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Container creation failed because key value container already exists
	at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$applyTransaction$2(ContainerStateMachine.java:1193)
{code}
Steps to reproduce: run TestOzoneClientRetriesOnExceptions out of 100 runs ( [https://github.com/shuan1026/ozone/actions/runs/32260061359]), 3 failed with this identical stack trace, all on containerID 1. Datanode-side log for one such failure:
{code:java}
14:18:42,671 leader elected for the container's pipeline (term 1)
14:18:42,788 CreateContainer for containerID 1 fails: Result: CONTAINER_ALREADY_EXISTS
14:18:42,800 ContainerStateMachine ApplyTransaction failed at logIndex 5
14:18:42,807 pipeline close action triggered as a result
{code}
Line 206 is the test's own manual xceiverClient.sendCommand(getCreateContainerRequest(...)), sent once per pre-allocated container to ensure it is OPEN before writing. By design this should be the only path that creates the container on the datanode, but the log shows it was already created by something else ~117ms after the pipeline's leader election, so the test's own call is rejected as a duplicate, non-idempotent create.

No @Flaky annotation exists on this test and no prior JIRA report was found for it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org