[jira] [Resolved] (HDDS-15605) Intermittent failure in TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException

["Attila Doroszlai (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Attila Doroszlai resolved HDDS-15605.
-------------------------------------
    Fix Version/s: 2.3.0
       Resolution: Fixed

> Intermittent failure in TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-15605
>                 URL: https://issues.apache.org/jira/browse/HDDS-15605
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: test
>            Reporter: Chi-Hsuan Huang
>            Assignee: Chi-Hsuan Huang
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 2.3.0
>
>
> h2. Symptom
> {{TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException}} fails intermittently \(observed \~1/40 on CI, not reproducible locally\) with an assertion failure, not a timeout:
> {code}
> TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException:398
> java.lang.AssertionError:
> Expecting empty but was: \[5efc24c5\-0b87\-4bf7\-80b0\-751fafcf3248\(null/null\)\]
> {code}
> Line 398 asserts {{keyOutputStream.getExcludeList\(\).getDatanodes\(\)}} is empty: only the closed container should be excluded, no datanode.
> h2. Root cause analysis
> In {{KeyOutputStream.handleException}} \(around lines 386\-400\), excluding a datanode and excluding the container are two independent decisions that can both fire:
> {code}
> Collection failedServers = streamEntry.getFailedServers\(\);
> if \(\!failedServers.isEmpty\(\)\) {
>   excludeList.addDatanodes\(failedServers\);          // populates getDatanodes\(\)
> }
> if \(containerExclusionException\) {
>   excludeList.addConatinerId\(...\);                  // container \(expected by the test\)
> } else {
>   excludeList.addPipeline\(pipelineId\);
> }
> {code}
> The test assumes the second write fails only with {{ClosedContainerException}}, so {{failedServers}} is empty. But the excluded datanode is printed as {{\(null/null\)}}, which is what {{XceiverClientRatis.addDatanodetoReply}} produces \(it builds {{DatanodeDetails}} from the Ratis peer UUID only, with no IP or hostname\). This points to a Ratis peer write/watch failure rather than a clean {{ClosedContainerException}}.
> Sequence:h1. {{TestHelper.waitForContainerClose}} closes the container, which also tears down the Ratis pipeline on the datanodes.
> h1. The subsequent write \(or its watch\-for\-commit\) to a Ratis peer can fail or time out while the pipeline is closing, so that peer is recorded in {{failedServers}}.
> h1. {{handleException}} then adds that datanode to the exclude list in addition to the container, so {{getDatanodes\(\)}} is non\-empty and the assertion on line 398 fails.
> This is a timing race between container close and the in\-flight Ratis write/watch, which is why it only shows up under load on CI.
> h2. Notes
> * Distinct from HDDS\-7878 \(resolved\), which tracked an intermittent _timeout_ in the same method. This is an _assertion_ failure with a different cause.
> * Observed in CI: [https://github.com/chihsuan/ozone/actions/runs/27691671664|https://github.com/chihsuan/ozone/actions/runs/27691671664] \(job: integration \(client\)\).
> * The assertion on line 398 \(and likely 399 for pipelines\) may be too strict given that container close can legitimately surface a transient datanode/pipeline failure.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org