[jira] [Created] (HDDS-17190) Intermittent failure in TestDecommissionAndMaintenance#testSingleNodeWithOpenPipelineCanGotoMaintenance

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17190:
---------------------------------

             Summary: Intermittent failure in TestDecommissionAndMaintenance#testSingleNodeWithOpenPipelineCanGotoMaintenance
                 Key: HDDS-17190
                 URL: https://issues.apache.org/jira/browse/HDDS-17190
             Project: Apache Ozone
          Issue Type: Sub-task
          Components: SCM, test
            Reporter: Siyao Meng
            Assignee: Siyao Meng


{code}
[ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 320.6 s <<< FAILURE! -- in org.apache.hadoop.hdds.scm.node.TestDecommissionAndMaintenance
[ERROR] org.apache.hadoop.hdds.scm.node.TestDecommissionAndMaintenance.testSingleNodeWithOpenPipelineCanGotoMaintenance -- Time elapsed: 59.78 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition.
	at org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
	at org.apache.hadoop.hdds.scm.node.NodeTestUtil.waitForDnToReachPersistedOpState(NodeTestUtil.java:98)
	at org.apache.hadoop.hdds.scm.node.TestDecommissionAndMaintenance.testSingleNodeWithOpenPipelineCanGotoMaintenance(TestDecommissionAndMaintenance.java:487)
{code}

h3. Root cause

{{waitForDnToReachPersistedOpState}} polls {{getPersistedOpState()}} on the {{DatanodeDetails}} object it is given. The test passes {{dn}}, captured from {{nm.getNode(dnID)}} before the datanode restarts. When a restarted datanode re-registers with a different port set, {{SCMNodeManager#register}} replaces the stored {{DatanodeInfo}} (the {{portsChanged}} path added by HDDS-12991). In the failing run the HTTP port was missing from the initial registration and present on re-registration:

{noformat}
SCMNodeManager - Updating ports for registered datanode ca3c16ee-...: [STANDALONE=15350, ..., CLIENT_RPC=15349] -> [STANDALONE=15350, ..., HTTP=15348, CLIENT_RPC=15349]
{noformat}

After that SCM updates the persisted state on the new object (the log shows {{IN_MAINTENANCE -> IN_SERVICE}}), while the test keeps polling the stale {{dn}} until it times out. The neighbouring assertions already use {{newDn}}. {{testNodeWithOpenPipelineCanBeDecommissionedAndRecommissioned}} has the same stale reference ({{toDecommission}}) after a restart.

h3. Fix

Look up the node from {{NodeManager}} after the restart ({{newDn}} / {{nm.getNode(dnID)}}) before waiting for the persisted state.

- https://github.com/apache/ozone/actions/runs/39445520577/job/118801747190



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org