You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/08/11 09:18:00 UTC

[jira] [Updated] (HDDS-16132) Intermittent timeout in TestSCMFollowerCatchupWithContainerReport (follower stuck in safe mode on idle cluster)

     [ https://issues.apache.org/jira/browse/HDDS-16132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16132:
------------------------------
    Attachment: HDDS-16132.001.patch

> Intermittent timeout in TestSCMFollowerCatchupWithContainerReport (follower stuck in safe mode on idle cluster)
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16132
>                 URL: https://issues.apache.org/jira/browse/HDDS-16132
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: SCM HA, test
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: HDDS-16132.001.patch
>
>
> {code}
> [ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 311.2 s <<< FAILURE! -- in org.apache.hadoop.hdds.scm.TestSCMFollowerCatchupWithContainerReport
> [ERROR] org.apache.hadoop.hdds.scm.TestSCMFollowerCatchupWithContainerReport.testFollowerCatchupOnIdleCluster -- Time elapsed: 125.6 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition.
> ...
> [ERROR] org.apache.hadoop.hdds.scm.TestSCMFollowerCatchupWithContainerReport.testFollowerCatchupAfterContainerCreate -- Time elapsed: 121.0 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> {code}
> Both failing methods time out at {{waitFor(() -> !newFollower.isInSafeMode(), ...)}} after restarting the follower SCM. In the logs the restarted follower stays in safe mode for the full 120s with {{registered datanodes (=0) >= required datanodes (=3)}}: no datanode ever registers because the follower's {{SCMDatanodeProtocolServer}} is never started.
> Possible cause (needs confirmation from the HDDS-14989 author): HDDS-14989 defers starting the follower's datanode RPC server to {{SCMStateMachine.tryStartDNServerAndRefreshSafeMode()}}, which is invoked only from Ratis event callbacks ({{applyTransaction}}, {{notifyLeaderChanged}}, {{notifyTermIndexUpdated}}) and is gated on {{isFollowerCaughtUp()}}. On a restarted follower over an idle cluster, {{notifyLeaderChanged}} can fire once while {{getLeaderCommitIndex()}} still returns -1, so the DN server is not started; with no further transactions there is no later callback to retry, and safe mode never clears. This looks like a product-side gap in the idle-follower path rather than pure CI flakiness (raising the 120s wait does not help).
> Seen in:
>  - https://github.com/adoroszlai/ozone-build-results/blob/master/2026/08/08/52075/integration-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMFollowerCatchupWithContainerReport.txt



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org