You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/10/09 00:41:00 UTC
[jira] [Updated] (HDDS-16801) Intermittent failure in TestPerVolumePushReplication#testHealthyVolumeReplicationAfterVolumeFailure
[ https://issues.apache.org/jira/browse/HDDS-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-16801:
----------------------------------
Labels: pull-request-available (was: )
> Intermittent failure in TestPerVolumePushReplication#testHealthyVolumeReplicationAfterVolumeFailure
> ---------------------------------------------------------------------------------------------------
>
> Key: HDDS-16801
> URL: https://issues.apache.org/jira/browse/HDDS-16801
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 101.2 s <<< FAILURE! -- in org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication
> [ERROR] org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testHealthyVolumeReplicationAfterVolumeFailure -- Time elapsed: 35.31 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition. Thread diagnostics:
> ...
> at org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.queuePushAndWaitForContainer(TestPerVolumePushReplication.java:386)
> at org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testHealthyVolumeReplicationAfterVolumeFailure(TestPerVolumePushReplication.java:219)
> {code}
> The source datanode dropped the injected push command:
> {code}
> WARN statemachine.StateContext (StateContext.java:getNextCommand(767)) - Detect and drop a SCMCommand replicateContainerCommand: cmdID: 1791499115402, encodedToken: "", term: 0, ..., containerId=1000003, ... from stale leader SCM, stale term 0, latest term 2.
> {code}
> h3. Root cause
> testPushAndScmReplicationWithPerVolumeEnabled restarts a datanode at the end, and testHealthyVolumeReplicationAfterVolumeFailure can pick that same datanode as the push source. queueReplicationCommand stamps the command with the datanode's own term via {{context.getTermOfLeaderSCM().ifPresent(cmd::setTerm)}}. On a just restarted datanode that term is not initialized yet, so the command keeps term 0. When StateContext later initializes the term from the max term in its command queue (2, from real SCM commands such as close pipeline), the term 0 push is dropped as coming from a stale leader SCM, and the wait for the container on the target times out.
> h3. Fix
> Stamp the command with the SCM leader term ({{getScmContext().getTermOfLeader()}}), as TestCloseContainerByPipeline and TestDeleteContainerHandler do, so it never depends on whether the datanode has learned the term yet.
> - https://github.com/apache/ozone/actions/runs/37850535429/job/113566423731
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org