You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2026/10/14 22:12:37 UTC
[PR] HDDS-17027. Intermittent failure in TestReplicationSupervisor#testReconciliationTaskMetrics [ozone]
smengcl opened a new pull request, #11542:
URL: https://github.com/apache/ozone/pull/11542
Generated-by: Claude Code (Opus 5.5)
## What changes were proposed in this pull request?
`TestReplicationSupervisor#testReconciliationTaskMetrics` fails intermittently on `assertTrue(getReplicationRequestTotalTime(...) > 0)`.
`ReplicationSupervisor` records task latency with `Time.monotonicNow()`, which has millisecond resolution. The reconciliation tasks in this test call a Mockito `ContainerController` whose `reconcileContainer` does nothing, so all tasks can complete within the same millisecond and the total latency stays 0.
This patch stubs `reconcileContainer` in the test to sleep 1 ms, so each successful task takes at least 1 ms and the total and average latency are always positive.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17027
## How was this patch tested?
- `mvn -pl :hdds-container-service test -Dtest=TestReplicationSupervisor`: 54/54 passed locally.
- The fix is deterministic by construction: a 1 ms sleep guarantees a difference of at least 1 between the two `monotonicNow()` readings around each successful task.
- 10x10 [flaky-test-check](https://github.com/smengcl/hadoop-ozone/actions/runs/38645780407) on this branch: in progress
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17027. Intermittent failure in TestReplicationSupervisor#testReconciliationTaskMetrics [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11542:
URL: https://github.com/apache/ozone/pull/11542#issuecomment-6282700956
Thanks @rich7420 @Russole for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17027. Intermittent failure in TestReplicationSupervisor#testReconciliationTaskMetrics [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #11542:
URL: https://github.com/apache/ozone/pull/11542#discussion_r4276965560
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationSupervisor.java:
##########
@@ -524,6 +524,12 @@ public void testReconciliationTaskMetrics(ContainerLayoutVersion layout) throws
ReplicationSupervisorMetrics replicationMetrics =
ReplicationSupervisorMetrics.create(replicationSupervisor);
+ // The latency metric has millisecond resolution, so make each successful reconciliation take at least 1 ms.
+ doAnswer(invocation -> {
+ Thread.sleep(1);
Review Comment:
`Thread.sleep(1)` can still be flaky on some platforms due to scheduler/timer granularity and may return early if interrupted; it also clears the interrupt status. Consider (mandatory) restoring the interrupt flag when interrupted, and (recommended) increasing the sleep duration (eg, 2–5ms) or using an uninterruptible sleep helper to make the test more deterministic across environments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17027. Intermittent failure in TestReplicationSupervisor#testReconciliationTaskMetrics [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #11542:
URL: https://github.com/apache/ozone/pull/11542#discussion_r4278073849
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationSupervisor.java:
##########
@@ -524,6 +524,12 @@ public void testReconciliationTaskMetrics(ContainerLayoutVersion layout) throws
ReplicationSupervisorMetrics replicationMetrics =
ReplicationSupervisorMetrics.create(replicationSupervisor);
+ // The latency metric has millisecond resolution, so make each successful reconciliation take at least 1 ms.
+ doAnswer(invocation -> {
+ Thread.sleep(1);
Review Comment:
Thread.sleep(1) does not return early on JDK 21: the JVM re-sleeps until the full duration has elapsed on the same clock that Time.monotonicNow() reads, so the before and after readings always differ by at least 1 ms. Coarser timer granularity only makes the sleep longer, which still satisfies > 0. Nothing interrupts this thread (the supervisor uses a direct executor, so the stub runs on the test thread), and if it were interrupted the exception would surface as a test failure rather than a silent pass. Keeping it at 1 ms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17027. Intermittent failure in TestReplicationSupervisor#testReconciliationTaskMetrics [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl merged PR #11542:
URL: https://github.com/apache/ozone/pull/11542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org