You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2026/10/08 00:55:59 UTC
[PR] HDDS-16780. Fix intermittent failure in TestPerVolumePushReplication#testDecommissionWithPerVolumePools [ozone]
smengcl opened a new pull request, #11441:
URL: https://github.com/apache/ozone/pull/11441
Generated-by: Claude Code (Opus 4.8)
## What changes were proposed in this pull request?
`TestPerVolumePushReplication#testDecommissionWithPerVolumePools` intermittently times out waiting for the datanode to reach `DECOMMISSIONED`.
The integration test `ozone-site.xml` uses `MockSpaceUsageCheckFactory$None`, which reports `Long.MAX_VALUE` capacity per volume. This test starts datanodes with 2 data volumes, so `SCMNodeManager#getNodeStatInternal` overflows when summing a node's storage reports and `SCMNodeStat` throws `Capacity cannot be negative`. `ReplicationManagerUtil#excludeFullNodes` calls `getNodeStat` for every datanode with a pending replica ADD, so while one replication is in flight, every other under replicated container fails processing. Decommission then moves about one container per ReplicationManager cycle (about 3s), and with 12 containers on the node it can exceed the 30s `waitForDnToReachOpState` timeout.
This patch makes the test use `MockSpaceUsageCheckFactory$HalfTera` (fixed 512 GB per volume), so the per node sum stays in range.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16780
## How was this patch tested?
- `mvn -o -pl :ozone-integration-test -am test -Dtest=TestPerVolumePushReplication`: 3/3 passed, no `Capacity cannot be negative` in the log, and the decommissioning node's 10 containers were all sufficiently replicated in the first admin monitor pass.
- Same command without the patch: passed locally, but logged 94 `Capacity cannot be negative` errors and took about 19s to replicate the 10 containers (one every 2 to 3s), close to the 30s limit. The failing CI run had 12 containers and 9 done at timeout.
- The overflow is deterministic for any datanode with 2 volumes under the `None` factory, so the fix removes the cause rather than widening a timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16780. Fix intermittent failure in TestPerVolumePushReplication#testDecommissionWithPerVolumePools [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #11441:
URL: https://github.com/apache/ozone/pull/11441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16780. Fix intermittent failure in TestPerVolumePushReplication#testDecommissionWithPerVolumePools [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11441:
URL: https://github.com/apache/ozone/pull/11441#issuecomment-6094854960
Thanks @adoroszlai for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16780. Fix intermittent failure in TestPerVolumePushReplication#testDecommissionWithPerVolumePools [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #11441:
URL: https://github.com/apache/ozone/pull/11441#issuecomment-6094733594
Thanks @smengcl for the patch. @Russole although merged, please feel free to review, we can address comments in follow-up. This test was failing frequently, would like to avoid unnecessary re-runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org