You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "chihsuan (via GitHub)" <gi...@apache.org> on 2026/06/24 14:18:32 UTC
[PR] HDDS-13434. Wait for replicas in TestOzoneDebugShell chunk-info check [ozone]
chihsuan opened a new pull request, #10600:
URL: https://github.com/apache/ozone/pull/10600
## What changes were proposed in this pull request?
Test-only change.
**Problem.** `TestOzoneDebugShell.testChunkInfoVerifyPathsAreDifferent` intermittently failed with `expected: <3> but was: <2>`. A RATIS THREE write is acknowledged on a Ratis majority, so right after the key is written one replica may not have applied the write yet. When `ozone debug replicas chunk-info` runs in that window, `ReadContainer` on the lagging datanode fails, the datanode is dropped from the response, and the test sees only 2 distinct block file paths instead of 3.
**Fix.** Wait until all three datanodes report the block (three distinct paths) before asserting, using `GenericTestUtils.waitFor`, instead of asserting on a single read taken immediately after the write. A persistent error or an unexpected path count is reported via the wait's last observed state, so genuine regressions still surface with context rather than an opaque timeout.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13434
## How was this patch tested?
- Ran the previously-flaky test locally; passes for all `BucketLayout` values.
- Confirmed the wait is not vacuous: with the target temporarily set to an unreachable path count, the test times out as expected, proving the wait actually gates the assertion.
- `flaky-test-check` on the fork, `TestOzoneIntegrationNonHA` `ALL`, 10x10: the `OzoneDebugShell` cases (including this test) passed in all 100 iterations. https://github.com/chihsuan/ozone/actions/runs/28101143264
- One unrelated pre-existing flake in `TestListKeysWithFSO` (`BUCKET_ALREADY_EXISTS` in test setup) appeared in a single split; it is not touched by this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13434. Intermittent failure in TestOzoneDebugShell#testChunkInfoVerifyPathsAreDifferent [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #10600:
URL: https://github.com/apache/ozone/pull/10600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13434. Wait for replicas in TestOzoneDebugShell chunk-info check [ozone]
Posted by "chihsuan (via GitHub)" <gi...@apache.org>.
chihsuan commented on code in PR #10600:
URL: https://github.com/apache/ozone/pull/10600#discussion_r3480515778
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneDebugShell.java:
##########
@@ -208,27 +210,49 @@ private int runChunkInfoCommand(String volumeName, String bucketName,
private int runChunkInfoAndVerifyPaths(String volumeName, String bucketName,
String keyName) throws Exception {
- int exitCode = 1;
- try (GenericTestUtils.SystemOutCapturer capture = new GenericTestUtils
- .SystemOutCapturer()) {
- exitCode = runChunkInfoCommand(volumeName, bucketName, keyName);
- Set<String> blockFilePaths = new HashSet<>();
- String output = capture.getOutput();
- ObjectMapper objectMapper = new ObjectMapper();
- // Parse the JSON array string into a JsonNode
- JsonNode jsonNode = objectMapper.readTree(output);
- JsonNode keyLocations = jsonNode.get("keyLocations").get(0);
- for (JsonNode element : keyLocations) {
- String fileName =
- element.get("file").toString();
- blockFilePaths.add(fileName);
- }
- // DN storage directories are set differently for each DN
- // in MiniOzoneCluster as datanode-0,datanode-1,datanode-2 which is why
- // we expect 3 paths here in the set.
- assertEquals(3, blockFilePaths.size());
+ AtomicInteger exitCode = new AtomicInteger(1);
+ AtomicInteger lastPathCount = new AtomicInteger(-1);
+ AtomicReference<Throwable> lastError = new AtomicReference<>();
Review Comment:
Thanks for the review! @chungen0126 Yeah, these aren't for thread-safety. I picked `Atomic*` specifically because the lambda has to write `exitCode`/diagnostics back out, and a plain `int`/`Throwable` can't be reassigned inside a lambda so they need to be mutable holders.
Besides, I observed that `Atomic*` is the existing idiom for `waitFor` holders in the repo (e.g. `TestSnapshotBackgroundServices`).
Open to changing it if you have any suggestions. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13434. Wait for replicas in TestOzoneDebugShell chunk-info check [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10600:
URL: https://github.com/apache/ozone/pull/10600#discussion_r3476432573
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneDebugShell.java:
##########
@@ -208,27 +210,49 @@ private int runChunkInfoCommand(String volumeName, String bucketName,
private int runChunkInfoAndVerifyPaths(String volumeName, String bucketName,
String keyName) throws Exception {
- int exitCode = 1;
- try (GenericTestUtils.SystemOutCapturer capture = new GenericTestUtils
- .SystemOutCapturer()) {
- exitCode = runChunkInfoCommand(volumeName, bucketName, keyName);
- Set<String> blockFilePaths = new HashSet<>();
- String output = capture.getOutput();
- ObjectMapper objectMapper = new ObjectMapper();
- // Parse the JSON array string into a JsonNode
- JsonNode jsonNode = objectMapper.readTree(output);
- JsonNode keyLocations = jsonNode.get("keyLocations").get(0);
- for (JsonNode element : keyLocations) {
- String fileName =
- element.get("file").toString();
- blockFilePaths.add(fileName);
- }
- // DN storage directories are set differently for each DN
- // in MiniOzoneCluster as datanode-0,datanode-1,datanode-2 which is why
- // we expect 3 paths here in the set.
- assertEquals(3, blockFilePaths.size());
+ AtomicInteger exitCode = new AtomicInteger(1);
+ AtomicInteger lastPathCount = new AtomicInteger(-1);
+ AtomicReference<Throwable> lastError = new AtomicReference<>();
Review Comment:
Thanks @chihsuan for working on this. I don't think these three variables need to be atomic. Could you walk me through the reasoning behind using atomic here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13434. Intermittent failure in TestOzoneDebugShell#testChunkInfoVerifyPathsAreDifferent [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10600:
URL: https://github.com/apache/ozone/pull/10600#issuecomment-4834719088
Thanks @chihsuan for the patch, @chungen0126 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13434. Intermittent failure in TestOzoneDebugShell#testChunkInfoVerifyPathsAreDifferent [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #10600:
URL: https://github.com/apache/ozone/pull/10600#discussion_r3492831235
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneDebugShell.java:
##########
@@ -208,27 +210,49 @@ private int runChunkInfoCommand(String volumeName, String bucketName,
private int runChunkInfoAndVerifyPaths(String volumeName, String bucketName,
String keyName) throws Exception {
- int exitCode = 1;
- try (GenericTestUtils.SystemOutCapturer capture = new GenericTestUtils
- .SystemOutCapturer()) {
- exitCode = runChunkInfoCommand(volumeName, bucketName, keyName);
- Set<String> blockFilePaths = new HashSet<>();
- String output = capture.getOutput();
- ObjectMapper objectMapper = new ObjectMapper();
- // Parse the JSON array string into a JsonNode
- JsonNode jsonNode = objectMapper.readTree(output);
- JsonNode keyLocations = jsonNode.get("keyLocations").get(0);
- for (JsonNode element : keyLocations) {
- String fileName =
- element.get("file").toString();
- blockFilePaths.add(fileName);
- }
- // DN storage directories are set differently for each DN
- // in MiniOzoneCluster as datanode-0,datanode-1,datanode-2 which is why
- // we expect 3 paths here in the set.
- assertEquals(3, blockFilePaths.size());
+ AtomicInteger exitCode = new AtomicInteger(1);
+ AtomicInteger lastPathCount = new AtomicInteger(-1);
+ AtomicReference<Throwable> lastError = new AtomicReference<>();
Review Comment:
Thanks @chihsuan for the explanation! Since it aligns with the existing idiom in the repo, keeping it as Atomic* looks good to me. Approved!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13434. Intermittent failure in TestOzoneDebugShell#testChunkInfoVerifyPathsAreDifferent [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10600:
URL: https://github.com/apache/ozone/pull/10600#issuecomment-4834156179
> * One unrelated pre-existing flake in `TestListKeysWithFSO` (`BUCKET_ALREADY_EXISTS` in test setup) appeared in a single split; it is not touched by this change.
Created HDDS-15709.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org