You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sarvekshayr (via GitHub)" <gi...@apache.org> on 2025/08/05 06:46:48 UTC
[PR] HDDS-13533. Show the summary of replicas verify checks [ozone]
sarvekshayr opened a new pull request, #8898:
URL: https://github.com/apache/ozone/pull/8898
## What changes were proposed in this pull request?
A replica verification summary is now printed at the end of the command to provide a quick overview of replica health, eliminating the need to manually parse the full JSON output.
## What is the link to the Apache JIRA
[HDDS-13533](https://issues.apache.org/jira/browse/HDDS-13533)
## How was this patch tested?
Tested on a docker cluster.
Sample output:
When all key checks are passed -
```
bash-5.1$ ozone debug replicas verify --checksums --block-existence --container-state /
{
"keys" : [ ],
"pass" : true
}
***************************************************
REPLICA VERIFICATION SUMMARY
***************************************************
Status: Success
Verification Scope: All Volumes
Verification Types: checksum, blockExistence, containerState
URI: /
Number of Volumes processed: 3
Number of Buckets processed: 4
Number of Keys processed: 20
Keys passed verification: 20
Keys failed verification: 0
Total Execution time: 00:00:01,119
***************************************************
```
If a check fails for a particular key -
```
bash-5.1$ ozone debug replicas verify --checksums --block-existence --container-state /
{
"keys" : [ {
"volumeName" : "vol-1-66790",
"bucketName" : "bucket-1-95566",
"name" : "key-4-27187",
"blocks" : [ {
"containerID" : 1,
"blockID" : 115816896921600018,
"replicas" : [ {
"datanode" : {
"uuid" : "2779b09e-e6e0-45ab-b92b-044ff8f1d1e5",
"hostname" : "ozone-datanode-1.ozone_default"
},
"checks" : [ {
"type" : "checksum",
"completed" : true,
"pass" : false,
"failures" : [ {
"message" : "Checksum mismatched: this.checksums(0) != that.checksums(0), thisStartIndex=0, this=ChecksumData{CRC32, bytesPerChecksum=16384, checksums=[E9 0B B7 A4]}, that=ChecksumData{CRC32, bytesPerChecksum=16384, checksums=[E6 0E 0A DC]}"
} ]
}, {
"type" : "blockExistence",
"completed" : true,
"pass" : true,
"failures" : [ ]
}, {
"type" : "containerState",
"completed" : true,
"pass" : true,
"failures" : [ ]
} ],
"replicaIndex" : 0
} ]
} ],
"pass" : false
} ],
"pass" : false
}
***************************************************
REPLICA VERIFICATION SUMMARY
***************************************************
Status: Completed with failures
Verification Scope: All Volumes
Verification Types: checksum, blockExistence, containerState
URI: /
Number of Volumes processed: 3
Number of Buckets processed: 4
Number of Keys processed: 20
Keys passed verification: 19
Keys failed verification: 1
Keys failed checksum verification: 1
Note: A key may fail multiple verification types, so total may exceed overall failures.
Total Execution time: 00:00:03,061
***************************************************
```
Suppose the command is aborted midway, a partial summary is printed with stats based on the completed verifications.
```
bash-5.1$ ozone debug replicas verify --checksums --block-existence --container-state /
^C
***************************************************
REPLICA VERIFICATION SUMMARY
***************************************************
Status: Success
Verification Scope: All Volumes
Verification Types: checksum, blockExistence, containerState
URI: /
Number of Volumes processed: 2
Number of Buckets processed: 1
Number of Keys processed: 5
Keys passed verification: 5
Keys failed verification: 0
Total Execution time: 00:00:00,775
***************************************************
```
Ran checksum verifier on the specified key -
```
bash-5.1$ ozone debug replicas verify --checksums vol-1-66790/bucket-1-95566/key-4-27187
{
"keys" : [ {
"volumeName" : "vol-1-66790",
"bucketName" : "bucket-1-95566",
"name" : "key-4-27187",
"blocks" : [ {
"containerID" : 1,
"blockID" : 115816896921600018,
"replicas" : [ {
"datanode" : {
"uuid" : "2779b09e-e6e0-45ab-b92b-044ff8f1d1e5",
"hostname" : "ozone-datanode-1.ozone_default"
},
"checks" : [ {
"type" : "checksum",
"completed" : true,
"pass" : false,
"failures" : [ {
"message" : "Checksum mismatched: this.checksums(0) != that.checksums(0), thisStartIndex=0, this=ChecksumData{CRC32, bytesPerChecksum=16384, checksums=[E9 0B B7 A4]}, that=ChecksumData{CRC32, bytesPerChecksum=16384, checksums=[E6 0E 0A DC]}"
} ]
} ],
"replicaIndex" : 0
} ]
} ],
"pass" : false
} ],
"pass" : false
}
***************************************************
REPLICA VERIFICATION SUMMARY
***************************************************
Status: Completed with failures
Verification Scope: Key
Verification Types: checksum
URI: vol-1-66790/bucket-1-95566/key-4-27187
Number of Volumes processed: 0
Number of Buckets processed: 0
Number of Keys processed: 1
Keys passed verification: 0
Keys failed verification: 1
Keys failed checksum verification: 1
Note: A key may fail multiple verification types, so total may exceed overall failures.
Total Execution time: 00:00:02,762
***************************************************
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13533. Show the summary of replicas verify checks [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on PR #8898:
URL: https://github.com/apache/ozone/pull/8898#issuecomment-3248822075
Thanks for the patch @sarvekshayr and the reviews @errose28 @priyeshkaratha @dombizita
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13533. Show the summary of replicas verify checks [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya merged PR #8898:
URL: https://github.com/apache/ozone/pull/8898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13533. Show the summary of replicas verify checks [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on code in PR #8898:
URL: https://github.com/apache/ozone/pull/8898#discussion_r2265951697
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/replicas/ReplicasVerify.java:
##########
@@ -72,22 +83,70 @@ public class ReplicasVerify extends Handler {
private List<ReplicaVerifier> replicaVerifiers;
+ private static final String DURATION_FORMAT = "HH:mm:ss,SSS";
+ private long startTime;
+ private long endTime;
+ private String verificationScope;
+ private final List<String> verificationTypes = new ArrayList<>();
+ private final AtomicInteger volumesProcessed = new AtomicInteger(0);
+ private final AtomicInteger bucketsProcessed = new AtomicInteger(0);
+ private final AtomicInteger keysProcessed = new AtomicInteger(0);
+ private final AtomicInteger keysPassed = new AtomicInteger(0);
+ private final AtomicInteger keysFailed = new AtomicInteger(0);
+ private final Map<String, AtomicInteger> failuresByType = new ConcurrentHashMap<>();
+ private volatile Throwable exception;
+
@Override
protected void execute(OzoneClient client, OzoneAddress address) throws IOException {
+ startTime = System.nanoTime();
+
+ if (!address.getKeyName().isEmpty()) {
+ verificationScope = "Key";
+ } else if (!address.getBucketName().isEmpty()) {
+ verificationScope = "Bucket";
+ } else if (!address.getVolumeName().isEmpty()) {
+ verificationScope = "Volume";
+ } else {
+ verificationScope = "All Volumes";
+ }
+
replicaVerifiers = new ArrayList<>();
if (verification.doExecuteChecksums) {
- replicaVerifiers.add(new ChecksumVerifier(getConf()));
+ ChecksumVerifier checksumVerifier = new ChecksumVerifier(getConf());
+ replicaVerifiers.add(checksumVerifier);
+ String checksumType = checksumVerifier.getType();
+ verificationTypes.add(checksumType);
+ failuresByType.put(checksumType, new AtomicInteger(0));
}
if (verification.doExecuteBlockExistence) {
- replicaVerifiers.add(new BlockExistenceVerifier(getConf()));
+ BlockExistenceVerifier blockVerifier = new BlockExistenceVerifier(getConf());
+ replicaVerifiers.add(blockVerifier);
+ String blockType = blockVerifier.getType();
+ verificationTypes.add(blockType);
+ failuresByType.put(blockType, new AtomicInteger(0));
}
+
if (verification.doExecuteReplicaState) {
- replicaVerifiers.add(new ContainerStateVerifier(getConf(), containerCacheSize));
+ ContainerStateVerifier stateVerifier = new ContainerStateVerifier(getConf(), containerCacheSize);
+ replicaVerifiers.add(stateVerifier);
+ String stateType = stateVerifier.getType();
+ verificationTypes.add(stateType);
+ failuresByType.put(stateType, new AtomicInteger(0));
}
- findCandidateKeys(client, address);
+ // Add shutdown hook to ensure summary is printed even if interrupted
+ addShutdownHook();
+
+ try {
+ findCandidateKeys(client, address);
+ endTime = System.nanoTime();
+ } catch (Exception e) {
+ exception = e;
+ endTime = System.nanoTime();
+ throw e;
+ }
Review Comment:
The capturing of endTime can be moved to a finally block.
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/replicas/ReplicasVerify.java:
##########
@@ -207,15 +271,95 @@ void processKey(OzoneClient ozoneClient, String volumeName, String bucketName, S
}
keyNode.put("pass", keyPass);
- if (!keyPass) {
+ if (keyPass) {
+ keysPassed.incrementAndGet();
+ } else {
+ keysFailed.incrementAndGet();
allKeysPassed.set(false);
+
+ for (String failedType : failedVerificationTypes) {
+ AtomicInteger counter = failuresByType.get(failedType);
+ if (counter != null) {
+ counter.incrementAndGet();
+ } else {
+ failuresByType.computeIfAbsent(failedType, k -> new AtomicInteger(0)).incrementAndGet();
+ }
+ }
}
if (!keyPass || allResults) {
keysArray.add(keyNode);
}
}
+ /**
+ * Adds ShutdownHook to print summary statistics.
+ */
+ private void addShutdownHook() {
+ ShutdownHookManager.get().addShutdownHook(() -> {
+ if (endTime == 0) {
+ endTime = System.nanoTime();
+ }
+ printSummary(System.out);
Review Comment:
+1 to this
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/replicas/ReplicasVerify.java:
##########
@@ -207,15 +271,95 @@ void processKey(OzoneClient ozoneClient, String volumeName, String bucketName, S
}
keyNode.put("pass", keyPass);
- if (!keyPass) {
+ if (keyPass) {
+ keysPassed.incrementAndGet();
+ } else {
+ keysFailed.incrementAndGet();
allKeysPassed.set(false);
+
+ for (String failedType : failedVerificationTypes) {
+ AtomicInteger counter = failuresByType.get(failedType);
+ if (counter != null) {
+ counter.incrementAndGet();
+ } else {
+ failuresByType.computeIfAbsent(failedType, k -> new AtomicInteger(0)).incrementAndGet();
+ }
+ }
Review Comment:
The null check is not needed as computIfAbsent will handle that case,
```suggestion
failedVerificationTypes.forEach(failedType -> failuresByType
.computeIfAbsent(failedType, k -> new AtomicInteger(0))
.incrementAndGet()
);
```
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/replicas/ReplicasVerify.java:
##########
@@ -72,22 +83,70 @@ public class ReplicasVerify extends Handler {
private List<ReplicaVerifier> replicaVerifiers;
+ private static final String DURATION_FORMAT = "HH:mm:ss,SSS";
+ private long startTime;
+ private long endTime;
+ private String verificationScope;
+ private final List<String> verificationTypes = new ArrayList<>();
+ private final AtomicInteger volumesProcessed = new AtomicInteger(0);
+ private final AtomicInteger bucketsProcessed = new AtomicInteger(0);
+ private final AtomicInteger keysProcessed = new AtomicInteger(0);
+ private final AtomicInteger keysPassed = new AtomicInteger(0);
+ private final AtomicInteger keysFailed = new AtomicInteger(0);
+ private final Map<String, AtomicInteger> failuresByType = new ConcurrentHashMap<>();
+ private volatile Throwable exception;
+
@Override
protected void execute(OzoneClient client, OzoneAddress address) throws IOException {
+ startTime = System.nanoTime();
+
+ if (!address.getKeyName().isEmpty()) {
+ verificationScope = "Key";
+ } else if (!address.getBucketName().isEmpty()) {
+ verificationScope = "Bucket";
+ } else if (!address.getVolumeName().isEmpty()) {
+ verificationScope = "Volume";
+ } else {
+ verificationScope = "All Volumes";
+ }
+
replicaVerifiers = new ArrayList<>();
if (verification.doExecuteChecksums) {
- replicaVerifiers.add(new ChecksumVerifier(getConf()));
+ ChecksumVerifier checksumVerifier = new ChecksumVerifier(getConf());
+ replicaVerifiers.add(checksumVerifier);
+ String checksumType = checksumVerifier.getType();
+ verificationTypes.add(checksumType);
+ failuresByType.put(checksumType, new AtomicInteger(0));
Review Comment:
Code similar to this is repeated a few times, we can extract this code out to a method, something like below:
```
// introduce below method
private void addVerifier(boolean condition, Supplier<ReplicaVerifier> verifierSupplier) {
if flag is true:
then perform the required steps
}
// call the method as below:
addVerifier(verification.doExecuteChecksums, () -> new ChecksumVerifier(getConf()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13533. Show the summary of replicas verify checks [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8898:
URL: https://github.com/apache/ozone/pull/8898#discussion_r2261294953
##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/replicas/ReplicasVerify.java:
##########
@@ -207,15 +271,95 @@ void processKey(OzoneClient ozoneClient, String volumeName, String bucketName, S
}
keyNode.put("pass", keyPass);
- if (!keyPass) {
+ if (keyPass) {
+ keysPassed.incrementAndGet();
+ } else {
+ keysFailed.incrementAndGet();
allKeysPassed.set(false);
+
+ for (String failedType : failedVerificationTypes) {
+ AtomicInteger counter = failuresByType.get(failedType);
+ if (counter != null) {
+ counter.incrementAndGet();
+ } else {
+ failuresByType.computeIfAbsent(failedType, k -> new AtomicInteger(0)).incrementAndGet();
+ }
+ }
}
if (!keyPass || allResults) {
keysArray.add(keyNode);
}
}
+ /**
+ * Adds ShutdownHook to print summary statistics.
+ */
+ private void addShutdownHook() {
+ ShutdownHookManager.get().addShutdownHook(() -> {
+ if (endTime == 0) {
+ endTime = System.nanoTime();
+ }
+ printSummary(System.out);
Review Comment:
We should print the summary to stderr so the json output can be redirected into tools like jq and does not need to be manually trimmed off. Even though it is not technically an "error", this pattern is used with standard commands like `time`, which prints its execution time summary to stderr to avoid interfering with the output of the command it was timing.
```suggestion
printSummary(System.err);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org