You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "errose28 (via GitHub)" <gi...@apache.org> on 2023/07/18 18:32:12 UTC
[GitHub] [ozone] errose28 opened a new pull request, #5086: HDDS-9022. DiskChecker incorrectly reporting errors.
errose28 opened a new pull request, #5086:
URL: https://github.com/apache/ozone/pull/5086
## What changes were proposed in this pull request?
Volume scanner was incorrectly started before volumes were initialized in the datanode version endpoint task. This change moves volume scanner startup after this completes. This is the same place that starts the container scanners.
## What is the link to the Apache JIRA
HDDS-9022
## How was this patch tested?
WIP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #5086: HDDS-9022. DiskChecker incorrectly reporting errors.
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #5086:
URL: https://github.com/apache/ozone/pull/5086#issuecomment-1640747730
Hi @mladjan-gadzic @xBis7 would you be able to test this patch on your setup that initially found the issue? Automated testing of a race like this will be difficult.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #5086: HDDS-9022. DiskChecker incorrectly reporting errors.
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #5086:
URL: https://github.com/apache/ozone/pull/5086#issuecomment-1641921773
Hey @errose28, this patch addresses another issue, but not the one we are experiencing. Let me provide some more info. I have tested it both on docker and on our cluster applying the following diff on top of your patch
```
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java
index e5f0046c0..fb12d15e4 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java
@@ -236,7 +236,7 @@ public void setBlockDeletionLimit(int limit) {
}
@Config(key = "periodic.disk.check.interval.minutes",
- defaultValue = "60",
+ defaultValue = "2",
type = ConfigType.LONG,
tags = { DATANODE },
description = "Periodic disk check run interval in minutes."
@@ -314,7 +314,7 @@ public void setBlockDeletionLimit(int limit) {
DISK_CHECK_FILE_SIZE_DEFAULT;
@Config(key = "disk.check.min.gap",
- defaultValue = "10m",
+ defaultValue = "1m",
type = ConfigType.TIME,
tags = { DATANODE },
description = "The minimum gap between two successive checks of the same"
diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/DiskCheckUtil.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/DiskCheckUtil.java
index b567841c0..81f4a877d 100644
--- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/DiskCheckUtil.java
+++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/DiskCheckUtil.java
@@ -132,6 +132,13 @@ public boolean checkPermissions(File storageDir) {
public boolean checkReadWrite(File storageDir,
File testFileDir, int numBytesToWrite) {
File testFile = new File(testFileDir, "disk-check-" + UUID.randomUUID());
+
+ String storageDirPath = storageDir == null ? "null" : storageDir.getAbsolutePath();
+ String testFileDirPath = testFileDir == null ? "null" : testFileDir.getAbsolutePath();
+ LOG.info("\nxbis: storageDir: " + storageDirPath +
+ "\nxbis: testFileDir: " + testFileDirPath +
+ "\nxbis: testFile: " + testFile.getAbsolutePath());
+
byte[] writtenBytes = new byte[numBytesToWrite];
RANDOM.nextBytes(writtenBytes);
try (FileOutputStream fos = new FileOutputStream(testFile)) {
```
* Docker logs
```
2023-07-19 10:54:02,032 [Periodic HDDS volume checker] INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/hdds/hdds
2023-07-19 10:54:02,035 [Periodic HDDS volume checker] INFO volume.StorageVolumeChecker: Scheduled health check for volume /data/hdds/hdds
2023-07-19 10:54:02,035 [DataNode DiskChecker thread 1] INFO utils.DiskCheckUtil:
xbis: storageDir: /data/hdds/hdds
xbis: testFileDir: /data/hdds/hdds/CID-3e323276-e1b0-4069-9fa1-1d93ccab4ff5/tmp/disk-check
xbis: testFile: /data/hdds/hdds/CID-3e323276-e1b0-4069-9fa1-1d93ccab4ff5/tmp/disk-check/disk-check-8e3d58b6-5b62-49de-825f-36e2f85d0d3a
2023-07-19 10:54:02,042 [Periodic HDDS volume checker] INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/metadata/ratis
2023-07-19 10:54:02,042 [Periodic HDDS volume checker] INFO volume.StorageVolumeChecker: Scheduled health check for volume /data/metadata/ratis
2023-07-19 10:54:02,043 [DataNode DiskChecker thread 1] INFO utils.DiskCheckUtil:
xbis: storageDir: /data/metadata/ratis
xbis: testFileDir: null
xbis: testFile: /opt/hadoop/disk-check-ec8c7aba-69ec-4f19-bf59-c5b31f843ef1
```
* Our cluster's logs
```
2023-07-19 11:32:33,529 [EndpointStateMachine task thread for ip-10-0-88-113.eu-west-1.compute.internal/10.0.88.113:9861 - 0 ] INFO org.apache.hadoop.ozone.container.common.volume.ThrottledAsyncChecker: Scheduling a check for /hadoop/ozone/data/disk1/datanode/data/hdds
2023-07-19 11:32:33,536 [DataNode DiskChecker thread 0] INFO org.apache.hadoop.ozone.container.common.utils.DiskCheckUtil:
xbis: storageDir: /hadoop/ozone/data/disk1/datanode/data/hdds
xbis: testFileDir: /hadoop/ozone/data/disk1/datanode/data/hdds/CID-e79cbfc1-02dd-4721-a796-e6790132940f/tmp/disk-check
xbis: testFile: /hadoop/ozone/data/disk1/datanode/data/hdds/CID-e79cbfc1-02dd-4721-a796-e6790132940f/tmp/disk-check/disk-check-fc315b6d-0108-4d02-b443-571083808ad4
2023-07-19 11:32:33,539 [EndpointStateMachine task thread for ip-10-0-88-113.eu-west-1.compute.internal/10.0.88.113:9861 - 0 ] INFO org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker: Scheduled health check for volume /hadoop/ozone/data/disk1/datanode/data/hdds
2023-07-19 11:32:33,543 [EndpointStateMachine task thread for ip-10-0-88-113.eu-west-1.compute.internal/10.0.88.113:9861 - 0 ] INFO org.apache.hadoop.ozone.container.common.volume.ThrottledAsyncChecker: Scheduling a check for /hadoop/ozone/data/disk1/datanode
2023-07-19 11:32:33,543 [EndpointStateMachine task thread for ip-10-0-88-113.eu-west-1.compute.internal/10.0.88.113:9861 - 0 ] INFO org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker: Scheduled health check for volume /hadoop/ozone/data/disk1/datanode
2023-07-19 11:32:33,543 [DataNode DiskChecker thread 0] INFO org.apache.hadoop.ozone.container.common.utils.DiskCheckUtil:
xbis: storageDir: /hadoop/ozone/data/disk1/datanode
xbis: testFileDir: null
xbis: testFile: /home/ozone/.ansible/tmp/ansible-moduletmp-1689766348.51-vg_D0x/disk-check-a767a491-e84f-4d5b-b04e-fe539dd05c83
2023-07-19 11:32:33,544 [DataNode DiskChecker thread 0] ERROR org.apache.hadoop.ozone.container.common.utils.DiskCheckUtil: Volume /hadoop/ozone/data/disk1/datanode failed health check. Could not find file /home/ozone/.ansible/tmp/ansible-moduletmp-1689766348.51-vg_D0x/disk-check-a767a491-e84f-4d5b-b04e-fe539dd05c83 for volume check.
java.io.FileNotFoundException: disk-check-a767a491-e84f-4d5b-b04e-fe539dd05c83 (No such file or directory)
```
`testFileDir` get's initialized for an HddsVolume and a DbVolume but not a StorageVolume. For a StorageVolume, its value is null and that results in creating the `testFile` to the current working directory of the java application. For docker that's `/opt/hadoop/` which is fine but for our cluster the current dir is not writable. That might be a common case for others as well. IMO, for a StorageVolume we should provide a specific path on disk or create a separate tmp dir or set the `testFileDir` to point to `storageDir`.
`VersionEndpointTask` calls `checkVolumeSet` [here](https://github.com/apache/ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/states/endpoint/VersionEndpointTask.java#L84-L88) only for DbVolumes and HddsVolumes but not StorageVolumes. We end up calling `checkVolume` which initializes tmp dirs [here](https://github.com/apache/ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/StorageVolumeUtil.java#L273). That's why there is no value for a StorageVolume.
BTW, this is an aws cluster that has been set up with ansible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 merged pull request #5086: HDDS-9022. DiskChecker incorrectly reporting errors.
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 merged PR #5086:
URL: https://github.com/apache/ozone/pull/5086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #5086: HDDS-9022. DiskChecker incorrectly reporting errors.
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #5086:
URL: https://github.com/apache/ozone/pull/5086#issuecomment-1640786223
@errose28 Thanks for the patch. This has been a blocker for us, since it was using a non writable dir for the test file and making all of our datanodes to incorrectly shut down. I will test it out in our cluster and get back to you by tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #5086: HDDS-9022. DiskChecker incorrectly reporting errors.
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #5086:
URL: https://github.com/apache/ozone/pull/5086#issuecomment-1642445739
Looks the issue only happens for metadata volumes after this patch. This has to do with the different layout and initialization of the metadata volume. I will update the patch to handle this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org