You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "duongkame (via GitHub)" <gi...@apache.org> on 2024/11/22 06:17:04 UTC
[PR] HDDS-11485. Datanode doesn't report volume unhealthy when V3 DB folder doesn't exist [ozone]
duongkame opened a new pull request, #7240:
URL: https://github.com/apache/ozone/pull/7240
## What changes were proposed in this pull request?
See HDDS-11485.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11485
## How was this patch tested?
Tested locally with the following scenario:
1. Create a test docker cluster with 3 datanodes, each datanodes has a volume configured at `/data/hdds/hdds`.
2. Go to a datanode, remove the database folder under the volume: `/data/hdds/hdds/CID-<cluster id>/DS-<storage id>`. Then restart the datanode.
Result.
Before this fix, the behavior is exactly as described as per HDDS-11485. The datanode loads the volume, and detects the missing database folder but doesn't report the it as unhealthy. When client writes data to the datanodes, ratis log crashes because of the NPE.
After this fix, when datanode loads the volume and see the missing datanodes. It reports the volume as unhealthy immediately, see datanode logs:
```
2024-09-26 15:44:41 2024-09-26 22:44:41,109 [ForkJoinPool.commonPool-worker-19] INFO volume.ThrottledAsyncChecker: Scheduling a check for /data/hdds/hdds
2024-09-26 15:44:41 2024-09-26 22:44:41,127 [ForkJoinPool.commonPool-worker-19] ERROR ozoneimpl.OzoneContainer: Load db store for HddsVolume /data/hdds/hdds failed
2024-09-26 15:44:41 java.io.IOException: Db parent dir /data/hdds/hdds/CID-730e257b-8168-41e8-b7e4-635a26af2f9b/DS-f053f5a7-efff-479a-8cef-e5d8b2ab852f not found for HddsVolume: /data/hdds/hdds
2024-09-26 15:44:41 at org.apache.hadoop.ozone.container.common.volume.HddsVolume.loadDbStore(HddsVolume.java:370)
2024-09-26 15:44:41 at org.apache.hadoop.ozone.container.common.utils.HddsVolumeUtil.loadVolume(HddsVolumeUtil.java:111)
2024-09-26 15:44:41 at org.apache.hadoop.ozone.container.common.utils.HddsVolumeUtil.lambda$loadAllHddsVolumeDbStore$0(HddsVolumeUtil.java:97)
2024-09-26 15:44:41 at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
2024-09-26 15:44:41 at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1728)
2024-09-26 15:44:41 at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
2024-09-26 15:44:41 at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
2024-09-26 15:44:41 at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
2024-09-26 15:44:41 at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
2024-09-26 15:44:41 at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
2024-09-26 15:44:41 2024-09-26 22:44:41,129 [main] INFO ozoneimpl.OzoneContainer: Load 1 volumes DbStore cost: 22ms
2024-09-26 15:44:41 2024-09-26 22:44:41,135 [58680955-2588-4bda-9101-0aabff9ec06c-DataNodeDiskCheckerThread-0] WARN volume.HddsVolume: Volume /data/hdds/hdds failed to access RocksDB: RocksDB parent directory is null, the volume might not have been loaded properly.
2024-09-26 15:44:41 2024-09-26 22:44:41,138 [58680955-2588-4bda-9101-0aabff9ec06c-VolumeCheckResultHandlerThread-0] WARN volume.StorageVolumeChecker: Volume /data/hdds/hdds detected as being unhealthy
2024-09-26 15:44:41 2024-09-26 22:44:41,139 [58680955-2588-4bda-9101-0aabff9ec06c-VolumeCheckResultHandlerThread-0] WARN volume.MutableVolumeSet: checkVolumeAsync callback got 1 failed volumes: [/data/hdds/hdds]
2024-09-26 15:44:41 2024-09-26 22:44:41,142 [58680955-2588-4bda-9101-0aabff9ec06c-VolumeCheckResultHandlerThread-0] INFO volume.MutableVolumeSet: Moving Volume : /data/hdds/hdds to failed Volumes
2024-09-26 15:44:41 2024-09-26 22:44:41,142 [58680955-2588-4bda-9101-0aabff9ec06c-VolumeCheckResultHandlerThread-0] ERROR volume.MutableVolumeSet: Not enough volumes in MutableVolumeSet. DatanodeUUID: 58680955-2588-4bda-9101-0aabff9ec06c, VolumeType: DATA_VOLUME, MaxVolumeFailuresTolerated: -1, ActiveVolumes: 0, FailedVolumes: 1
2024-09-26 15:44:41 2024-09-26 22:44:41,142 [58680955-2588-4bda-9101-0aabff9ec06c-VolumeCheckResultHandlerThread-0] ERROR statemachine.DatanodeStateMachine: DatanodeStateMachine Shutdown due to too many bad volumes, check hdds.datanode.failed.data.volumes.tolerated and hdds.datanode.failed.metadata.volumes.tolerated and hdds.datanode.failed.db.volumes.tolerated
```
In the logs above, the unhealthy volume is also escalated to a datanode failure because it's the datanode's only volume.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11485. Datanode doesn't report volume unhealthy when V3 DB folder doesn't exist [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #7240: HDDS-11485. Datanode doesn't report volume unhealthy when V3 DB folder doesn't exist
URL: https://github.com/apache/ozone/pull/7240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11485. Datanode doesn't report volume unhealthy when V3 DB folder doesn't exist [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #7240: HDDS-11485. Datanode doesn't report volume unhealthy when V3 DB folder doesn't exist
URL: https://github.com/apache/ozone/pull/7240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11485. Datanode doesn't report volume unhealthy when V3 DB folder doesn't exist [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7240:
URL: https://github.com/apache/ozone/pull/7240#issuecomment-2439074342
Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11485. Datanode doesn't report volume unhealthy when V3 DB folder doesn't exist [ozone]
Posted by "aswinshakil (via GitHub)" <gi...@apache.org>.
aswinshakil commented on code in PR #7240:
URL: https://github.com/apache/ozone/pull/7240#discussion_r1778981173
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java:
##########
@@ -357,12 +357,14 @@ public void loadDbStore(boolean readOnly) throws IOException {
File storageIdDir = new File(clusterIdDir, getStorageID());
if (!storageIdDir.exists()) {
+ dbLoadFailure.set(true);
Review Comment:
Should we add this to line 354 where the `clusterIdDir` doesn't exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11485. Datanode doesn't report volume unhealthy when V3 DB folder doesn't exist [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7240:
URL: https://github.com/apache/ozone/pull/7240#issuecomment-2542582272
Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11485. Datanode doesn't report volume unhealthy when V3 DB folder doesn't exist [ozone]
Posted by "duongkame (via GitHub)" <gi...@apache.org>.
duongkame commented on code in PR #7240:
URL: https://github.com/apache/ozone/pull/7240#discussion_r1779006058
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java:
##########
@@ -357,12 +357,14 @@ public void loadDbStore(boolean readOnly) throws IOException {
File storageIdDir = new File(clusterIdDir, getStorageID());
if (!storageIdDir.exists()) {
+ dbLoadFailure.set(true);
Review Comment:
"Nonexisting clusterIdDir" is considered as NOT_FORMATTED, and the who structure is recreated when loading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org