You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ethan Rose (Jira)" <ji...@apache.org> on 2024/09/24 19:36:00 UTC

[jira] [Commented] (HDDS-11485) Datanode doesn't report corrupted volume unhealthy

    [ https://issues.apache.org/jira/browse/HDDS-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884424#comment-17884424 ] 

Ethan Rose commented on HDDS-11485:
-----------------------------------

The NPE happens in memory [here|https://github.com/apache/ozone/blob/15d803b9db27240f24222e292d70dd137cc2ee95/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java#L197] before the on-demand volume scan can be triggered to touch the hardware. Background volume scan should have caught this but it [defaults to 60 minutes|https://github.com/apache/ozone/blob/3355701853c10f47c0acafbef502a3d73e48926a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java#L84] and likely did not get a chance to run if this was a short lived test cluster.

It looks like the issue is with loading the volume's information into memory before any disk check can be done.

> Datanode doesn't report corrupted volume unhealthy
> --------------------------------------------------
>
>                 Key: HDDS-11485
>                 URL: https://issues.apache.org/jira/browse/HDDS-11485
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Duong
>            Priority: Major
>
> Tried a simple test in a running cluster, I go to one datanode volume directory, delete the datastore folder (the "DS-<storage id>") under the parent folder "CID-<cluster id>".
> {code:java}
>  <Volume-Root>
>    |-CID-7c19eaaf-701a-4758-b221-9b0de17c0547
>    |---DS-0f05073c-0f96-47e4-a335-f4aa67f20e2c
>    |-----container.db
>    |-----db.checkpoints
>    |-----db.snapshots
>    |-------checkpointState
>    |---tmp
>    |-----deleted-containers
>    |-----disk-check
>  |-VERSION {code}
> and restart the datanode. The result is, when the datanode is up, it finds out that the datastore folder is missing...
> {code:java}
> 2024-09-24 10:32:13 2024-09-24 17:32:13,078 [ForkJoinPool.commonPool-worker-19] ERROR ozoneimpl.OzoneContainer: Load db store for HddsVolume /data/hdds/hdds failed
> 2024-09-24 10:32:13 java.io.IOException: Db parent dir /data/hdds/hdds/CID-7c19eaaf-701a-4758-b221-9b0de17c0547/DS-0f05073c-0f96-47e4-a335-f4aa67f20e2c not found for HddsVolume: /data/hdds/hdds
> 2024-09-24 10:32:13     at org.apache.hadoop.ozone.container.common.volume.HddsVolume.loadDbStore(HddsVolume.java:369)
> 2024-09-24 10:32:13     at org.apache.hadoop.ozone.container.common.utils.HddsVolumeUtil.loadVolume(HddsVolumeUtil.java:111)
> 2024-09-24 10:32:13     at org.apache.hadoop.ozone.container.common.utils.HddsVolumeUtil.lambda$loadAllHddsVolumeDbStore$0(HddsVolumeUtil.java:97)
> 2024-09-24 10:32:13     at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
> 2024-09-24 10:32:13     at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1728)
> 2024-09-24 10:32:13     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
> 2024-09-24 10:32:13     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
> 2024-09-24 10:32:13     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
> 2024-09-24 10:32:13     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
> 2024-09-24 10:32:13     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
> 2024-09-24 10:32:13 2024-09-24 17:32:13,079 [main] INFO ozoneimpl.OzoneContainer: Load 1 volumes DbStore cost: 17ms {code}
> but the error is ignored and the volume is still loaded and used for container creation and data writes. The problematic volume can't handle the container creation and result in a NPE. 
> {code:java}
> 2024-09-12 22:19:48,326 WARN [0823d579-50b5-4c90-ab9d-17b012fd5f85-ChunkWriter-6-0]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Operation: CreateContainer , Trace ID:  , Message: java.lang.NullPointerException: Base Directory cannot be null , Result: CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: java.lang.NullPointerException: Base Directory cannot be null
>     at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:234)
>     at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:504)
>     at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:300)
>     at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:195)
>     at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
>     at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:194)
>     at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:505)
>     at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$writeStateMachineData$3(ContainerStateMachine.java:559)
>     at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>     at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException: Base Directory cannot be null
>     at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:921)
>     at org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerLocationUtil.getContainerDBFile(KeyValueContainerLocationUtil.java:127)
>     at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.getContainerDBFile(KeyValueContainer.java:938)
>     at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:197)
>     at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:380)
>     at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:248)
>     at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:231)  {code}
> as this container creation happens during ContainerStateMachine#write (state machine serialization), the NPE crashed the Pipeline's raft log and put the Pipeline in an unusable state. 
> The unhealthy volume should be detected and excluded during container creation or any StateMachine operation. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org