You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2025/05/22 21:06:00 UTC
[jira] [Updated] (HDDS-13062) Upon OM startup, verify ozone.metadata.dirs and ozone.om.db.dirs are in the same mount
[ https://issues.apache.org/jira/browse/HDDS-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siyao Meng updated HDDS-13062:
------------------------------
Summary: Upon OM startup, verify ozone.metadata.dirs and ozone.om.db.dirs are in the same mount (was: Upon OM startup, verify that ozone.metadata.dirs and ozone.om.db.dirs to be in the same mount)
> Upon OM startup, verify ozone.metadata.dirs and ozone.om.db.dirs are in the same mount
> --------------------------------------------------------------------------------------
>
> Key: HDDS-13062
> URL: https://issues.apache.org/jira/browse/HDDS-13062
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> h2. Problem
> When {{ozone.metadata.dirs}} and {{ozone.om.db.dirs}} are set to different mounts, the following exception would occur:
> {code}
> 2025-05-03 20:49:12,016 ERROR [pool-30-thread-1]-org.apache.hadoop.ozone.om.OzoneManager: Failed to move downloaded DB checkpoint /hadoop-ozone/datanode01/om/ozone-metadata/snapshot/om.db.candidate to metadata directory /hadoop-ozone/datanode02/om/data/om.db. Exception: {}. Resetting to original DB.
> java.nio.file.FileSystemException: /hadoop-ozone/datanode02/om/data/om.db/000079.sst -> /hadoop-ozone/datanode01/om/ozone-metadata/snapshot/om.db.candidate/000079.sst: Invalid cross-device link
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
> at java.nio.file.Files.createLink(Files.java:1086)
> at org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.linkFiles(OmSnapshotUtils.java:169)
> at org.apache.hadoop.ozone.om.OzoneManager.moveCheckpointFiles(OzoneManager.java:3880)
> at org.apache.hadoop.ozone.om.OzoneManager.replaceOMDBWithCheckpoint(OzoneManager.java:3860)
> at org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3734)
> at org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3669)
> at org.apache.hadoop.ozone.om.OzoneManager.installSnapshotFromLeader(OzoneManager.java:3646)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$5(OzoneManagerStateMachine.java:505)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> 2025-05-03 20:49:12,016 ERROR [pool-30-thread-1]-org.apache.hadoop.ozone.om.OzoneManager: Failed to install Snapshot from om1546339340 as OM failed to replace DB with downloaded checkpoint. Reloading old OM state.
> java.nio.file.FileSystemException: /hadoop-ozone/datanode02/om/data/om.db/000079.sst -> /hadoop-ozone/datanode01/om/ozone-metadata/snapshot/om.db.candidate/000079.sst: Invalid cross-device link
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
> at java.nio.file.Files.createLink(Files.java:1086)
> at org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.linkFiles(OmSnapshotUtils.java:169)
> at org.apache.hadoop.ozone.om.OzoneManager.moveCheckpointFiles(OzoneManager.java:3880)
> at org.apache.hadoop.ozone.om.OzoneManager.replaceOMDBWithCheckpoint(OzoneManager.java:3860)
> at org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3734)
> at org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3669)
> at org.apache.hadoop.ozone.om.OzoneManager.installSnapshotFromLeader(OzoneManager.java:3646)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$5(OzoneManagerStateMachine.java:505)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> At this point, the follower OM won't ever succeed bootstrapping and will lag behind the leader indefinitely.
> h2. Solution
> 1. Upon OM startup, verify that {{ozone.metadata.dirs}} and {{ozone.om.db.dirs}} are in the same mount
> 2. If possible, also check that the partition mount supports hardlinks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org