You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sumitagrawl (via GitHub)" <gi...@apache.org> on 2025/04/30 09:01:59 UTC
[PR] HDDS-12924. datanode used space calculation optimization [ozone]
sumitagrawl opened a new pull request, #8365:
URL: https://github.com/apache/ozone/pull/8365
## What changes were proposed in this pull request?
This PR consists of datanode used space calculation" approach. Based on the final approach, need further implement the solution
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12924
## How was this patch tested?
NA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #8365:
URL: https://github.com/apache/ozone/pull/8365#issuecomment-2858460400
As discussion over community meeting, Approach 2 is preferred, where ozone usages will be DU for metadata + summation of all container data size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2151321917
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/DUOptimized.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.fs;
+
+import java.io.File;
+import java.util.function.Supplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Make use of DU class that uses the unix 'du' program to calculate space usage of metadata excluding container data.
+ * Container data usages is identified from container set.
+ *
+ * @see SpaceUsageSource
+ */
+public class DUOptimized implements SpaceUsageSource {
+ private static final Logger LOG = LoggerFactory.getLogger(DUOptimized.class);
+
+ private final DU metaPathDU;
+ private Supplier<Supplier<Long>> containerUsedSpaceProvider;
+
+ public DUOptimized(File path, Supplier<File> exclusionProvider) {
+ metaPathDU = new DU(exclusionProvider, path);
+ }
+
+ @Override
+ public long getUsedSpace() {
+ long metaPathSize = metaPathDU.getUsedSpace();
+ if (null == containerUsedSpaceProvider) {
+ return metaPathSize;
+ }
+ Supplier<Long> gatherContainerUsages = containerUsedSpaceProvider.get();
+ long containerUsedSpace = gatherContainerUsages.get();
+ LOG.info("Disk du usages {}, container data usages {}", metaPathSize, containerUsedSpace);
Review Comment:
```suggestion
LOG.info("Disk metaPath du usages {}, container data usages {}", metaPathSize, containerUsedSpace);
```
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java:
##########
@@ -405,6 +408,38 @@ public long getFreeSpaceToSpare(long volumeCapacity) {
return getDatanodeConfig().getMinFreeSpace(volumeCapacity);
}
+ @Override
+ public void setGatherContainerUsages(Function<HddsVolume, Long> gatherContainerUsages) {
+ this.gatherContainerUsages = gatherContainerUsages;
+ }
+
+ @Override
+ protected long containerUsedSpace() {
+ return gatherContainerUsages.apply(this);
+ }
+
+ @Override
+ public File getContainerDirsPath() {
+ if (getStorageState() != VolumeState.NORMAL) {
+ return null;
+ }
+ File hddsVolumeRootDir = getHddsRootDir();
+ //filtering storage directory
+ File[] storageDirs = hddsVolumeRootDir.listFiles(File::isDirectory);
+ if (storageDirs == null) {
+ LOG.error("IO error for the volume {}, directory not found", hddsVolumeRootDir);
+ return null;
+ }
+ File clusterIDDir = new File(hddsVolumeRootDir, getClusterID());
+ if (storageDirs.length == 1 && !clusterIDDir.exists()) {
+ // If this volume was formatted pre SCM HA, this will be the SCM ID.
+ // A cluster ID symlink will exist in this case only if this cluster is finalized for SCM HA.
+ // If the volume was formatted post SCM HA, this will be the cluster ID.
+ clusterIDDir = storageDirs[0];
+ }
+ return new File(clusterIDDir, Storage.STORAGE_DIR_CURRENT);
Review Comment:
What happens if `clusterIDDir` doesn't exist and `storageDirs.length != 1`?
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/DU.java:
##########
@@ -96,11 +106,21 @@ public long getUsed() {
@Override
public String toString() {
+ if (exclusionProvider != null) {
+ return String.join(" ", getExecString()) + "\n" + value.get() + "\t" + getPath();
+ }
Review Comment:
No need to have condition here as `getExecString` already handles with and without `exclusionProvider.`
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsageCheckParams.java:
##########
@@ -19,6 +19,7 @@
import static com.google.common.base.Preconditions.checkArgument;
+import com.google.common.base.Supplier;
Review Comment:
```suggestion
import java.util.function.Supplier;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2153683267
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java:
##########
@@ -405,6 +408,38 @@ public long getFreeSpaceToSpare(long volumeCapacity) {
return getDatanodeConfig().getMinFreeSpace(volumeCapacity);
}
+ @Override
+ public void setGatherContainerUsages(Function<HddsVolume, Long> gatherContainerUsages) {
+ this.gatherContainerUsages = gatherContainerUsages;
+ }
+
+ @Override
+ protected long containerUsedSpace() {
+ return gatherContainerUsages.apply(this);
+ }
+
+ @Override
+ public File getContainerDirsPath() {
+ if (getStorageState() != VolumeState.NORMAL) {
+ return null;
+ }
+ File hddsVolumeRootDir = getHddsRootDir();
+ //filtering storage directory
+ File[] storageDirs = hddsVolumeRootDir.listFiles(File::isDirectory);
+ if (storageDirs == null) {
+ LOG.error("IO error for the volume {}, directory not found", hddsVolumeRootDir);
+ return null;
+ }
+ File clusterIDDir = new File(hddsVolumeRootDir, getClusterID());
+ if (storageDirs.length == 1 && !clusterIDDir.exists()) {
+ // If this volume was formatted pre SCM HA, this will be the SCM ID.
+ // A cluster ID symlink will exist in this case only if this cluster is finalized for SCM HA.
+ // If the volume was formatted post SCM HA, this will be the cluster ID.
+ clusterIDDir = storageDirs[0];
+ }
+ return new File(clusterIDDir, Storage.STORAGE_DIR_CURRENT);
Review Comment:
volume will be marked failed if it does not exist. This is done at org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader#readVolume while loading volume and container set.
In begning of this method, we check volume to be NORMAL state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2144710652
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/MutableVolumeSet.java:
##########
@@ -18,6 +18,7 @@
package org.apache.hadoop.ozone.container.common.volume;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Function;
Review Comment:
Should use `java.util.function.Function` instead, same as other files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2074750561
##########
hadoop-hdds/docs/content/design/dn-usedspace-calculation.md:
##########
@@ -0,0 +1,87 @@
+---
+title: Datanode used space calculation
+summary: Describe proposal for optimization in datanode used space calculation.
+date: 2025-04-30
+jira: HDDS-12924
+status: draft
+author: Sumit Agrawal
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+Datanode makes use of DU to identify disk space uses for ozone in the volume.
+DU is a time-consuming operation if run over large occupied disk, like when running over disk path having 10s of TB container data. This will be slow operation.
+
+
+## Approach 1: Run DU over non-ozone path
+
+Challenges:
+
+- Root path is not known for the device mounted on. So this needs extra configuration.
+- Permission problem: du will not count the space if permission is not there on a certain path.
+
+
+Based on the above concern, it's `not feasible` to do du over non-ozone path.
+
+
+## Approach 2: Run DU over meta path only (excluding container dir path)
+
+Ozone space usages includes summation of:
+- Sum of all Container data size as present in memory
+- DU over volume path as current (excluding container path)
+
+`Used space = sum(<all containers data size>, <du over ozone path excluding container data>)`
+
+### Limitation:
+- Space is not counted as ozone space for below cases:
+ - Ozone used size for duplicate containers are not counted (ignored during startup for EC case)
+ - Ozone used size for containers corrupted are not counted, not deleted
+ - Container path meta files like container yaml, exported rocks db’s sst files, are not counted, which might result in few GB of data in a large cluster.
+
+ These spaces will be added up to non-ozone used space, and especially un-accounted containers need to be cleaned up.
+
+In future, the container space (due to duplicacy or corruption) needs to be removed based on error logs.
+
+
+### Impact of inaccuracy of Used Space
+
+1. Used space in reporting (may be few GB as per above)
+2. Adjustment between Ozone available space and Reserved available space
+
+This inaccuracy does not have much impact over solution (as its present in existing) and due to nature of “du” running async and parallel write operation being in progress.
+
+Approach to fits better in this scenario, can be provided as `OptimizedDU` and keeping previous support as well.
+
+## Approach 3 : Used space will be only container data
Review Comment:
Used space has two purposes, one is let user know how many data are saved in the system from user's perspective, from this point of view, approach 3 is simple and can fulfill this goal, another is used to prevent disk from full, which needs a more accurate value, considering the whole data on disk, not only the valid container data. I would like to know are we targeting both goals, or just the first goal?
##########
hadoop-hdds/docs/content/design/dn-usedspace-calculation.md:
##########
@@ -0,0 +1,87 @@
+---
+title: Datanode used space calculation
+summary: Describe proposal for optimization in datanode used space calculation.
+date: 2025-04-30
+jira: HDDS-12924
+status: draft
+author: Sumit Agrawal
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+Datanode makes use of DU to identify disk space uses for ozone in the volume.
+DU is a time-consuming operation if run over large occupied disk, like when running over disk path having 10s of TB container data. This will be slow operation.
+
+
+## Approach 1: Run DU over non-ozone path
+
+Challenges:
+
+- Root path is not known for the device mounted on. So this needs extra configuration.
+- Permission problem: du will not count the space if permission is not there on a certain path.
+
+
+Based on the above concern, it's `not feasible` to do du over non-ozone path.
+
+
+## Approach 2: Run DU over meta path only (excluding container dir path)
+
+Ozone space usages includes summation of:
+- Sum of all Container data size as present in memory
+- DU over volume path as current (excluding container path)
+
+`Used space = sum(<all containers data size>, <du over ozone path excluding container data>)`
+
+### Limitation:
+- Space is not counted as ozone space for below cases:
+ - Ozone used size for duplicate containers are not counted (ignored during startup for EC case)
+ - Ozone used size for containers corrupted are not counted, not deleted
+ - Container path meta files like container yaml, exported rocks db’s sst files, are not counted, which might result in few GB of data in a large cluster.
+
+ These spaces will be added up to non-ozone used space, and especially un-accounted containers need to be cleaned up.
+
+In future, the container space (due to duplicacy or corruption) needs to be removed based on error logs.
+
+
+### Impact of inaccuracy of Used Space
+
+1. Used space in reporting (may be few GB as per above)
+2. Adjustment between Ozone available space and Reserved available space
+
+This inaccuracy does not have much impact over solution (as its present in existing) and due to nature of “du” running async and parallel write operation being in progress.
+
+Approach to fits better in this scenario, can be provided as `OptimizedDU` and keeping previous support as well.
+
+## Approach 3 : Used space will be only container data
Review Comment:
Used space has two purposes, one is let user know how many data are saved in the system from user's perspective, from this point of view, approach 3 is simple and can fulfill this goal, another is used to prevent disk from full, which needs a more accurate value, considering the whole data on disk, not only the valid container data. I would like to know are we targeting both goals, or just the first goal in this task?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2069840812
##########
hadoop-hdds/docs/content/design/dn-usedspace-calculation.md:
##########
@@ -0,0 +1,87 @@
+---
+title: Datanode used space calculation
+summary: Describe proposal for optimization in datanode used space calculation.
+date: 2025-04-30
+jira: HDDS-12924
+status: draft
+author: Sumit Agrawal
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+Datanode makes use of DU to identify disk space uses for ozone in the volume.
+DU is a time-consuming operation if run over large occupied disk, like when running over disk path having 10s of TB container data. This will be slow operation.
+
+
+## Approach 1: Run DU over non-ozone path
+
+Challenges:
+
+- Root path is not known for the device mounted on. So this needs extra configuration.
+- Permission problem: du will not count the space if permission is not there on a certain path.
+
+
+Based on the above concern, it's `not feasible` to do du over non-ozone path.
Review Comment:
Thoughts? @errose28
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2153684454
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsageCheckParams.java:
##########
@@ -19,6 +19,7 @@
import static com.google.common.base.Preconditions.checkArgument;
+import com.google.common.base.Supplier;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2076737925
##########
hadoop-hdds/docs/content/design/dn-usedspace-calculation.md:
##########
@@ -0,0 +1,87 @@
+---
+title: Datanode used space calculation
+summary: Describe proposal for optimization in datanode used space calculation.
+date: 2025-04-30
+jira: HDDS-12924
+status: draft
+author: Sumit Agrawal
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+Datanode makes use of DU to identify disk space uses for ozone in the volume.
+DU is a time-consuming operation if run over large occupied disk, like when running over disk path having 10s of TB container data. This will be slow operation.
+
+
+## Approach 1: Run DU over non-ozone path
+
+Challenges:
+
+- Root path is not known for the device mounted on. So this needs extra configuration.
+- Permission problem: du will not count the space if permission is not there on a certain path.
+
+
+Based on the above concern, it's `not feasible` to do du over non-ozone path.
+
+
+## Approach 2: Run DU over meta path only (excluding container dir path)
+
+Ozone space usages includes summation of:
+- Sum of all Container data size as present in memory
+- DU over volume path as current (excluding container path)
+
+`Used space = sum(<all containers data size>, <du over ozone path excluding container data>)`
+
+### Limitation:
+- Space is not counted as ozone space for below cases:
+ - Ozone used size for duplicate containers are not counted (ignored during startup for EC case)
+ - Ozone used size for containers corrupted are not counted, not deleted
+ - Container path meta files like container yaml, exported rocks db’s sst files, are not counted, which might result in few GB of data in a large cluster.
+
+ These spaces will be added up to non-ozone used space, and especially un-accounted containers need to be cleaned up.
+
+In future, the container space (due to duplicacy or corruption) needs to be removed based on error logs.
+
+
+### Impact of inaccuracy of Used Space
+
+1. Used space in reporting (may be few GB as per above)
+2. Adjustment between Ozone available space and Reserved available space
+
+This inaccuracy does not have much impact over solution (as its present in existing) and due to nature of “du” running async and parallel write operation being in progress.
+
+Approach to fits better in this scenario, can be provided as `OptimizedDU` and keeping previous support as well.
+
+## Approach 3 : Used space will be only container data
Review Comment:
So considering accuracy with performance, Approach 2 seems better in that scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2153684696
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/DUOptimized.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.fs;
+
+import java.io.File;
+import java.util.function.Supplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Make use of DU class that uses the unix 'du' program to calculate space usage of metadata excluding container data.
+ * Container data usages is identified from container set.
+ *
+ * @see SpaceUsageSource
+ */
+public class DUOptimized implements SpaceUsageSource {
+ private static final Logger LOG = LoggerFactory.getLogger(DUOptimized.class);
+
+ private final DU metaPathDU;
+ private Supplier<Supplier<Long>> containerUsedSpaceProvider;
+
+ public DUOptimized(File path, Supplier<File> exclusionProvider) {
+ metaPathDU = new DU(exclusionProvider, path);
+ }
+
+ @Override
+ public long getUsedSpace() {
+ long metaPathSize = metaPathDU.getUsedSpace();
+ if (null == containerUsedSpaceProvider) {
+ return metaPathSize;
+ }
+ Supplier<Long> gatherContainerUsages = containerUsedSpaceProvider.get();
+ long containerUsedSpace = gatherContainerUsages.get();
+ LOG.info("Disk du usages {}, container data usages {}", metaPathSize, containerUsedSpace);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2153676183
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/DU.java:
##########
@@ -96,11 +106,21 @@ public long getUsed() {
@Override
public String toString() {
+ if (exclusionProvider != null) {
+ return String.join(" ", getExecString()) + "\n" + value.get() + "\t" + getPath();
+ }
Review Comment:
else part do not use command, but use commandString which is different as it already performed join as old logic.
So this is done in this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2076737221
##########
hadoop-hdds/docs/content/design/dn-usedspace-calculation.md:
##########
@@ -0,0 +1,87 @@
+---
+title: Datanode used space calculation
+summary: Describe proposal for optimization in datanode used space calculation.
+date: 2025-04-30
+jira: HDDS-12924
+status: draft
+author: Sumit Agrawal
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+Datanode makes use of DU to identify disk space uses for ozone in the volume.
+DU is a time-consuming operation if run over large occupied disk, like when running over disk path having 10s of TB container data. This will be slow operation.
+
+
+## Approach 1: Run DU over non-ozone path
+
+Challenges:
+
+- Root path is not known for the device mounted on. So this needs extra configuration.
+- Permission problem: du will not count the space if permission is not there on a certain path.
+
+
+Based on the above concern, it's `not feasible` to do du over non-ozone path.
+
+
+## Approach 2: Run DU over meta path only (excluding container dir path)
+
+Ozone space usages includes summation of:
+- Sum of all Container data size as present in memory
+- DU over volume path as current (excluding container path)
+
+`Used space = sum(<all containers data size>, <du over ozone path excluding container data>)`
+
+### Limitation:
+- Space is not counted as ozone space for below cases:
+ - Ozone used size for duplicate containers are not counted (ignored during startup for EC case)
+ - Ozone used size for containers corrupted are not counted, not deleted
+ - Container path meta files like container yaml, exported rocks db’s sst files, are not counted, which might result in few GB of data in a large cluster.
+
+ These spaces will be added up to non-ozone used space, and especially un-accounted containers need to be cleaned up.
+
+In future, the container space (due to duplicacy or corruption) needs to be removed based on error logs.
+
+
+### Impact of inaccuracy of Used Space
+
+1. Used space in reporting (may be few GB as per above)
+2. Adjustment between Ozone available space and Reserved available space
+
+This inaccuracy does not have much impact over solution (as its present in existing) and due to nature of “du” running async and parallel write operation being in progress.
+
+Approach to fits better in this scenario, can be provided as `OptimizedDU` and keeping previous support as well.
+
+## Approach 3 : Used space will be only container data
Review Comment:
From Approach 2: Goal to report more close ozone usages result with performance improvement by limiting du run over only metadata.
From Approach 3: If little more in-accuracy is acceptable, this is simplest one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 merged PR #8365:
URL: https://github.com/apache/ozone/pull/8365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2149187430
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/MutableVolumeSet.java:
##########
@@ -18,6 +18,7 @@
package org.apache.hadoop.ozone.container.common.volume;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Function;
Review Comment:
updated
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java:
##########
@@ -575,6 +578,15 @@ public ContainerSet getContainerSet() {
return containerSet;
}
+ public Long gatherContainerUsages(HddsVolume storageVolume) {
+ Iterator<Container<?>> itr = containerSet.getContainerIterator(storageVolume);
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on code in PR #8365:
URL: https://github.com/apache/ozone/pull/8365#discussion_r2144527377
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java:
##########
@@ -575,6 +578,15 @@ public ContainerSet getContainerSet() {
return containerSet;
}
+ public Long gatherContainerUsages(HddsVolume storageVolume) {
+ Iterator<Container<?>> itr = containerSet.getContainerIterator(storageVolume);
Review Comment:
`getContainerIterator` sorts the containers first - do we need containers to be in sorted order here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12924. datanode used space calculation optimization [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #8365:
URL: https://github.com/apache/ozone/pull/8365#issuecomment-2853242713
running `du` at a higher frequency on a filesystem populated with thousands of containers and millions of blocks will be expensive. I prefer the approach in this doc where we always keep some min space free by including it in our calculation for disk used and base the disk used on container metatadata. We could optionally run `df` at a lower frequency to ensure the divergence between actual vs real is minimal (taking into account space reserved for other consumers of the volume)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org