You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "symious (via GitHub)" <gi...@apache.org> on 2025/02/19 07:00:58 UTC
[PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
symious opened a new pull request, #7927:
URL: https://github.com/apache/ozone/pull/7927
## What changes were proposed in this pull request?
The current logic for "hdds.datanode.dir.du.reserved" is as follows:
```
private long getRemainingReserved(){
return Math.max(reservedInBytes - getOtherUsed(), 0L);
}
```
which means if OtherUsed() is larger than reservedInBytes, then remainingReserved will be count to 0.
When we set a "hdds.datanode.dir.du.reserved" to 100GB, we actually want the disk to spare 100GB in case of "SPACE not enough exceptions".
But normally servers have a system level block reservation, which is 5%. So for a 10T disk, the system level reserved space is about 500GB, when we set the configuration to 100GB, the "remaningReserved" is calculated as 0, so for capacity and availabilty, reservation is not counted.
In current calculation logic ,in order to reserve a 100GB space, we need to set configuration to "600GB" (500 + 100) or "0.06" (0.05 + 0.01 for percent).
This ticket is to change the logic of the reservation calculation to have a more intuitive aspect for the users, thus no need to take care of the Other usages.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12373
## How was this patch tested?
unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1968754817
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -94,19 +97,8 @@ public void decrementUsedSpace(long reclaimedSpace) {
source.decrementUsedSpace(reclaimedSpace);
}
- /**
- * Get the space used by others except hdds.
- * DU is refreshed periodically and could be not exact,
- * so there could be that DU value > totalUsed when there are deletes.
- * @return other used space
- */
- private static long getOtherUsed(SpaceUsageSource usage) {
- long totalUsed = usage.getCapacity() - usage.getAvailable();
- return Math.max(totalUsed - usage.getUsedSpace(), 0L);
- }
-
- private long getRemainingReserved(SpaceUsageSource usage) {
- return Math.max(reservedInBytes - getOtherUsed(usage), 0L);
+ private static long getUsableCapacity(SpaceUsageSource usage) {
+ return usage.getUsedSpace() + usage.getAvailable();
Review Comment:
Thanks, LGTM. Let's wait for others' feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1965238743
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -68,23 +68,24 @@ SpaceUsageSource realUsage() {
/**
* <pre>
* {@code
- * Calculate available space use method B.
- * |----used----| (avail) |++++++++reserved++++++++|
- * | fsAvail |-------other-------|
- * ->|~~~~|<-
- * remainingReserved
+ * Calculate available space.
+ * |----used----| fsFree |
+ * |----used----| fsAvail |---other---|
+ * |----used----| (avail) |+++reserved+++|---other---|
+ * |<- fsCapacity ->|
+ * |<- usableCapacity ->|
+ * |<- capacity ->|
* }
* </pre>
- * B) avail = fsAvail - Max(reserved - other, 0);
+ * capacity = max(usableCapacity - reserved, 0);
+ * avail = max(fsAvail - reserved, 0);
*/
public SpaceUsageSource getCurrentUsage() {
SpaceUsageSource real = realUsage();
- return reservedInBytes == 0
- ? real
- : new SpaceUsageSource.Fixed(
- Math.max(real.getCapacity() - reservedInBytes, 0),
- Math.max(real.getAvailable() - getRemainingReserved(real), 0),
+ return new SpaceUsageSource.Fixed(
+ Math.max(getUsableCapacity(real) - reservedInBytes, 0),
+ Math.max(real.getAvailable() - reservedInBytes, 0),
real.getUsedSpace());
Review Comment:
Let me change the logic later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1962700531
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -68,23 +68,24 @@ SpaceUsageSource realUsage() {
/**
* <pre>
* {@code
- * Calculate available space use method B.
- * |----used----| (avail) |++++++++reserved++++++++|
- * | fsAvail |-------other-------|
- * ->|~~~~|<-
- * remainingReserved
+ * Calculate available space.
+ * |----used----| fsFree |
+ * |----used----| fsAvail |---other---|
+ * |----used----| (avail) |+++reserved+++|---other---|
+ * |<- fsCapacity ->|
+ * |<- usableCapacity ->|
+ * |<- capacity ->|
* }
* </pre>
- * B) avail = fsAvail - Max(reserved - other, 0);
+ * capacity = max(usableCapacity - reserved, 0);
+ * avail = max(fsAvail - reserved, 0);
*/
public SpaceUsageSource getCurrentUsage() {
SpaceUsageSource real = realUsage();
- return reservedInBytes == 0
- ? real
- : new SpaceUsageSource.Fixed(
- Math.max(real.getCapacity() - reservedInBytes, 0),
- Math.max(real.getAvailable() - getRemainingReserved(real), 0),
+ return new SpaceUsageSource.Fixed(
+ Math.max(getUsableCapacity(real) - reservedInBytes, 0),
+ Math.max(real.getAvailable() - reservedInBytes, 0),
real.getUsedSpace());
Review Comment:
Updated doc for VolumeInfo (https://github.com/apache/ozone/pull/7927/files#diff-997b4865f15926be6c743ff6b64a26af489882ede8ab4e10b7e5b9741b4427a7R59), usableCapacity should be same for (fsCapacity - other) and (used + fsAvail).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1961174086
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -69,13 +69,11 @@ SpaceUsageSource realUsage() {
* <pre>
* {@code
* Calculate available space use method B.
- * |----used----| (avail) |++++++++reserved++++++++|
- * | fsAvail |-------other-------|
- * ->|~~~~|<-
- * remainingReserved
+ * |----used----| fsAvail |---other---|
+ * |----used----| (avail) |+++reserved+++|---other---|
* }
* </pre>
- * B) avail = fsAvail - Max(reserved - other, 0);
+ * B) avail = fsAvail - Max(reserved, 0);
Review Comment:
@symious , it's a good idea. The change looks good to me. Is there any chance that "reserved" is a negative value? If not, then here we can just use "fsAvail - reserved".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on PR #7927:
URL: https://github.com/apache/ozone/pull/7927#issuecomment-2724010537
@symious I've also been looking into this area and I'm planning to review this pull request soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Use File.getUsableSpace() instead of File.getFreeSpace() to calcuate usedSpace in DedicatedDiskSpaceUsage [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r2002925236
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/DedicatedDiskSpaceUsage.java:
##########
@@ -51,7 +51,7 @@ public long getUsedSpace() {
* @return used space
*/
private long calculateUsedSpace() {
- return getCapacity() - getFile().getFreeSpace();
+ return getCapacity() - getFile().getUsableSpace();
Review Comment:
For the record: javadoc on `calculateUsedSpace` still mentions "free space". I guess we can update it in any patches touching this code, no need for a new PR just for this.
Also: `getAvailble()` also returns `getUsableSpace()`, so we could clarify the logic by changing to:
`getCapacity() - getAvailable()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #7927:
URL: https://github.com/apache/ozone/pull/7927#issuecomment-2684395158
@symious , current we have
1. `hdds.datanode.dir.du.reserved` and `hdds.datanode.dir.du.reserved.percent`, similar properties inherited from HDFS, which defines the space left for non-Ozone use.
2. `hdds.datanode.volume.min.free.space` and `hdds.datanode.volume.min.free.space.percent`, introduced in [HDDS-8254](https://issues.apache.org/jira/browse/HDDS-8254), which defines that if the volume free space reaches this value, volume will be treated as full.
I totally agree it's a good idea. Based the current state, I would propose instead of change `hdds.datanode.dir.du.reserved` and `hdds.datanode.dir.du.reserved.percent` concepts, reuse `hdds.datanode.volume.min.free.space` and `hdds.datanode.volume.min.free.space.percent`. Make the SCM pipeline allocation, container allocation, container candidate selection in container balancer, container selection in replication manager, all aware of this min.free.space, so that disk full. can be avoided. What do you think @symious?
cc @sadanand48, @vtutrinov .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Use File.getUsableSpace() instead of File.getFreeSpace() to calcuate usedSpace in DedicatedDiskSpaceUsage [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi merged PR #7927:
URL: https://github.com/apache/ozone/pull/7927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1961483185
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestHddsVolume.java:
##########
@@ -322,7 +322,7 @@ public void testReportUsedSmallerThanActualUsed() throws IOException {
// actual used = 300(new allocates)
Review Comment:
Please update the javadoc of testReportUsedBiggerThanActualUsed() method too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #7927:
URL: https://github.com/apache/ozone/pull/7927#issuecomment-2673908021
One thing I just thought about is, with this new change, if user has ever set the "hdds.datanode.dir.du.reserved" or "hdds.datanode.dir.du.reserved.percent", for those non ozone services, will be too much from reservation point of view after Ozone version upgrade. For example, If Ozone DN is co-deployed with YARN service, which need disk space for shuffle data and all other data. User may set the "hdds.datanode.dir.du.reserved.percent" to 20% or 30%. Then it will be too much for this new calculation logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1961226453
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -69,13 +69,11 @@ SpaceUsageSource realUsage() {
* <pre>
* {@code
* Calculate available space use method B.
- * |----used----| (avail) |++++++++reserved++++++++|
- * | fsAvail |-------other-------|
- * ->|~~~~|<-
- * remainingReserved
+ * |----used----| fsAvail |---other---|
+ * |----used----| (avail) |+++reserved+++|---other---|
* }
* </pre>
- * B) avail = fsAvail - Max(reserved - other, 0);
+ * B) avail = fsAvail - Max(reserved, 0);
Review Comment:
Updated, PTAL.
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestReservedVolumeSpace.java:
##########
@@ -95,15 +95,18 @@ public void testVolumeCapacityAfterReserve() throws Exception {
HDDS_DATANODE_DIR_DU_RESERVED_PERCENT_DEFAULT);
long volumeCapacity = hddsVolume.getCurrentUsage().getCapacity();
+ long volumeAvailable = hddsVolume.getCurrentUsage().getAvailable();
Review Comment:
Updated, PTAL.
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestReservedVolumeSpace.java:
##########
@@ -95,15 +95,18 @@ public void testVolumeCapacityAfterReserve() throws Exception {
HDDS_DATANODE_DIR_DU_RESERVED_PERCENT_DEFAULT);
long volumeCapacity = hddsVolume.getCurrentUsage().getCapacity();
+ long volumeAvailable = hddsVolume.getCurrentUsage().getAvailable();
VolumeUsage usage = hddsVolume.getVolumeInfo().get().getUsageForTesting();
//Gets the actual total capacity
long totalCapacity = usage.realUsage().getCapacity();
+ long totalAvailable = usage.realUsage().getAvailable();
Review Comment:
Updated, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7927:
URL: https://github.com/apache/ozone/pull/7927#issuecomment-2668011005
Thanks @symious for updating the patch, LGTM. Added some other reviewers who worked on space logic previously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1967155166
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -94,19 +97,8 @@ public void decrementUsedSpace(long reclaimedSpace) {
source.decrementUsedSpace(reclaimedSpace);
}
- /**
- * Get the space used by others except hdds.
- * DU is refreshed periodically and could be not exact,
- * so there could be that DU value > totalUsed when there are deletes.
- * @return other used space
- */
- private static long getOtherUsed(SpaceUsageSource usage) {
- long totalUsed = usage.getCapacity() - usage.getAvailable();
- return Math.max(totalUsed - usage.getUsedSpace(), 0L);
- }
-
- private long getRemainingReserved(SpaceUsageSource usage) {
- return Math.max(reservedInBytes - getOtherUsed(usage), 0L);
+ private static long getUsableCapacity(SpaceUsageSource usage) {
+ return usage.getUsedSpace() + usage.getAvailable();
Review Comment:
This method is unused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r2000321051
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/AbstractSpaceUsageSource.java:
##########
@@ -86,6 +86,7 @@ public long getAvailable() {
@Override
public long getCapacity() {
- return file.getTotalSpace();
+ // system reserve: fsFree - fsUsable
+ return file.getTotalSpace() - (file.getFreeSpace() - file.getUsableSpace());
Review Comment:
Through I understand that there is different between getFreeSpace and getUsableSpace, I would suggest not change this, for we would like to know the full capacity of the volume.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7927:
URL: https://github.com/apache/ozone/pull/7927#issuecomment-2674238198
> IMO, the reserved configuration is to make sure there are enough space left for Ozone usage, and for this goal, a 50GB space left would be enough.
According to config doc, `hdds.datanode.volume.min.free.space` (and percent) is for Ozone usage, and `hdds.datanode.dir.du.reserved` (inherited from HDFS) is for non-Ozone usage.
> And Ozone should not worry about the other usages (YARN or system reserve), even Ozone spare a 20% reserve space for YARN, it's not guranteed that YARN will only use 20%.
I agree, maybe we should consider deprecating this setting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1961888832
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestHddsVolume.java:
##########
@@ -322,7 +322,7 @@ public void testReportUsedSmallerThanActualUsed() throws IOException {
// actual used = 300(new allocates)
Review Comment:
Updated, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1961197946
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestReservedVolumeSpace.java:
##########
@@ -95,15 +95,18 @@ public void testVolumeCapacityAfterReserve() throws Exception {
HDDS_DATANODE_DIR_DU_RESERVED_PERCENT_DEFAULT);
long volumeCapacity = hddsVolume.getCurrentUsage().getCapacity();
+ long volumeAvailable = hddsVolume.getCurrentUsage().getAvailable();
Review Comment:
nit: please get usage once:
```java
SpaceUsageSource currentUsage = hddsVolume.getCurrentUsage();
long volumeCapacity = currentUsage.getCapacity();
long volumeAvailable = currentUsage.getAvailable();
```
(needs import, so not adding as github suggestion)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1967145950
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -68,23 +68,24 @@ SpaceUsageSource realUsage() {
/**
* <pre>
* {@code
- * Calculate available space use method B.
- * |----used----| (avail) |++++++++reserved++++++++|
- * | fsAvail |-------other-------|
- * ->|~~~~|<-
- * remainingReserved
+ * Calculate available space.
+ * |----used----| fsFree |
+ * |----used----| fsAvail |---other---|
+ * |----used----| (avail) |+++reserved+++|---other---|
+ * |<- fsCapacity ->|
+ * |<- usableCapacity ->|
+ * |<- capacity ->|
* }
* </pre>
- * B) avail = fsAvail - Max(reserved - other, 0);
+ * capacity = max(usableCapacity - reserved, 0);
+ * avail = max(fsAvail - reserved, 0);
*/
public SpaceUsageSource getCurrentUsage() {
SpaceUsageSource real = realUsage();
- return reservedInBytes == 0
- ? real
- : new SpaceUsageSource.Fixed(
- Math.max(real.getCapacity() - reservedInBytes, 0),
- Math.max(real.getAvailable() - getRemainingReserved(real), 0),
+ return new SpaceUsageSource.Fixed(
+ Math.max(getUsableCapacity(real) - reservedInBytes, 0),
+ Math.max(real.getAvailable() - reservedInBytes, 0),
real.getUsedSpace());
Review Comment:
@adoroszlai The logic is updated, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #7927:
URL: https://github.com/apache/ozone/pull/7927#issuecomment-2720167785
Disscussed with @ChenSammi , it's better to keep the logic of the "du.reserved" config, so we only change log and remove the system reserved from calculation here.
@ChenSammi @adoroszlai @sadanand48 PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #7927:
URL: https://github.com/apache/ozone/pull/7927#issuecomment-2687494221
> > Make the SCM pipeline allocation, container allocation, container candidate selection in container balancer, container selection in replication manager, all aware of this min.free.space, so that disk full can be avoided starting from SCM.
>
> @ChenSammi Are you suggesting to use "min.free.space" instead of "dir.du.reserved"?
Yes, I'm suggesting use `hdds.datanode.dir.du.reserved` and `hdds.datanode.dir.du.reserved.percent` to achieve the same goal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1965202257
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -68,23 +68,24 @@ SpaceUsageSource realUsage() {
/**
* <pre>
* {@code
- * Calculate available space use method B.
- * |----used----| (avail) |++++++++reserved++++++++|
- * | fsAvail |-------other-------|
- * ->|~~~~|<-
- * remainingReserved
+ * Calculate available space.
+ * |----used----| fsFree |
+ * |----used----| fsAvail |---other---|
+ * |----used----| (avail) |+++reserved+++|---other---|
+ * |<- fsCapacity ->|
+ * |<- usableCapacity ->|
+ * |<- capacity ->|
* }
* </pre>
- * B) avail = fsAvail - Max(reserved - other, 0);
+ * capacity = max(usableCapacity - reserved, 0);
+ * avail = max(fsAvail - reserved, 0);
*/
public SpaceUsageSource getCurrentUsage() {
SpaceUsageSource real = realUsage();
- return reservedInBytes == 0
- ? real
- : new SpaceUsageSource.Fixed(
- Math.max(real.getCapacity() - reservedInBytes, 0),
- Math.max(real.getAvailable() - getRemainingReserved(real), 0),
+ return new SpaceUsageSource.Fixed(
+ Math.max(getUsableCapacity(real) - reservedInBytes, 0),
+ Math.max(real.getAvailable() - reservedInBytes, 0),
real.getUsedSpace());
Review Comment:
The thing is we are always losing some info when using "DF", and "DU" is too slow not for PROD usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1961962841
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -68,23 +68,24 @@ SpaceUsageSource realUsage() {
/**
* <pre>
* {@code
- * Calculate available space use method B.
- * |----used----| (avail) |++++++++reserved++++++++|
- * | fsAvail |-------other-------|
- * ->|~~~~|<-
- * remainingReserved
+ * Calculate available space.
+ * |----used----| fsFree |
+ * |----used----| fsAvail |---other---|
+ * |----used----| (avail) |+++reserved+++|---other---|
+ * |<- fsCapacity ->|
+ * |<- usableCapacity ->|
+ * |<- capacity ->|
* }
* </pre>
- * B) avail = fsAvail - Max(reserved - other, 0);
+ * capacity = max(usableCapacity - reserved, 0);
+ * avail = max(fsAvail - reserved, 0);
*/
public SpaceUsageSource getCurrentUsage() {
SpaceUsageSource real = realUsage();
- return reservedInBytes == 0
- ? real
- : new SpaceUsageSource.Fixed(
- Math.max(real.getCapacity() - reservedInBytes, 0),
- Math.max(real.getAvailable() - getRemainingReserved(real), 0),
+ return new SpaceUsageSource.Fixed(
+ Math.max(getUsableCapacity(real) - reservedInBytes, 0),
+ Math.max(real.getAvailable() - reservedInBytes, 0),
real.getUsedSpace());
Review Comment:
1. `capacity` should not be calculated as `used + available`. That's throwing away a piece of information.
2. If we don't reserve any space, real usage should be reported as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #7927:
URL: https://github.com/apache/ozone/pull/7927#issuecomment-2687094814
> Make the SCM pipeline allocation, container allocation, container candidate selection in container balancer, container selection in replication manager, all aware of this min.free.space, so that disk full can be avoided starting from SCM.
@ChenSammi Are you suggesting to use "min.free.space" instead of "dir.du.reserved"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r2000324900
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/AbstractSpaceUsageSource.java:
##########
@@ -86,6 +86,7 @@ public long getAvailable() {
@Override
public long getCapacity() {
- return file.getTotalSpace();
+ // system reserve: fsFree - fsUsable
+ return file.getTotalSpace() - (file.getFreeSpace() - file.getUsableSpace());
Review Comment:
Updated, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1961200236
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestReservedVolumeSpace.java:
##########
@@ -95,15 +95,18 @@ public void testVolumeCapacityAfterReserve() throws Exception {
HDDS_DATANODE_DIR_DU_RESERVED_PERCENT_DEFAULT);
long volumeCapacity = hddsVolume.getCurrentUsage().getCapacity();
+ long volumeAvailable = hddsVolume.getCurrentUsage().getAvailable();
VolumeUsage usage = hddsVolume.getVolumeInfo().get().getUsageForTesting();
//Gets the actual total capacity
long totalCapacity = usage.realUsage().getCapacity();
+ long totalAvailable = usage.realUsage().getAvailable();
Review Comment:
Similarly here, please assign `usage.realUsage()` to variable.
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestReservedVolumeSpace.java:
##########
@@ -95,15 +95,18 @@ public void testVolumeCapacityAfterReserve() throws Exception {
HDDS_DATANODE_DIR_DU_RESERVED_PERCENT_DEFAULT);
long volumeCapacity = hddsVolume.getCurrentUsage().getCapacity();
+ long volumeAvailable = hddsVolume.getCurrentUsage().getAvailable();
Review Comment:
Please get usage once:
```java
SpaceUsageSource currentUsage = hddsVolume.getCurrentUsage();
long volumeCapacity = currentUsage.getCapacity();
long volumeAvailable = currentUsage.getAvailable();
```
(needs import, so not adding as github suggestion)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1965140992
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -68,23 +68,24 @@ SpaceUsageSource realUsage() {
/**
* <pre>
* {@code
- * Calculate available space use method B.
- * |----used----| (avail) |++++++++reserved++++++++|
- * | fsAvail |-------other-------|
- * ->|~~~~|<-
- * remainingReserved
+ * Calculate available space.
+ * |----used----| fsFree |
+ * |----used----| fsAvail |---other---|
+ * |----used----| (avail) |+++reserved+++|---other---|
+ * |<- fsCapacity ->|
+ * |<- usableCapacity ->|
+ * |<- capacity ->|
* }
* </pre>
- * B) avail = fsAvail - Max(reserved - other, 0);
+ * capacity = max(usableCapacity - reserved, 0);
+ * avail = max(fsAvail - reserved, 0);
*/
public SpaceUsageSource getCurrentUsage() {
SpaceUsageSource real = realUsage();
- return reservedInBytes == 0
- ? real
- : new SpaceUsageSource.Fixed(
- Math.max(real.getCapacity() - reservedInBytes, 0),
- Math.max(real.getAvailable() - getRemainingReserved(real), 0),
+ return new SpaceUsageSource.Fixed(
+ Math.max(getUsableCapacity(real) - reservedInBytes, 0),
+ Math.max(real.getAvailable() - reservedInBytes, 0),
real.getUsedSpace());
Review Comment:
`SpaceUsageSource real` reflects capacity from the filesystem (`File.getTotalSpace()`). Now in the value returned from `getCurrentUsage()` that information is lost.
If `sysReserved` cannot be used, ever, and total disk space (`fsCapacity`) is irrelevant, then we should change the way source calculates capacity here:
https://github.com/apache/ozone/blob/e0bd2cc71621087e41cbc0b79f72b6487d271264/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs/AbstractSpaceUsageSource.java#L88-L89
and change `getCurrentUsage()`:
```java
public SpaceUsageSource getCurrentUsage() {
SpaceUsageSource real = realUsage();
return reservedInBytes == 0
? real
: new SpaceUsageSource.Fixed(
Math.max(real.getCapacity() - reservedInBytes, 0),
Math.max(real.getAvailable() - reservedInBytes, 0),
real.getUsedSpace());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #7927:
URL: https://github.com/apache/ozone/pull/7927#issuecomment-2674029430
> User may set the "hdds.datanode.dir.du.reserved.percent" to 20% or 30%. Then it will be too much for this new calculation logic.
IMO, the reserved configuration is to make sure there are enough space left for Ozone usage, and for this goal, a 50GB space left would be enough. And Ozone should not worry about the other usages (YARN or system reserve), even Ozone spare a 20% reserve space for YARN, it's not guranteed that YARN will only use 20%.
It should be more safe for Ozone to always reserve a configured space regradless of other usages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12373. Change calculation logic for volume reserved space [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on code in PR #7927:
URL: https://github.com/apache/ozone/pull/7927#discussion_r1968736507
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeUsage.java:
##########
@@ -94,19 +97,8 @@ public void decrementUsedSpace(long reclaimedSpace) {
source.decrementUsedSpace(reclaimedSpace);
}
- /**
- * Get the space used by others except hdds.
- * DU is refreshed periodically and could be not exact,
- * so there could be that DU value > totalUsed when there are deletes.
- * @return other used space
- */
- private static long getOtherUsed(SpaceUsageSource usage) {
- long totalUsed = usage.getCapacity() - usage.getAvailable();
- return Math.max(totalUsed - usage.getUsedSpace(), 0L);
- }
-
- private long getRemainingReserved(SpaceUsageSource usage) {
- return Math.max(reservedInBytes - getOtherUsed(usage), 0L);
+ private static long getUsableCapacity(SpaceUsageSource usage) {
+ return usage.getUsedSpace() + usage.getAvailable();
Review Comment:
Updated, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org