[jira] [Updated] (HDDS-5316) Fix datanode reserved space calculation

["Mark Gui (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Gui updated HDDS-5316:
---------------------------
    Description: 
Here we found a bad case with datanode reserved space:

Say we have 1TB volume and 500GB reserved space configed, e.g. /data1:500GB.

So we intend to reserve this 500GB to another app, e.g. yarn, then yarn consumed all 500GB.

Then we found that the available space of ozone is 0, which is not intended.

The root cause is the following piece:

 
{code:java}
// VolumeInfo.java
public long getAvailable() {
 return Math.max(usage.getAvailable() - reservedInBytes, 0);
}
...
// VolumeUsage.java
public long getAvailable() {
  long l = source.getCapacity() - source.getUsedSpace();         // 1TB - 500GB = 500GB
  return Math.max(Math.min(l, source.getAvailable()), 0);
}{code}
 

Here usage.getAvailable() will return 500GB, and it should not sub reservedInBytes there, reservedInBytes should only be sub at

`getCapacity()`.

  was:
Here we found a bad case with datanode reserved space:

Say we have 1TB volume and 500GB reserved space configed, e.g. /data1:500GB.

So we intend to reserve this 500GB to another app, e.g. yarn, then yarn consumed all 500GB.

Then we found that the available space of ozone is 0, which is not intended.

The root cause is the following piece:

 
{code:java}
// VolumeInfo.java
public long getAvailable() {
 return Math.max(usage.getAvailable() - reservedInBytes, 0);
}
...
// VolumeUsage.java
public long getAvailable() {
  long l = source.getCapacity() - source.getUsedSpace();
  return Math.max(Math.min(l, source.getAvailable()), 0);
}{code}
 


> Fix datanode reserved space calculation
> ---------------------------------------
>
>                 Key: HDDS-5316
>                 URL: https://issues.apache.org/jira/browse/HDDS-5316
>             Project: Apache Ozone
>          Issue Type: Sub-task
>            Reporter: Mark Gui
>            Assignee: Mark Gui
>            Priority: Major
>
> Here we found a bad case with datanode reserved space:
> Say we have 1TB volume and 500GB reserved space configed, e.g. /data1:500GB.
> So we intend to reserve this 500GB to another app, e.g. yarn, then yarn consumed all 500GB.
> Then we found that the available space of ozone is 0, which is not intended.
> The root cause is the following piece:
>  
> {code:java}
> // VolumeInfo.java
> public long getAvailable() {
>  return Math.max(usage.getAvailable() - reservedInBytes, 0);
> }
> ...
> // VolumeUsage.java
> public long getAvailable() {
>   long l = source.getCapacity() - source.getUsedSpace();         // 1TB - 500GB = 500GB
>   return Math.max(Math.min(l, source.getAvailable()), 0);
> }{code}
>  
> Here usage.getAvailable() will return 500GB, and it should not sub reservedInBytes there, reservedInBytes should only be sub at
> `getCapacity()`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org