You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "mohan3d (via GitHub)" <gi...@apache.org> on 2023/01/31 13:49:11 UTC
[GitHub] [ozone] mohan3d opened a new pull request, #4226: add getStatus
mohan3d opened a new pull request, #4226:
URL: https://github.com/apache/ozone/pull/4226
## What changes were proposed in this pull request?
Report correct quota, used bytes and available quota.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7858
## How was this patch tested?
Unit tests/Manual tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1105699664
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
@mohan3d IMO volume.getUsedBytes() handling is not required as this is performance extensive operation where it will access all buckets in volume which is I/O. And documents / design enforces the restriction only for performance purpose.
Additionally, quota feature was not stable earlier, so handling is not required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1093529231
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Suppose a volume vol1 has 2 buckets buck1 and buck2, if vol1 quota is 1GB and buck1 quota is 1GB, if I am not wrong user won't be able to write to buck2 as buck1 has reserved all of vol1's space. In that case Total size for buck2= 0 and not 1GB I guess.
cc @sumitagrawl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1096483747
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Thanks again @sadanand48 , this way the calculations is reduced to:
if bucket has quota set: return (bucket.quota, bucket.usedBytes, quota - usedBytes)
else if (volume has quota set): return (volume.quota, volume.usedBytes, quota - usedBytes)
else: return (Long.MAX_VALUE, bucket.usedBytes, Long.MAX_VALUE - usedBytes)
Please look into last case where there is no quota set for both volume and bucket. does this make sense or return default values (Long.MAX_VALUE, 0, Long.MAX_VALUE).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1094343966
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Yes, I think total available size for a given bucket on which there is no quota set =
**volumequota - ( for each bucket in volume : { sum of bucketquota if enabled else usedbytes})**
although I don't think that there is an api for this in OM ( to get available size for a bucket.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1094386714
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Thanks @sadanand48 for responding.
> although I don't think that there is an api for this in OM ( to get available size for a bucket.)
If you mean the given quota it can be obtained from OzoneBucket object (bucket.getQuotaInBytes())
or usedBytes also can be obtained in the same way (bucket.getUsedBytes())
Regarding your comment on the way to calculate if there is no quota set. I think this is true if there is no quota set in any bucket in the volume if I understand well.
Let me summarize all cases we might have:
1. volume and buckets which all of them have quota set
2. volume and buckets (volume has quota, all buckets have no quota)
3. volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
> volume and buckets which all of them have quota set
This one is straightforward volume quota = sum (bucket.quota for each bucket in the volume). hence the calculations are like this; FsStatus={capacity=bucket.getQuotaInBytes, used=bucket.getUsedBytes(), remaining=capacity-used}
> volume and buckets (volume has quota, all buckets have no quota)
Since there is no quota on any bucket the calculations can be like this; FsStatus={capacity=volume.getQuotaInBytes, used=<calculate total bytes used in all volume's buckets> remaining=capacity-used}
Not sure about the way to calculate it. Maybe there is a need to implement an API in OM for it or simple solution like mentioned earlier.
```java
Iterator<? extends OzoneBucket> it = vol.listBuckets(null);
long volUsedBytes = 0L;
while (bucketIterator.hasNext()) {
volUsedBytes += bucketIterator.next().getUsedBytes();
}
```
> volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
Last one with mixed buckets is a little bit complicated, for example
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=1GB)
df /vol1/buck1 (should return 1GB as the quota given to the bucket is 3GB).
Another example:
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=2.5GB)
df /vol1/buck1 (should return 0.5GB which is the min of (available quota of the volume and quota-used bytes of the bucket) = min(5GB - 2.5GB, 3GB-2GB) = min(0.5GB, 1GB) = 0.5GB
df a bucket with no quota set should follow the earlier calculations (min of (available quota of the volume and quota-used bytes of the bucket))
Keep in mind these 2 and 3 cases will only happen in earlier version of Ozone. should we drop this and keep the calculations simplified? should we care for earlier versions (where you can create a bucket without quota when volume has quota)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1094386714
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Thanks @sadanand48 for responding.
> although I don't think that there is an api for this in OM ( to get available size for a bucket.)
If you mean the given quota it can be obtained from OzoneBucket object (bucket.getQuotaInBytes())
or usedBytes also can be obtained in the same way (bucket.getUsedBytes())
Regarding your comment on the way to calculate if there is no quota set. I think this is true if there is no quota set in any bucket in the volume if I understand well.
Let me summarize all cases we might have:
1. volume and buckets which all of them have quota set
2. volume and buckets (volume has quota, all buckets have no quota)
3. volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
> volume and buckets which all of them have quota set
This one is straightforward volume quota = sum (bucket.quota for each bucket in the volume). hence the calculations are like this; FsStatus={capacity=bucket.getQuotaInBytes, used=bucket.getUsedBytes(), remaining=capacity-used}
> volume and buckets (volume has quota, all buckets have no quota)
Since there is no quota on any bucket the calculations can be like this; FsStatus={capacity=volume.getQuotaInBytes, used=<calculate total bytes used in all volume's buckets> remaining=capacity-used}
Not sure about the way to calculate it. Maybe there is a need to implement an API in OM for it or simple solution like mentioned earlier.
```java
Iterator<? extends OzoneBucket> bucketIterator = vol.listBuckets(null);
long volUsedBytes = 0L;
while (bucketIterator.hasNext()) {
volUsedBytes += bucketIterator.next().getUsedBytes();
}
```
> volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
Last one with mixed buckets is a little bit complicated, for example
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=1GB)
df /vol1/buck1 (should return 1GB as the quota given to the bucket is 3GB).
Another example:
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=2.5GB)
df /vol1/buck1 (should return 0.5GB which is the min of (available quota of the volume and quota-used bytes of the bucket) = min(5GB - 2.5GB, 3GB-2GB) = min(0.5GB, 1GB) = 0.5GB
df a bucket with no quota set should follow the earlier calculations (min of (available quota of the volume and quota-used bytes of the bucket))
**Keep in mind these 2 and 3 cases will only happen in earlier version of Ozone**. should we drop this and keep the calculations simplified? should we care for earlier versions (where you can create a bucket without quota when volume has quota)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1094386714
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Thanks @sadanand48 for responding.
> although I don't think that there is an api for this in OM ( to get available size for a bucket.)
If you mean the given quota it can be obtained from OzoneBucket object (bucket.getQuotaInBytes())
or usedBytes also can be obtained in the same way (bucket.getUsedBytes())
Regarding your comment on the way to calculate if there is no quota set. I think this is true if there is no quota set in any bucket in the volume if I understand well.
Let me summarize all cases we might have:
1. volume and buckets which all of them have quota set
2. volume and buckets (volume has quota, all buckets have no quota)
3. volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
> volume and buckets which all of them have quota set
This one is straightforward volume quota = sum (bucket.quota for each bucket in the volume). hence the calculations are like this; FsStatus={capacity=bucket.getQuotaInBytes, used=bucket.getUsedBytes(), remaining=capacity-used}
> volume and buckets (volume has quota, all buckets have no quota)
Since there is no quota on any bucket the calculations can be like this; FsStatus={capacity=volume.getQuotaInBytes, used=<calculate total bytes used in all volume's buckets> remaining=capacity-used}
Not sure about the way to calculate it. Maybe there is a need to implement an API in OM for it or simple solution like mentioned earlier.
```java
Iterator<? extends OzoneBucket> it = vol.listBuckets(null);
long volUsedBytes = 0L;
while (bucketIterator.hasNext()) {
volUsedBytes += bucketIterator.next().getUsedBytes();
}
```
> volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
Last one with mixed buckets is a little bit complicated, for example
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=1GB)
df /vol1/buck1 (should return 1GB as the quota given to the bucket is 3GB).
Another example:
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=2.5GB)
df /vol1/buck1 (should return 0.5GB which is the min of (available quota of the volume and quota-used bytes of the bucket) = min(5GB - 2.5GB, 3GB-2GB) = min(0.5GB, 1GB) = 0.5GB
df a bucket with no quota set should follow the earlier calculations (min of (available quota of the volume and quota-used bytes of the bucket))
**Keep in mind these 2 and 3 cases will only happen in earlier version of Ozone**. should we drop this and keep the calculations simplified? should we care for earlier versions (where you can create a bucket without quota when volume has quota)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1093946682
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Yeah this can happen in earlier versions (before [HDDS-7751](https://issues.apache.org/jira/browse/HDDS-7751)). Now it cannot happen anymore because a user will not be able to create a bucket with no quota if volume has quota.
Also total quota of buckets in a volume cannot exceed the volume's quota.
In your example vol1 has 1GB quota, and buck1 has 1GB quota. then the user will not be able to create buck2 with 1GB of quota.
It makes sense in the first case (you mentioned, if there is no quota set for bucket) to report volumes total given quota and all bytes used in volume **sum (bucket.usedBytes for each bucket in volume)** something like that.
But I fear that there might be a case where volume has quota, and some of the buckets in that volume has quota but not all of them.
For example vol1 has 3GB of quota, /vol1/buck1 has 2GB quota. /vol1/buck2 has no quota in this case available quota should be **min(volume quota - sum (bucket.usedBytes for each bucket in volume), bucket quota (if there is) - usedBytes in the bucket**).
What is your suggestion to deal with earlier version case (before [HDDS-7751](https://issues.apache.org/jira/browse/HDDS-7751))
@sadanand48
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1096483747
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Thanks again @sadanand48 , this way the calculations is reduced to:
if bucket has quota set: return (bucket.quota, bucket.usedBytes, quota - usedBytes)
else if (volume has quota set): return (volume.quota, volume.usedBytes, quota - usedBytes)
else: return (Long.MAX_VALUE, bucket.usedBytes, Long.MAX_VALUE - usedBytes)
Please take a look into last case where there is no quota set for both volume and bucket. does this make sense or return default values (Long.MAX_VALUE, 0, Long.MAX_VALUE).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1095716508
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Thanks @mohan3d for explaining the scenarios. HDDS-7751 was an OM server side fix but this PR makes a change on the client ( by introducing new API getStatus). It can happen that new client (with getStatus implemented) talks to old OM server (without HDDS-7751) , so I think we need to handle for cases 2 & 3. The calculations look good for all the cases you mentioned and I think it is simpler to use vol.listBuckets like in the code snippet
> Another example:
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=2.5GB)
Although I wonder if this case is possible, If volume quota is set to 5GB , and buck1 has 3GB quota , then in the other bucket you cannot write more than (5-3=2GB) right, so usedBytes cannot be more than 2GB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on PR #4226:
URL: https://github.com/apache/ozone/pull/4226#issuecomment-1429643594
Thanks @sumitagrawl for the review, you mean handling the volume case is not required or implementing getStatus is not useful at the moment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d closed pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d closed pull request #4226: HDDS-7858. implement getStatus
URL: https://github.com/apache/ozone/pull/4226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1093946682
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Yeah this can happen in earlier versions (before [HDDS-7751](https://issues.apache.org/jira/browse/HDDS-7751)). Now it cannot happen anymore because a user will not be able to create a bucket with no quota if volume has quota.
Also total quota of buckets in a volume cannot exceed the volume's quota.
In your example vol1 has 1GB quota, and buck1 has 1GB quota. then the user will not be able to create buck2 with 1GB of quota.
It makes sense in the first case (you mentioned, if there is no quota set for bucket) to report volumes total given quota and all bytes used in volume **sum (bucket.usedBytes for each bucket in volume)** something like that.
But I fear that there might be a case where volume has quota, and some of the buckets in that volume has quota but not all of them.
For example vol1 has 3GB of quota, /vol1/buck1 has 2GB quota. /vol1/buck2 has no quota in this case available quota should be **min(volume quota - sum (bucket.usedBytes for each bucket in volume), bucket quota (if there is) - usedBytes in the bucket**).
What is your suggestion to deal with earlier version case (before [HDDS-7751](https://issues.apache.org/jira/browse/HDDS-7751))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1094386714
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Thanks @sadanand48 for responding.
> although I don't think that there is an api for this in OM ( to get available size for a bucket.)
If you mean the given quota it can be obtained from OzoneBucket object (bucket.getQuotaInBytes())
or usedBytes also can be obtained in the same way (bucket.getUsedBytes())
Regarding your comment on the way to calculate if there is no quota set. I think this is true if there is no quota set in any bucket in the volume if I understand well.
Let me summarize all cases we might have:
1. volume and buckets which all of them have quota set
2. volume and buckets (volume has quota, all buckets have no quota)
3. volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
> volume and buckets which all of them have quota set
This one is straightforward volume quota = sum (bucket.quota for each bucket in the volume). hence the calculations are like this; FsStatus={capacity=bucket.getQuotaInBytes, used=bucket.getUsedBytes(), remaining=capacity-used}
> volume and buckets (volume has quota, all buckets have no quota)
Since there is no quota on any bucket the calculations can be like this; FsStatus={capacity=volume.getQuotaInBytes, used=<calculate total bytes used in all volume's buckets> remaining=capacity-used}
Not sure about the way to calculate it. Maybe there is a need to implement an API in OM for it or simple solution like mentioned earlier.
```java
Iterator<? extends OzoneBucket> it = vol.listBuckets(null);
long volUsedBytes = 0L;
while (bucketIterator.hasNext()) {
volUsedBytes += bucketIterator.next().getUsedBytes();
}
```
> volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
Last one with mixed buckets is a little bit complicated, for example
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=1GB)
df /vol1/buck1 (should return 1GB as the quota given to the bucket is 3GB).
Another example:
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=2.5GB)
df /vol1/buck1 (should return 0.5GB which is the min of (available quota of the volume and quota-used bytes of the bucket) = min(5GB - 2.5GB, 3GB-2GB) = min(0.5GB, 1GB) = 0.5GB
df a bucket with no quota set should follow the earlier calculations (min of (available quota of the volume and quota-used bytes of the bucket))
**Keep in mind these 2 and 3 cases will only happen in earlier version of Ozone**. should we drop this and keep the calculations simplified? should we care for earlier versions (where you can create a bucket without quota when volume has quota)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] mohan3d commented on a diff in pull request #4226: HDDS-7858. implement getStatus
Posted by "mohan3d (via GitHub)" <gi...@apache.org>.
mohan3d commented on code in PR #4226:
URL: https://github.com/apache/ozone/pull/4226#discussion_r1094386714
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1010,6 +1013,24 @@ listingPageSize, uri, workingDir, getUsername())
return statusList;
}
+ @Override
+ public FsStatus getStatus(Path p) throws IOException {
+ BasicOzoneClientAdapterImpl adapterImpl =
+ (BasicOzoneClientAdapterImpl) adapter;
+ OzoneBucket bucket = adapterImpl.getBucket();
+ long usedBytes = bucket.getUsedBytes();
+ long quota = Long.MAX_VALUE;
+ if (bucket.getQuotaInBytes() > -1) {
+ quota = bucket.getQuotaInBytes();
+ } else {
+ OzoneVolume volume = adapterImpl.getVolume();
+ if (volume.getQuotaInBytes() > -1) {
+ quota = volume.getQuotaInBytes();
Review Comment:
Thanks @sadanand48 for responding.
> although I don't think that there is an api for this in OM ( to get available size for a bucket.)
If you mean the given quota it can be obtained from OzoneBucket object (bucket.getQuotaInBytes())
or usedBytes also can be obtained in the same way (bucket.getUsedBytes())
Regarding your comment on the way to calculate if there is no quota set. I think this is true if there is no quota set in any bucket in the volume if I understand well.
Let me summarize all cases we might have:
1. volume and buckets which all of them have quota set
2. volume and buckets (volume has quota, all buckets have no quota)
3. volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
> volume and buckets which all of them have quota set
This one is straightforward volume quota = sum (bucket.quota for each bucket in the volume). hence the calculations are like this; FsStatus={capacity=bucket.getQuotaInBytes, used=bucket.getUsedBytes(), remaining=capacity-used}
> volume and buckets (volume has quota, all buckets have no quota)
Since there is no quota on any bucket the calculations can be like this; FsStatus={capacity=volume.getQuotaInBytes, used=<calculate total bytes used in all volume's buckets> remaining=capacity-used}
Not sure about the way to calculate it. Maybe there is a need to implement an API in OM for it or simple solution like mentioned earlier.
```java
Iterator<? extends OzoneBucket> it = vol.listBuckets(null);
long volUsedBytes = 0L;
while (bucketIterator.hasNext()) {
volUsedBytes += bucketIterator.next().getUsedBytes();
}
```
> volume and buckets (volume has quota, mixed buckets some of them have quota and the rest no quota)
Last one with mixed buckets is a little bit complicated, for example
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=1GB)
df /vol1/buck1 (should return 1GB as the quota given to the bucket is 3GB).
Another example:
/vol1 (quota=5GB)
/vol1/buck1 (quota=3GB, usedBytes=2GB)
/vol1/buck2 (quota=-1, usedBytes=2.5GB)
df /vol1/buck1 (should return 0.5GB which is the min of (available quota of the volume and quota-used bytes of the bucket) = min(5GB - 2.5GB, 3GB-2GB) = min(0.5GB, 1GB) = 0.5GB
df a bucket with no quota set should follow the earlier calculations (min of (available quota of the volume and quota-used bytes of the bucket))
**Keep in mind these 2 and 3 cases will only happen in earlier version of Ozone**. should we drop this and keep the calculations simplified? should we care for earlier versions (where you can create a bucket without quota when volume has quota)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org