You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/12/18 05:54:00 UTC
[GitHub] [ozone] captainzmc opened a new pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
captainzmc opened a new pull request #1723:
URL: https://github.com/apache/ozone/pull/1723
## What changes were proposed in this pull request?
If volume's quota is enabled then bucket's quota cannot be cleared. We need to prompt the user to clear volume quota first
[For more details.](https://github.com/apache/ozone/pull/1677#issuecomment-745026394)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4588
## How was this patch tested?
ut added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] linyiqun edited a comment on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
linyiqun edited a comment on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-750347739
> > I am thinking for this case, it's okay to just clear bucket quota no matter if the volume quota was set? Since the volume quota check will still make the sense, when we clear one bucket quota.
>
> There is a very [detailed discussion](https://github.com/apache/ozone/pull/1677#issuecomment-745026394) of this.
@captainzmc , although we removed the volume usedBytes because of the quota update issue. But instead of, we sum the buckets used bytes when bucket is created. This logic can still make the sense for when volume quota is set and the bucket quota is cleared, [OMBucketCreateRequest.java#L331](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java#L331)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-749551033
> I am thinking for this case, it's okay to just clear bucket quota no matter if the volume quota was set? Since the volume quota check will still make the sense, when we clear one bucket quota.
There is a very [detailed discussion](https://github.com/apache/ozone/pull/1677#issuecomment-745026394) of this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] linyiqun edited a comment on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
linyiqun edited a comment on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-750347739
> > I am thinking for this case, it's okay to just clear bucket quota no matter if the volume quota was set? Since the volume quota check will still make the sense, when we clear one bucket quota.
>
> There is a very [detailed discussion](https://github.com/apache/ozone/pull/1677#issuecomment-745026394) of this.
@captainzmc , although we removed the volume usedBytes because of the quota update issue. Can we also try to sum the buckets used bytes when bucket is created that will be used for the volume quota check? As I see we already do the similar check for the quota valid check, [OMBucketCreateRequest.java#L331](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java#L331)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] amaliujia commented on a change in pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#discussion_r546189920
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketSetPropertyRequest.java
##########
@@ -228,6 +228,13 @@ public boolean checkQuotaBytesValid(OMMetadataManager metadataManager,
throws IOException {
long quotaInBytes = omBucketArgs.getQuotaInBytes();
+ if (quotaInBytes == OzoneConsts.QUOTA_RESET &&
+ omVolumeArgs.getQuotaInBytes() != OzoneConsts.QUOTA_RESET) {
+ throw new OMException("Before clear bucket quotaInBytes," +
Review comment:
Nit: will it better to use more descriptive names than variable names. For example, replace `quotaInBytes` by space quota?
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketSetPropertyRequest.java
##########
@@ -259,9 +266,16 @@ public boolean checkQuotaBytesValid(OMMetadataManager metadataManager,
}
public boolean checkQuotaNamespaceValid(OmVolumeArgs omVolumeArgs,
- OmBucketArgs omBucketArgs) {
+ OmBucketArgs omBucketArgs) throws IOException{
long quotaInNamespace = omBucketArgs.getQuotaInNamespace();
+ if (quotaInNamespace == OzoneConsts.QUOTA_RESET &&
+ omVolumeArgs.getQuotaInNamespace() != OzoneConsts.QUOTA_RESET) {
+ throw new OMException("Before clear bucket " +
+ "quotaInNamespace, volume quotaInNamespace need to be clear first.",
Review comment:
Nit: same, maybe replace `quotaInNamespace` by namespace quota?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] linyiqun commented on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
linyiqun commented on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-750727730
If there is the performance consideration for this, I am okay for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-751598901
Removed the namespace restriction. cc @amaliujia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] amaliujia commented on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
amaliujia commented on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-751626695
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc edited a comment on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc edited a comment on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-750715216
> @captainzmc , although we removed the volume usedBytes because of the quota update issue. Can we also try to sum the buckets used bytes when bucket is created that will be used for the volume quota check? As I see we already do the similar check for the quota valid check, [OMBucketCreateRequest.java#L331](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java#L331)
Thanks for @linyiqun‘ advice.
We've considered this option before, but it can cause performance problems.
Let's assume that we have 1000 buckets under volume.
First set quota, which is not a very frequent action in a cluster, and unlike the write operation, has a low latency requirement. So we calculated that all bucket quota is acceptable.
However, each write operation calculating the usedBytes of all buckets for volume spaceQuotaChecker may affect write performance. The more buckets there are, the more performance is affected per write. This maybe unacceptable to the users.
Also if we do this, we need to lock the volume when we count all the usedBytes of our buckets, because we don't want the usedBytes of our buckets to change back and forth when we calculate the total usedBytes, which will also affect the performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc merged pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc merged pull request #1723:
URL: https://github.com/apache/ozone/pull/1723
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ayushtkn commented on a change in pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
ayushtkn commented on a change in pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#discussion_r547082444
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
##########
@@ -311,6 +311,23 @@ public void testSetAndClrQuota() throws IOException {
ozoneBucket.getQuotaInBytes());
Assert.assertEquals(1000L, ozoneBucket.getQuotaInNamespace());
+ try {
+ ozoneBucket.clearSpaceQuota();
+ Assert.fail("clear quota should be failed");
+ } catch (IOException ex) {
+ GenericTestUtils.assertExceptionContains("Before clear bucket " +
+ "quotaInBytes, volume quotaInBytes need to be clear first.", ex);
+ }
Review comment:
Can try something like this -
```
LambdaTestUtils.intercept(IOException.class, "Before clear bucket "
+ "quotaInBytes, volume quotaInBytes need to be clear first.",
() -> ozoneBucket.clearSpaceQuota());
```
Shall save lines and looks better
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
##########
@@ -311,6 +311,23 @@ public void testSetAndClrQuota() throws IOException {
ozoneBucket.getQuotaInBytes());
Assert.assertEquals(1000L, ozoneBucket.getQuotaInNamespace());
+ try {
+ ozoneBucket.clearSpaceQuota();
+ Assert.fail("clear quota should be failed");
+ } catch (IOException ex) {
+ GenericTestUtils.assertExceptionContains("Before clear bucket " +
+ "quotaInBytes, volume quotaInBytes need to be clear first.", ex);
+ }
+
+ try {
+ ozoneBucket.clearCountQuota();
+ Assert.fail("clear quota should be failed");
+ } catch (IOException ex) {
+ GenericTestUtils.assertExceptionContains("Before clear bucket " +
+ "quotaInNamespace, volume quotaInNamespace need to be clear first.",
+ ex);
+ }
Review comment:
Similarly here -
```
LambdaTestUtils.intercept(IOException.class, "Before clear bucket "
+ "quotaInNamespace, volume quotaInNamespace need to be clear first.",
() -> ozoneBucket.clearCountQuota());
```
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketSetPropertyRequest.java
##########
@@ -259,9 +266,16 @@ public boolean checkQuotaBytesValid(OMMetadataManager metadataManager,
}
public boolean checkQuotaNamespaceValid(OmVolumeArgs omVolumeArgs,
- OmBucketArgs omBucketArgs) {
+ OmBucketArgs omBucketArgs) throws IOException{
long quotaInNamespace = omBucketArgs.getQuotaInNamespace();
+ if (quotaInNamespace == OzoneConsts.QUOTA_RESET &&
+ omVolumeArgs.getQuotaInNamespace() != OzoneConsts.QUOTA_RESET) {
+ throw new OMException("Before clear bucket " +
+ "quotaInNamespace, volume quotaInNamespace need to be clear first.",
Review comment:
Seems some grammatical error in the message, Can you check and reframe if possible,
May be something like a warning, Can not clear bucket namespace quota because volume namespace quota is not unset. Or may be something better....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-749551201
Fixed review issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-749548541
Fixed review issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc removed a comment on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc removed a comment on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-749548541
Fixed review issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] linyiqun commented on a change in pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
linyiqun commented on a change in pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#discussion_r546224631
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketSetPropertyRequest.java
##########
@@ -259,9 +266,16 @@ public boolean checkQuotaBytesValid(OMMetadataManager metadataManager,
}
public boolean checkQuotaNamespaceValid(OmVolumeArgs omVolumeArgs,
- OmBucketArgs omBucketArgs) {
+ OmBucketArgs omBucketArgs) throws IOException{
long quotaInNamespace = omBucketArgs.getQuotaInNamespace();
+ if (quotaInNamespace == OzoneConsts.QUOTA_RESET &&
+ omVolumeArgs.getQuotaInNamespace() != OzoneConsts.QUOTA_RESET) {
+ throw new OMException("Before clear bucket " +
+ "quotaInNamespace, volume quotaInNamespace need to be clear first.",
Review comment:
Agreed with @amaliujia 's suggestion. I also prefer to do below rename changing for more readable:
quotaInNamespace -> namespaceQuota
quotaInNamespace -> spaceQuota
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-752018305
Merged this patch, Thanks for @linyiqun @amaliujia @ayushtkn for the review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] linyiqun edited a comment on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
linyiqun edited a comment on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-750727730
>However, each write operation calculating the usedBytes of all buckets for volume quota checker may affect write performance. The more buckets there are, the more performance is affected per write. This maybe unacceptable to the users.
Also if we do this, we need to lock the volume when we count all the usedBytes of our buckets, because we don't want the usedBytes of our buckets to change back and forth when we calculate the total usedBytes, which will also affect the performance.
If there is the performance consideration for this, I am okay for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] linyiqun edited a comment on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
linyiqun edited a comment on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-750347739
> > I am thinking for this case, it's okay to just clear bucket quota no matter if the volume quota was set? Since the volume quota check will still make the sense, when we clear one bucket quota.
>
> There is a very [detailed discussion](https://github.com/apache/ozone/pull/1677#issuecomment-745026394) of this.
@captainzmc , although we removed the volume usedBytes because of the quota update issue. But instead of, we sum the buckets used bytes when bucket is created. This logic can still make the sense, [OMBucketCreateRequest.java#L331](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java#L331)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc edited a comment on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc edited a comment on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-750715216
> @captainzmc , although we removed the volume usedBytes because of the quota update issue. Can we also try to sum the buckets used bytes when bucket is created that will be used for the volume quota check? As I see we already do the similar check for the quota valid check, [OMBucketCreateRequest.java#L331](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java#L331)
Thanks for @linyiqun‘ advice.
We've considered this option before, but it can cause performance problems.
Let's assume that we have 1000 buckets under volume.
First set quota, which is not a very frequent action in a cluster, and unlike the write operation, has a low latency requirement. So we calculated that all bucket quota is acceptable.
However, each write operation calculating the usedBytes of all buckets for volume quota checker may affect write performance. The more buckets there are, the more performance is affected per write. This maybe unacceptable to the users.
Also if we do this, we need to lock the volume when we count all the usedBytes of our buckets, because we don't want the usedBytes of our buckets to change back and forth when we calculate the total usedBytes, which will also affect the performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-750715216
> @captainzmc , although we removed the volume usedBytes because of the quota update issue. Can we also try to sum the buckets used bytes when bucket is created that will be used for the volume quota check? As I see we already do the similar check for the quota valid check, [OMBucketCreateRequest.java#L331](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java#L331)
Thanks for @linyiqun‘ advice.
We've considered this option before, but it can cause performance problems.
Let's assume that we have 1000 buckets under volume.
First set quota, which is not a very frequent action in a cluster, and unlike the write operation, has a low latency requirement. So we calculated that all bucket quota is acceptable.
However, each write operation calculating the usedBytes of all buckets for volume spaceQuotaChecker may affect write performance. The more buckets there are, the more performance is affected per write. This maybe unacceptable to the users.
And if we do this, we need to lock the volume when we count all the usedBytes of our buckets, because we don't want the usedBytes of our buckets to change back and forth when we calculate the total usedBytes, which will also affect the performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc edited a comment on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
captainzmc edited a comment on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-751598901
Removed the namespace restriction, as discussed in #1736. cc @amaliujia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] linyiqun commented on pull request #1723: HDDS-4588. If volume's quota is enabled then bucket's quota cannot be cleared
Posted by GitBox <gi...@apache.org>.
linyiqun commented on pull request #1723:
URL: https://github.com/apache/ozone/pull/1723#issuecomment-750347739
> > I am thinking for this case, it's okay to just clear bucket quota no matter if the volume quota was set? Since the volume quota check will still make the sense, when we clear one bucket quota.
>
> There is a very [detailed discussion](https://github.com/apache/ozone/pull/1677#issuecomment-745026394) of this.
@captainzmc , although we removed the usedBytes because of the quota update issue. But instead of, we sum the buckets used bytes when bucket is created. This logic can still make the sense, [OMBucketCreateRequest.java#L331](https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java#L331)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org