You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/09/17 02:40:01 UTC
[GitHub] [hadoop-ozone] captainzmc opened a new pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
captainzmc opened a new pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431
## What changes were proposed in this pull request?
In addition, the current Quota setting does not take effect. HDDS-541 gives all the work needed to perfect Quota.
This PR is a subtask of HDDS-541.
First, we increase usedBytes of Bucket. Later, we will judge whether the Bucket can be written based on this when we write the key.
Volume has implemented increase usedBytes when write, and this is based on HDDS-4053.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4254
## How was this patch tested?
added UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi merged pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
ChenSammi merged pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431#issuecomment-697133328
Thanks @ChenSammi for the review. Review issues has been fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431#issuecomment-697328180
LGTM, +1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614881
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java
##########
@@ -99,6 +103,11 @@ protected void addToDBBatch(OMMetadataManager omMetadataManager,
omMetadataManager.getVolumeTable().putWithBatch(batchOperation,
omMetadataManager.getVolumeKey(omVolumeArgs.getVolume()),
omVolumeArgs);
+ // update bucket usedBytes.
+ omMetadataManager.getBucketTable().putWithBatch(batchOperation,
+ omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
+ omBucketInfo.getBucketName()),
+ omBucketInfo);
Review comment:
the same. please double check the rest files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc closed pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
captainzmc closed pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614881
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java
##########
@@ -99,6 +103,11 @@ protected void addToDBBatch(OMMetadataManager omMetadataManager,
omMetadataManager.getVolumeTable().putWithBatch(batchOperation,
omMetadataManager.getVolumeKey(omVolumeArgs.getVolume()),
omVolumeArgs);
+ // update bucket usedBytes.
+ omMetadataManager.getBucketTable().putWithBatch(batchOperation,
+ omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
+ omBucketInfo.getBucketName()),
+ omBucketInfo);
Review comment:
the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614631
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCommitResponse.java
##########
@@ -77,6 +80,11 @@ public void addToDBBatch(OMMetadataManager omMetadataManager,
omMetadataManager.getVolumeTable().putWithBatch(batchOperation,
omMetadataManager.getVolumeKey(omVolumeArgs.getVolume()),
omVolumeArgs);
+ // update bucket usedBytes.
+ omMetadataManager.getBucketTable().putWithBatch(batchOperation,
+ omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
+ omBucketInfo.getBucketName()),
+ omBucketInfo);
Review comment:
same as above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431#issuecomment-693789583
Here are the freon test results:
Using 100 threads to write data to different buckets under the same volume, adding quota Usage has little impact on performance. (LongAdder disperses the concurrency of a single cell to each cell, improving concurrency efficiency compared to atomicLong). In my test, I used three virtual machines, each with a key size of 10K.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614416
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMAllocateBlockResponse.java
##########
@@ -72,5 +76,10 @@ public void addToDBBatch(OMMetadataManager omMetadataManager,
omMetadataManager.getVolumeTable().putWithBatch(batchOperation,
omMetadataManager.getVolumeKey(omVolumeArgs.getVolume()),
omVolumeArgs);
+ // update bucket usedBytes.
+ omMetadataManager.getBucketTable().putWithBatch(batchOperation,
+ omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
+ omBucketInfo.getBucketName()),
+ omBucketInfo);
Review comment:
the last two lines can be merged into one line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614416
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMAllocateBlockResponse.java
##########
@@ -72,5 +76,10 @@ public void addToDBBatch(OMMetadataManager omMetadataManager,
omMetadataManager.getVolumeTable().putWithBatch(batchOperation,
omMetadataManager.getVolumeKey(omVolumeArgs.getVolume()),
omVolumeArgs);
+ // update bucket usedBytes.
+ omMetadataManager.getBucketTable().putWithBatch(batchOperation,
+ omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
+ omBucketInfo.getBucketName()),
+ omBucketInfo);
Review comment:
the last two lines can be merged into one line.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCommitResponse.java
##########
@@ -77,6 +80,11 @@ public void addToDBBatch(OMMetadataManager omMetadataManager,
omMetadataManager.getVolumeTable().putWithBatch(batchOperation,
omMetadataManager.getVolumeKey(omVolumeArgs.getVolume()),
omVolumeArgs);
+ // update bucket usedBytes.
+ omMetadataManager.getBucketTable().putWithBatch(batchOperation,
+ omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
+ omBucketInfo.getBucketName()),
+ omBucketInfo);
Review comment:
same as above.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java
##########
@@ -99,6 +103,11 @@ protected void addToDBBatch(OMMetadataManager omMetadataManager,
omMetadataManager.getVolumeTable().putWithBatch(batchOperation,
omMetadataManager.getVolumeKey(omVolumeArgs.getVolume()),
omVolumeArgs);
+ // update bucket usedBytes.
+ omMetadataManager.getBucketTable().putWithBatch(batchOperation,
+ omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
+ omBucketInfo.getBucketName()),
+ omBucketInfo);
Review comment:
the same.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java
##########
@@ -99,6 +103,11 @@ protected void addToDBBatch(OMMetadataManager omMetadataManager,
omMetadataManager.getVolumeTable().putWithBatch(batchOperation,
omMetadataManager.getVolumeKey(omVolumeArgs.getVolume()),
omVolumeArgs);
+ // update bucket usedBytes.
+ omMetadataManager.getBucketTable().putWithBatch(batchOperation,
+ omMetadataManager.getBucketKey(omVolumeArgs.getVolume(),
+ omBucketInfo.getBucketName()),
+ omBucketInfo);
Review comment:
the same. please double check the rest files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1431:
URL: https://github.com/apache/hadoop-ozone/pull/1431#issuecomment-693957270
hi @ChenSammi @xiaoyuyao, this PR is base on #1296. Can you help to review this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org