You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/02/24 20:51:32 UTC
[GitHub] [ozone] sodonnel opened a new pull request #3136: HDDS-6279. EC: Calculate EC replication correctly when updating bucket usage
sodonnel opened a new pull request #3136:
URL: https://github.com/apache/ozone/pull/3136
## What changes were proposed in this pull request?
OM keeps a running total of the space consumed in a bucket each time a key is created or deleted. The space tracked, is the total space used by the key, including replication. As it stands, the datasize is multipied by the Factor to give the total space used. For EC, this is not correct.
This PR is to change all the places the bucket total is adjusted in OM, to cater for both Ratis and EC.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6279
## How was this patch tested?
New tests for the replication calculation. Manually for EC keys for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3136: HDDS-6279. EC: Calculate EC replication correctly when updating bucket usage
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3136:
URL: https://github.com/apache/ozone/pull/3136#discussion_r814529935
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
##########
@@ -229,7 +229,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
// AllocatedBlock. The space occupied by the Key shall be based on
// the actual Key size, and the total Block size applied before should
// be subtracted.
- long correctedSpace = omKeyInfo.getDataSize() * factor -
+ long correctedSpace = omKeyInfo.getReplicatedSize() -
Review comment:
This calculation is correct for EC keys. When a block is allocated, it adds blockSize * factor to the bucket usage, assuming the block will be filled. Factor here is:
```
int factor = omKeyInfo.getReplicationConfig().getRequiredNodes();
```
So for EC 3-2 it will be 5, for 6-3 it will be 9 etc. If the block was exactly filled, this would be the actual space used by the EC block including replication.
When the block is completed, we calculate the actual space used, and then remove the pre-allocated amount, leave the correct valie of usage added to the bucket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on a change in pull request #3136: HDDS-6279. EC: Calculate EC replication correctly when updating bucket usage
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #3136:
URL: https://github.com/apache/ozone/pull/3136#discussion_r815015783
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
##########
@@ -229,7 +229,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
// AllocatedBlock. The space occupied by the Key shall be based on
// the actual Key size, and the total Block size applied before should
// be subtracted.
- long correctedSpace = omKeyInfo.getDataSize() * factor -
+ long correctedSpace = omKeyInfo.getReplicatedSize() -
Review comment:
Thanks for the details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel merged pull request #3136: HDDS-6279. EC: Calculate EC replication correctly when updating bucket usage
Posted by GitBox <gi...@apache.org>.
sodonnel merged pull request #3136:
URL: https://github.com/apache/ozone/pull/3136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on a change in pull request #3136: HDDS-6279. EC: Calculate EC replication correctly when updating bucket usage
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #3136:
URL: https://github.com/apache/ozone/pull/3136#discussion_r814492861
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
##########
@@ -229,7 +229,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
// AllocatedBlock. The space occupied by the Key shall be based on
// the actual Key size, and the total Block size applied before should
// be subtracted.
- long correctedSpace = omKeyInfo.getDataSize() * factor -
+ long correctedSpace = omKeyInfo.getReplicatedSize() -
Review comment:
This subtraction is using blockSize*factor, can we think what we need to consider here for EC keys?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org