You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ivandika3 (via GitHub)" <gi...@apache.org> on 2023/09/20 01:19:00 UTC
[GitHub] [ozone] ivandika3 opened a new pull request, #5325: HDDS-9194. Implement cleanup service for MultipartInfoTable
ivandika3 opened a new pull request, #5325:
URL: https://github.com/apache/ozone/pull/5325
## What changes were proposed in this pull request?
Cleanup service to clean up expired MPUs that are not successfully completed / aborted.
Some notable information:
- The code structure is similar to the implementation of the OpenKeyCleanupService.
- The Request/Response is similar to the S3MultipartUploadAbortRequest/Response, but unlike normal abort, the request will not fail if the MPU open key does not exist in the openKeyTable (e.g previously deleted in the previous openKeyCleanupService)
- The MPU cleanup service has `ozone.om.open.mpu.parts.cleanup.limit.per.task `which limit the maximum number of MPU parts to delete rounded up to the nearest MPU (to prevent issues where MPUs that have parts more than `ozone.om.open.mpu.parts.cleanup.limit.per.task` never going to be deleted).
- For each run, `Map<OmBucketInfo, List<OmMultipartAbortInfo>>` is created which contains the batch information to abort the expired MPUs, grouped by bucket for more efficient locking.
- Logic regarding multipart abort response is refactored to `AbstractS3MultipartAbortResponse` to accomodate the batched approach of the cleanup service.
- Since `OmMultipartAbortInfo` contains bucket layout information to handle cases that MPUs belongs to multiple buckets with different layout, `AbstractS3MultipartAbortResponse` and the subclasses will get the bucket layout from `OmMultipartAbortInfo#getBucketLayout` instead of `OmKeyResponse#getBucketLayout`
- We will skip MPU cleanup if the underlying bucket does not exist
- HDDS-9318 will enforce that bucket cannot be deleted before all the MPUs under the bucket are completed/aborted/cleaned up
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9194
## How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ivandika3 commented on pull request #5325: HDDS-9194. Implement cleanup service for MultipartInfoTable
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #5325:
URL: https://github.com/apache/ozone/pull/5325#issuecomment-1726760998
Clean CI run: https://github.com/ivandika3/ozone/actions/runs/6237422589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9194. Implement cleanup service for MultipartInfoTable [ozone]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen merged PR #5325:
URL: https://github.com/apache/ozone/pull/5325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #5325: HDDS-9194. Implement cleanup service for MultipartInfoTable
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5325:
URL: https://github.com/apache/ozone/pull/5325#issuecomment-1730305741
@ashishkumar50 @SaketaChalamchala please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9194. Implement cleanup service for MultipartInfoTable [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #5325:
URL: https://github.com/apache/ozone/pull/5325#issuecomment-1744235957
@kaijchen Thank you for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ivandika3 commented on pull request #5325: HDDS-9194. Implement cleanup service for MultipartInfoTable
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #5325:
URL: https://github.com/apache/ozone/pull/5325#issuecomment-1726779469
@kaijchen @errose28 Could you take a look if you have time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org