You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ivan Andika (Jira)" <ji...@apache.org> on 2023/09/21 01:40:00 UTC
[jira] [Comment Edited] (HDDS-9095) Cleanup expired incomplete MPUs
[ https://issues.apache.org/jira/browse/HDDS-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749007#comment-17749007 ]
Ivan Andika edited comment on HDDS-9095 at 9/21/23 1:39 AM:
------------------------------------------------------------
Hi [~erose] [~ckj] , related to HDDS-4120, currently OpenKeyCleanupService deletes all the open keys in the openKeyTable regardless whether the open keys are MPU-related or not. This might cause orphan MPU in MultipartInfoTable, since to abort the MultipartInfo, we need to check openKeyTable for MPU-related open keys, which will fail if the open keys are already deleted. Hence, we might need to only delete non-MPU open keys in the openKeyCleanupService.
I have raised HDDS-9017 to set isMultipartKey to true for MPU-related open keys. In this way, MPU open keys can be excluded in getExpiredOpenKeys. The expired MPU keys can be aborted by a similar background task to abort expired MPUs (this ticket).
Currently there are two compatibility issues that need to be handled :
# For old MPU open keys that did not set isMultipartKey to true, how can openKeyCleanupService decides to not delete these keys? There are few possible solutions
## We repair the OM DB manually (similar to the current effort in HDDS-8097) to set isMultipartKey to true. Afterwards, we only delete open keys with isMultipartKey = false.
## We use regexes / pattern matching for the open MPU DB keys and do not include these keys in the getExpiredOpenKeys. The patterns are as follows
### Open MPU Key Info (dbKey = /\{vol}/\{bucket}/\{key}{*}/\{uploadId}{*}) where uploadId = "UUID.randomUUID().toString() + "-" + UniqueId.next()"
### Open MPU Part Key Info (dbKey = /\{vol}/\{bucket}/\{key}{*}/\{clientId}{*}) where clientId = UniqueId#next (This is also the pattern of normal MPU keys have, but we might not need since S3 abort request do not check against the existence of this key)
### The pattern matching can be used as a fast check with the possibility of false positives, in which case, we can check the MPU table whether there is an indeed incomplete MPU.
# We might need to deal with the cleaning up MPUs whose open keys have been deleted by the openKeyCleanupService.
## This might require us to have a new request/response similar to MPU abort request/response, but without failing when open MPU keys do not exist. This is not that elegant since it will duplicate most of the MPU abort logic.
Any feedback is appreciated. Thanks.
was (Author: JIRAUSER298977):
Hi [~erose] [~ckj] , related to HDDS-4120, currently OpenKeyCleanupService deletes all the open keys in the openKeyTable regardless whether the open keys are MPU-related or not. This might cause orphan MPU in MultipartInfoTable, since to abort the MultipartInfo, we need to check openKeyTable for MPU-related open keys, which will fail if the open keys are already deleted. Hence, we might need to only delete non-MPU open keys in the openKeyCleanupService.
I have raised HDDS-9017 to set isMultipartKey to true for MPU-related open keys. In this way, MPU open keys can be excluded in getExpiredOpenKeys. The expired MPU keys can be aborted by a similar background task to abort expired MPUs (this ticket).
Currently there are two compatibility issues that need to be handled :
# (This can be done as separate ticket) For old MPU open keys that did not set isMultipartKey to true, how can openKeyCleanupService decides to not delete these keys? There are few possible solutions
## We repair the OM DB manually (similar to the current effort in HDDS-8097) to set isMultipartKey to true. Afterwards, we only delete open keys with isMultipartKey = false.
## We use regexes for the open MPU DB keys and do not include these keys in the getExpiredOpenKeys. The patterns are as follows
### Open MPU Key Info (dbKey = /\{vol}/\{bucket}/\{key}{*}/\{uploadId}{*}) where uploadId = "UUID.randomUUID().toString() + "-" + UniqueId.next()"
### Open MPU Part Key Info (dbKey = /\{vol}/\{bucket}/\{key}{*}/\{clientId}{*}) where clientId = UniqueId#next (This is also the pattern of normal MPU keys have, but we might not need since S3 abort request do not check against the existence of this key)
# (This can be handled in this ticket) We might need to deal with the cleaning up MPUs whose open keys have been deleted by the openKeyCleanupService.
## This might require us to have a new request/response similar to MPU abort request/response, but without failing when open MPU keys do not exist. This is not that elegant since it will duplicate most of the MPU abort logic.
Any feedback is appreciated. Thanks.
> Cleanup expired incomplete MPUs
> -------------------------------
>
> Key: HDDS-9095
> URL: https://issues.apache.org/jira/browse/HDDS-9095
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
> Fix For: 1.4.0
>
>
> In our cluster, there are around few thousands MPU keys in multipartInfoTable, with some of them being few months old.
> The reason is that these MPU keys are already initiated, and possibly committed few parts, but was not completed / aborted by the user. These spaces can be freed.
> Similar to the cleanup service OM open key table (HDDS-4120), we can implement clean up on MultipartInfoTable (and related open keys in OpenKeyTable) using a background job. -However, instead of using a new OM request / response, we can reuse the OM MPU abort request / response instead, which already handles the cleanup (i.e. expired inflight MPU can be aborted after a defined expiry threshold).-
>
> This is similar to {{AbortIncompleteMultipartUpload}} lifecycle configurations in AWS S3 ([https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/]).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org