You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/07/21 10:20:07 UTC
[GitHub] [ozone] symious opened a new pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
symious opened a new pull request #2448:
URL: https://github.com/apache/ozone/pull/2448
## What changes were proposed in this pull request?
Currently the keyLocation of different versions is stored redundantly in OmKeyInfo.
OmKeyInfo:
keyLocationVersions: List<OmKeyLocationInfoGroup> stores different versions of information in a list.
OmKeyLocationInfoGroup:
locationVersionMap: Map<Long, List<OmKeyLocationInfo>> stores different versions of location in a map.
If the versions are large, the redundent location is causing the large GC overhead for OM, in our cluster, the OM even crashes because of OOM.
This ticket is to remove the redundant location information to keep OM healthy.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5472
## How was this patch tested?
unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kuenishi commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
kuenishi commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-888144644
@bharatviswa504 Just curious, what's the sense behind the term "to disable versioning" ? For now it's unavailable anyway. Nobody can use, enable or disable versioning as of today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-887278595
IMHO lets use this PR to solve the problem of duplicating blocks in each version. And after this we can get during createKey return only latest version (#2432) and then we can use #2433 for using version flag in code path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-884785045
The failed test passed locally. Will try to retrigger the test later.
@arp7 @elek @adoroszlai @bharatviswa504 Could you help to check this PR?
This change might have some conflict with the initial design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885491606
> +1 to to remove the redundant blocks from the latest version and have in each version only that particular block version.
> I think we can initiate a discussion on the design of OmKeyLocationInfoGroup?
+1, you can come up with a doc, which explains the problem and the proposal and share it with community for feedback. This way it will let others know about this.
cc @kuenishi also as he is also interested in this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-886405596
@kuenishi Thanks for the review. I think option 2 is preferable too.
Since the current overhead is caused by the redundancy in multi-version storing, if we have this patch or the initial patch in #2433, the issue would be solved quite a lot. The overhead we have now to rewrite a file 1000 times would cost users to rewrite over 500000 times with the patch.
The issue that the version can really become very large since it's unlimited will exists, and the structure of RocksDB storing keys might need some changes, but IMHO it can be considered when developing the feature of versioning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885415543
@symious
I have similar thoughts to remove the older blocks from the latest version, as I see no use from duplicating these blocks in to latest version along with keeping it in the old version.
I have explained about this problem [here](https://issues.apache.org/jira/browse/HDDS-5393?focusedCommentId=17383243&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17383243)
Lets hear thoughts from others to know if any reason or is this is some mistake in code.
This PR also touching the same, but in this PR, we are targeting to remove older blocks in latest version, with out touching versioning. This PR tries to do more like when versioning is not enabled deleting older blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-890865693
> Can we merge this PR now? @bharatviswa504
As no more comments let's go ahead with merge. If any comments, let's revisit in further Jiras.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-888153065
> @bharatviswa504 Just curious, what's the sense behind the term "to disable versioning" ? For now it's unavailable anyway. Nobody can use, enable or disable versioning as of today.
Even though we don't support versioning, we do create new version of key when overriding, and list all versions during put/get/list. If bucket versioning is false, we can avoid all this, and we shall have only one version at a time. But still we need to be compatible with AWS S3, (I still need to explore how AWS S3 behaves)
I mean we have versioning boolean in BucketArgs, BucketInfo, we are not using it, use this boolean and allow only one version if versioning is false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885570000
@bharatviswa504 Initiated the doc, could you help to check? https://docs.google.com/document/d/1UYVETpSHyVppNPjHq8VuXj2t2Y0LYQNRYa6awSTCROA/edit?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-888769154
@ChenSammi Thanks for the review: )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kuenishi commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
kuenishi commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-887962261
> IMHO lets use this PR to solve the problem of duplicating blocks in each version. And after this we can get during createKey return only latest version (#2432) and then we can use #2433 for using version flag in code path.
+1. I'll rebase #2433 after this PR merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-887282916
> IMHO lets use this PR to solve the problem of duplicating blocks in each version. And after this we can get during createKey return only latest version (#2432) and then we can use #2433 for using version flag in code path.
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-890790249
Can we merge this PR now? @bharatviswa504
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885415543
@symious
I have similar thoughts to remove the older blocks from the latest version, as I see no use from duplicating these blocks in to latest version along with keeping it in the old version.
I have explained about this problem [here](https://issues.apache.org/jira/browse/HDDS-5393?focusedCommentId=17383243&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17383243)
Lets hear thoughts from others to know if any reason or is this is some mistake in code.
This PR(#2433) also touching the same, but in this PR, we are targeting to remove older blocks in latest version, with out touching versioning. This PR(#2433) tries to do more like when versioning is not enabled deleting older blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885491606
> +1 to to remove the redundant blocks from the latest version and have in each version only that particular block version.
+1, you can come up with a doc, which explains the problem and the proposal and share it with community for feedback. This way it will let others know about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885415543
@symious
I have similar thoughts to remove the older blocks from the latest version, as I see no use from duplicating these blocks in to latest version along with keeping it in the old version.
This PR also touching the same, but in this PR, we are targeting to remove older blocks in latest version, with out touching versioning. This PR tries to do more like when versioning is not enabled deleting older blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885498873
> > +1 to to remove the redundant blocks from the latest version and have in each version only that particular block version.
> > I think we can initiate a discussion on the design of OmKeyLocationInfoGroup?
>
> +1, you can come up with a doc, which explains the problem and the proposal and share it with community for feedback. This way it will let others know about this.
>
> cc @kuenishi also as he is also interested in this
Sure, will create the doc later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kuenishi commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
kuenishi commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-888739952
Thank you, I got it. It's disabled by default for now, but from performance perspective, not by feature, it'd be better disabled anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885448139
@bharatviswa504 Thanks for the review.
I have checked the PR of #2433 , but I'm not sure we need to use another variable to control if we need to delete the old locations only for buckets with versioning disabled.
Personally, I'm very glad to see the feature of versioning in Ozone, and I think if it's released, it should be very welcomed by users. IMHO, if the feature was released, everyone likes the feature, most of the users will have this feature enabled, and the issue will still exist.
As we discussed in https://github.com/apache/ozone/pull/2261#issuecomment-843308499, if there are 1000 versions, it's ok to have 1000 storage and process units for all versions, the issue we have now is its redundancy causing 500500 storage and process units. I think 1000 is acceptable with the advantage that we don't even have to change the original design too much but restored all the original locations.
If there is some automatic program rewriting the same forever, causing a very large version location, I think what we need is a version limit mechanism that can control the malicious case here.
I think this would be a better way to align with the future release of versioning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-888155808
I think maybe it's the issues we are facing now causing the users not to want to use versioning. If the issue is solved or mitigated, users won't complain too much since there isn't so much overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
symious commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885487897
> As I said I am +1 to to remove the redundant blocks from the latest version and have in each version only that particular block version.
I think we can initiate a discussion on the design of OmKeyLocationInfoGroup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-887278595
IMHO lets use this PR to solve the problem of duplicating blocks in each version. And after this we can get during createKey return only latest version (#2432) and then we can use #2433 for using version flag in code path.
In this way, with above PR's most of the problems are solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-888082425
Let us give few days for couple of folks to take a look at it, before proceeding with merge.
Had an offline discussion with @ChenSammi, she said she will take a look at it. (As couple of customers in tencent wants to disable versioning, the PR's here can help)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-890865693
+1 from me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi merged pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
ChenSammi merged pull request #2448:
URL: https://github.com/apache/ozone/pull/2448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-888766424
One of our customers reports OM out of service because of very heavy GC. After some investigation, they are uploading objects through s3g periodically with same names, and in the mean while, fetching the objects corrently. Here is top part of the jstack output.
num #instances #bytes class name
----------------------------------------------
1: 56349154 3213812672 [Ljava.lang.Object;
2: 54898991 2635151568 org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo
3: 54907944 1757054208 java.util.HashMap$Node
4: 54898991 1756767712 org.apache.hadoop.hdds.client.ContainerBlockID
5: 21320616 1535084352 org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$KeyLocation
6: 56349257 1352382168 java.util.ArrayList
7: 54898991 1317575784 org.apache.hadoop.hdds.client.BlockID
8: 21320616 1193954496 org.apache.hadoop.hdds.protocol.proto.HddsProtos$ContainerBlockID
9: 21320617 1023389616 org.apache.hadoop.hdds.protocol.proto.HddsProtos$BlockID
The total heap size is 18G+. The root cause is just the same as your cases.
After the issue is workaround by deleting some unsued objects, the customer raised a requirement to disable the versioning on the bucket.
I checked the AWS S3 behavior, bucket versioning is by default disabled. https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html.
I'm glad that we have started to solve this problem, for example, HDDS-5472, HDDS-5393, HDDS-5461 and another PR to control the client side. I totally agree that we can do it step by step.
For this patch, my +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kuenishi commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
kuenishi commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-886387860
Thanks for heads up, @symious . I didn't even expect this issue caused OOM. This patch is minimal and nice (same patch as my initial one). I'm fine either patch is merged.
My patch (#2433) is a little bit complex than #2448, but when it comes to supporting versioning, it will reduce the total effort of implementing object versioning. In either case, my opinion is we have to change the way how we store multiple versions in RocksDB, because S3 API does not have any limit on total number of versions per key. If we store all versions in a single key in RocksDB, the size of the value can be infinitely large that potentially breaks every assumption about value size in OM and RocksDB. My suggestion is to append some sorted (descending) suffix to key name to use scan API of RocksDB. Otherwise, just getting latest version of an object will involve getting all versions, leading to unnecessary read load to underlying storage device.
In that perspective of potential design of versioning, proposal no.2 in @symious 's doc would be preferable. My :+1: is for single key-value in RocksDB storing single set of blocks that belongs to single version of an object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885491606
> +1 to to remove the redundant blocks from the latest version and have in each version only that particular block version.
> I think we can initiate a discussion on the design of OmKeyLocationInfoGroup?
+1, you can come up with a doc, which explains the problem and the proposal and share it with community for feedback. This way it will let others know about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885415543
@symious
I have similar thoughts to remove the older blocks from the latest version, as I see no use from duplicating these blocks in to latest version along with keeping it in the old version.
Lets hear thoughts from others to know if any reason or is this is some mistake in code.
This PR also touching the same, but in this PR, we are targeting to remove older blocks in latest version, with out touching versioning. This PR tries to do more like when versioning is not enabled deleting older blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-885456153
>but I'm not sure we need to use another variable to control if we need to delete the old locations only for buckets with >versioning disabled.
If Versioning is disabled, when overriding key the PR fix is to delete older version and have only one version. (And also it has this PR fix also.)
>discussed in [#2261 (comment)](https://github.com/apache/ozone/pull/2261#issuecomment-843308499), if there are 1000
>versions, it's ok to have 1000 storage and process units for all versions, the issue we have now is its redundancy causing >500500 storage and process units. I think 1000 is acceptable with the advantage that we don't even have to change the >original design too much but restored all the original locations.
As I said I am +1 to to remove the redundant blocks from the latest version and have in each version only that particular block version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org