You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/07/29 03:11:36 UTC
[GitHub] [ozone] ChenSammi commented on pull request #2448: HDDS-5472. Old versions of location in OmKeyLocationInfoGroup causes OOM of OM
ChenSammi commented on pull request #2448:
URL: https://github.com/apache/ozone/pull/2448#issuecomment-888766424
One of our customers reports OM out of service because of very heavy GC. After some investigation, they are uploading objects through s3g periodically with same names, and in the mean while, fetching the objects corrently. Here is top part of the jstack output.
num #instances #bytes class name
----------------------------------------------
1: 56349154 3213812672 [Ljava.lang.Object;
2: 54898991 2635151568 org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo
3: 54907944 1757054208 java.util.HashMap$Node
4: 54898991 1756767712 org.apache.hadoop.hdds.client.ContainerBlockID
5: 21320616 1535084352 org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$KeyLocation
6: 56349257 1352382168 java.util.ArrayList
7: 54898991 1317575784 org.apache.hadoop.hdds.client.BlockID
8: 21320616 1193954496 org.apache.hadoop.hdds.protocol.proto.HddsProtos$ContainerBlockID
9: 21320617 1023389616 org.apache.hadoop.hdds.protocol.proto.HddsProtos$BlockID
The total heap size is 18G+. The root cause is just the same as your cases.
After the issue is workaround by deleting some unsued objects, the customer raised a requirement to disable the versioning on the bucket.
I checked the AWS S3 behavior, bucket versioning is by default disabled. https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html.
I'm glad that we have started to solve this problem, for example, HDDS-5472, HDDS-5393, HDDS-5461 and another PR to control the client side. I totally agree that we can do it step by step.
For this patch, my +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org