You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sumitagrawl (via GitHub)" <gi...@apache.org> on 2023/11/30 05:07:06 UTC
[PR] HDDS-9803. limit number of bucket to avoid OOM [ozone]
sumitagrawl opened a new pull request, #5705:
URL: https://github.com/apache/ozone/pull/5705
## What changes were proposed in this pull request?
limit the number of buckets with default to 100000, to avoid OOM when huge number of buckets are created. This is as buckets are full cached in OM.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9803
## How was this patch tested?
- Unit test is added to check bucket limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9803. Limit total number of buckets to avoid OutOfMemoryError in OM [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5705:
URL: https://github.com/apache/ozone/pull/5705#issuecomment-1840057905
> @sumitagrawl can you please resolve the conflicting files?
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9803. Limit total number of buckets to avoid OutOfMemoryError in OM [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5705:
URL: https://github.com/apache/ozone/pull/5705#issuecomment-1839093942
@sumitagrawl can you please resolve the conflicting files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9803. limit number of bucket to avoid OOM [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 commented on PR #5705:
URL: https://github.com/apache/ozone/pull/5705#issuecomment-1833268084
@sumitagrawl can you check if the test failures are related?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9803. limit number of bucket to avoid OOM [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5705:
URL: https://github.com/apache/ozone/pull/5705#issuecomment-1835439997
> Shouldn't we also limit volumes at the same time? Creating too many volumes can also result in OOME.
We already have max number of volume per user restriction, "ozone.om.user.max.volume" with default value 1024. May be this is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9803. limit number of bucket to avoid OOM [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #5705:
URL: https://github.com/apache/ozone/pull/5705#issuecomment-1833626312
This is a direct and efficient solution.
But wouldn't it be better if we limited the number of buckets in memory instead of just limiting the number of buckets that can be created, since it's possible that more than just buckets have this problem, and `volume` is `FULL_CACHE` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9803. limit number of bucket to avoid OOM [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #5705:
URL: https://github.com/apache/ozone/pull/5705#discussion_r1410979031
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -4245,4 +4245,12 @@
to existing buckets till this operation is completed.
</description>
</property>
+ <property>
+ <name>ozone.om.max.bucket</name>
Review Comment:
Nit: `ozone.om.max.buckets`.
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TypedTable.java:
##########
@@ -441,6 +441,9 @@ public String toString() {
@Override
public long getEstimatedKeyCount() throws IOException {
+ if (cache instanceof FullTableCache) {
Review Comment:
I'd prefer adding `CacheType type()` in `TableCache`, and using `cache.type() == FULL_CACHE` instead of `instanceof` check.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java:
##########
@@ -142,6 +145,23 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
.setCreateBucketRequest(newCreateBucketRequest.build()).build();
}
+ private static void validateMaxBucket(OzoneManager ozoneManager)
+ throws IOException {
+ int maxBucket = ozoneManager.getConfiguration().getInt(
+ OMConfigKeys.OZONE_OM_MAX_BUCKET,
+ OMConfigKeys.OZONE_OM_MAX_BUCKET_DEFAULT);
+ if (maxBucket <= 0) {
+ maxBucket = OMConfigKeys.OZONE_OM_MAX_BUCKET_DEFAULT;
+ }
+ long nrOfBuckets = ozoneManager.getMetadataManager().getBucketTable()
+ .getEstimatedKeyCount();
+ if (nrOfBuckets >= maxBucket) {
+ throw new OMException("Number of bucket crosses limit " +
Review Comment:
Nit: I think `"Cannot create more than " + maxBucket + " buckets"` would be a better message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9803. Limit total number of buckets to avoid OutOfMemoryError in OM [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on PR #5705:
URL: https://github.com/apache/ozone/pull/5705#issuecomment-1844881839
Thanks @sumitagrawl for the change , @xichen01, @adoroszlai for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9803. Limit total number of buckets to avoid OutOfMemoryError in OM [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 merged PR #5705:
URL: https://github.com/apache/ozone/pull/5705
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9803. limit number of bucket to avoid OOM [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5705:
URL: https://github.com/apache/ozone/pull/5705#issuecomment-1833900327
> This is a direct and efficient solution. But wouldn't it be better if we limited the number of buckets in memory instead of just limiting the number of buckets that can be created, since it's possible that more than just buckets have this problem, and `volume` is `FULL_CACHE` as well.
Thanks @xichen01 for review, bucket is full cache table, and for performance, it was cached.
- most of read/write operation involves bucket access / update. making it partial cache may impact performance and have impact to most of code where assumption is done for bucket to be full cache.
So IMO, its not viable to make bucket as partial cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org