You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2025/10/17 19:09:52 UTC
[PR] [improve][meta] Add byte size limit to AbstractMetadataStore's childrenCache [pulsar]
lhotari opened a new pull request, #24868:
URL: https://github.com/apache/pulsar/pull/24868
### Motivation
AbstractMetadataStore contains an unbounded childrenCache which might consume a lot of heap memory when a lot of namespaces with a lot of topics are listed. The cache has a timeout of 10 minutes so the memory usage problem is limited to certain use cases.
### Modifications
Add a limit to the childrenCache so that oldest entries start to get expired when the cache consumes more than 20% of the maximum heap size.
Tests are TBD.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][meta] Add byte size limit to AbstractMetadataStore's childrenCache [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #24868:
URL: https://github.com/apache/pulsar/pull/24868#issuecomment-3461367707
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][meta] Add byte size limit to AbstractMetadataStore's childrenCache [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #24868:
URL: https://github.com/apache/pulsar/pull/24868#discussion_r2450252102
##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java:
##########
@@ -152,6 +180,20 @@ public CompletableFuture<Boolean> asyncReload(String key, Boolean oldValue,
this.metadataStoreStats = new MetadataStoreStats(metadataStoreName, openTelemetry);
}
+ /**
+ * Return the maximum size of the children cache in bytes.
+ * @return maximum size of the children cache in bytes.
+ */
+ protected long getChildrenCacheMaxSizeBytes() {
+ long heapMaxSizeBytes = Runtime.getRuntime().maxMemory();
+ // default 20% of max heap size, this should be sufficient to prevent OOME in the use case
+ // when a lot of namespaces with lots of topics are listed in the metadata store.
+ long defaultSizeBytes = heapMaxSizeBytes / 5;
+ // min size 20MB
+ int minSizeBytes = 1024 * 1024 * 20;
Review Comment:
It would be better if it were configurable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][meta] Add byte size limit to AbstractMetadataStore's childrenCache [pulsar]
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #24868:
URL: https://github.com/apache/pulsar/pull/24868#discussion_r2450252102
##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java:
##########
@@ -152,6 +180,20 @@ public CompletableFuture<Boolean> asyncReload(String key, Boolean oldValue,
this.metadataStoreStats = new MetadataStoreStats(metadataStoreName, openTelemetry);
}
+ /**
+ * Return the maximum size of the children cache in bytes.
+ * @return maximum size of the children cache in bytes.
+ */
+ protected long getChildrenCacheMaxSizeBytes() {
+ long heapMaxSizeBytes = Runtime.getRuntime().maxMemory();
+ // default 20% of max heap size, this should be sufficient to prevent OOME in the use case
+ // when a lot of namespaces with lots of topics are listed in the metadata store.
+ long defaultSizeBytes = heapMaxSizeBytes / 5;
+ // min size 20MB
+ int minSizeBytes = 1024 * 1024 * 20;
Review Comment:
LGTM, It would be better if it were configurable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][meta] Add byte size limit to AbstractMetadataStore's childrenCache [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #24868:
URL: https://github.com/apache/pulsar/pull/24868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org