You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "thetumbled (via GitHub)" <gi...@apache.org> on 2025/09/22 10:04:25 UTC
[PR] [improve][pip] PIP-444: Rate limit for deleting ledger to alleviate the zk pressure. [pulsar]
thetumbled opened a new pull request, #24771:
URL: https://github.com/apache/pulsar/pull/24771
PIP documentation.
Implementation PR: https://github.com/apache/pulsar/pull/24760
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-444: Rate limit for deleting ledger to alleviate the zk pressure. [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on code in PR #24771:
URL: https://github.com/apache/pulsar/pull/24771#discussion_r2390055658
##########
pip/pip-444.md:
##########
@@ -0,0 +1,55 @@
+
+# PIP-444: Rate limit for deleting ledger to alleviate the zk pressure.
+
+# Background knowledge
+
+Apache Pulsar uses Apache BookKeeper as its storage layer.
+Each topic consists of multiple ledgers, which are created periodically based on the configuration.
+Each ledger corresponds to a ZooKeeper (ZK) node for metadata storage. When a ledger is deleted, the corresponding ZK node will also be deleted.
+Retention policies determine how long messages are retained in a topic. When the retention period is reached, the corresponding ledgers will be deleted.
+
+# Motivation
+
+When the retention of a large topic is reduced, a significant number of ledgers need to be deleted.
+Since this deletion operation is not rate-limited, it could results in ZooKeeper (ZK) latency of several minutes,
+which threatens the stability of the entire Pulsar cluster.
+
+# Goals
+
+Add rate limit feature for deleting ledgers.
+
+# Detailed Design
+
+Introduce a new configuration parameter:
+```
+ @FieldContext(
+ category = CATEGORY_STORAGE_ML,
+ doc = "Max number of concurrent requests for deleting ledgers at broker level"
+ )
+ private int managedLedgerDeleteMaxConcurrentRequests = 1000;
+```
+These parameters can be configured in the `broker.conf` file.
+
+When the `managedLedgerDeleteMaxConcurrentRequests` parameter is set to a value greater than 0,
+the concurrency of ledger deletions will be limited to the specified value.
+
+# Backward & Forward Compatibility
+
+Fully backward compatible and forward compatible.
+
+## Upgrade
+
+New feature is enabled by default with a rate limit of 1000 deletions per second and concurrency of 5.
Review Comment:
```suggestion
New feature is enabled by default with a rate limit of 1000 deletions per second and concurrency of 1.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-444: Rate limit for deleting ledger to alleviate the zk pressure. [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled merged PR #24771:
URL: https://github.com/apache/pulsar/pull/24771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][pip] PIP-444: Rate limit for deleting ledger to alleviate the zk pressure. [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #24771:
URL: https://github.com/apache/pulsar/pull/24771#discussion_r2384871191
##########
pip/pip-444.md:
##########
@@ -0,0 +1,62 @@
+
+# PIP-444: Rate limit for deleting ledger to alleviate the zk pressure.
+
+# Background knowledge
+
+Apache Pulsar uses Apache BookKeeper as its storage layer.
+Each topic consists of multiple ledgers, which are created periodically based on the configuration.
+Each ledger corresponds to a ZooKeeper (ZK) node for metadata storage. When a ledger is deleted, the corresponding ZK node will also be deleted.
+Retention policies determine how long messages are retained in a topic. When the retention period is reached, the corresponding ledgers will be deleted.
+
+# Motivation
+
+When the retention of a large topic is reduced, a significant number of ledgers need to be deleted.
+Since this deletion operation is not rate-limited, it could results in ZooKeeper (ZK) latency of several minutes,
+which threatens the stability of the entire Pulsar cluster.
+
+# Goals
+
+Add rate limit feature for deleting ledgers.
+
+# Detailed Design
+
+Introduce two new configuration parameters:
+```
+ @FieldContext(
+ category = CATEGORY_STORAGE_ML,
+ doc = "Max number of concurrent requests for deleting ledgers at broker level"
+ )
+ private int managedLedgerDeleteMaxConcurrentRequests = 1000;
+ @FieldContext(
+ category = CATEGORY_STORAGE_ML,
+ doc = "Number of threads to be used for deleting ledgers at broker level"
+ )
+ private int managedLedgerDeleteThreadPoolSize = 5;
Review Comment:
Using 5 threads only to call an asynchronous operation is a waste, it's better to change the default value to 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org