Re: [PR] [feat][broker] PIP-470: Close inactive topics without deleting their data [pulsar]

["poorbarcode (via GitHub)" <gi...@apache.org>]

poorbarcode commented on code in PR #25574:
URL: https://github.com/apache/pulsar/pull/25574#discussion_r3849464441


##########
pip/pip-470.md:
##########
@@ -0,0 +1,301 @@
+# PIP-470: Close Inactive Topics Without Deleting Them
+
+# Background Knowledge
+
+Pulsar has a long-standing inactive-topic GC mechanism controlled by the broker
+configuration `brokerDeleteInactiveTopicsEnabled`. When enabled, the broker
+periodically iterates over loaded topics, detects topics with no active
+producers/consumers (and optionally no subscriptions or no backlog), and
+**deletes** them — both the in-memory topic instance and the persistent data
+in BookKeeper, plus topic-level metadata.
+
+The detection logic lives in `PersistentTopic.checkGC()` /
+`NonPersistentTopic.checkGC()`. The policy object is
+`InactiveTopicPolicies`, which has two modes:
+
+- `delete_when_no_subscriptions`
+- `delete_when_subscriptions_caught_up`
+
+Both modes lead to permanent deletion of the topic and its data.
+
+Each loaded `PersistentTopic` pins a non-trivial amount of broker memory:
+a managed-ledger with its cache, subscription/cursor state, rate limiters,
+dispatchers, schema references, and — notably — per-topic metric series.
+Deployments with millions of low-traffic topics routinely hit memory pressure
+and metrics cardinality pressure even when most topics are idle most of the
+time.
+
+Pulsar also supports topic **unload** via the admin API
+(`pulsar-admin topics unload <topic>`) and namespace-bundle unload. Unload
+evicts the topic from the broker's topic map and releases all broker-side
+resources, while the ledgers in BookKeeper are preserved. The next
+produce/consume call transparently reloads the topic. There is currently no
+automatic mechanism that unloads idle topics based on inactivity.
+
+# Motivation
+
+Operators with very large topic counts (commonly tens of thousands to millions
+per cluster, with a long tail of low-traffic topics) face two problems today:
+
+1. **Broker memory pressure.** Every loaded topic holds a managed-ledger,
+   cursors, rate limiters, dispatchers, and metric series. An idle topic that
+   hasn't been produced/consumed for hours still occupies all of that memory.
+2. **Metrics cardinality.** Per-topic metrics grow linearly with the number
+   of loaded topics. Scrape endpoints get large and the monitoring system
+   gets expensive.
+
+The natural remedy today is `brokerDeleteInactiveTopicsEnabled`, but that
+**deletes the data**. Many users explicitly do not want deletion — they want
+to keep the data around for later consumers, replay, compliance retention, etc.
+Their only options today are:
+
+- Leave all idle topics loaded and accept the memory/metrics cost.
+- Build an external cron job that polls `topics stats` for zero
+  producers/consumers and calls `pulsar-admin topics unload` per topic.
+
+The external-cron approach works but is awkward: it reimplements the existing
+broker inactivity detection, has to run with admin credentials, adds a moving
+part to the deployment, and lacks integration with the existing
+`InactiveTopicPolicies` configuration surface.
+
+This PIP adds a first-class broker option that performs the same idle-topic
+detection the broker already does for delete-GC, but performs a **close**
+(unload) instead of a delete when the topic is determined to be inactive.
+
+# Goals
+
+## In Scope
+
+- A new broker configuration `brokerCloseInactiveTopicsEnabled` (default
+  `false`) that, when enabled, causes the inactivity monitor to **close**
+  (unload) topics determined to be inactive rather than delete them.
+- Mutual exclusion with `brokerDeleteInactiveTopicsEnabled`: only one of the

Review Comment:
   We'd better add a new limitation: the new feature only `brokerDeleteInactiveTopicsMode` is `delete_when_no_subscriptions`. Otherwise, the active consumer will load up the topic again after an inactivation, which leads to frequent invalid unloading and loading up



##########
pip/pip-470.md:
##########
@@ -0,0 +1,301 @@
+# PIP-470: Close Inactive Topics Without Deleting Them
+
+# Background Knowledge
+
+Pulsar has a long-standing inactive-topic GC mechanism controlled by the broker
+configuration `brokerDeleteInactiveTopicsEnabled`. When enabled, the broker
+periodically iterates over loaded topics, detects topics with no active
+producers/consumers (and optionally no subscriptions or no backlog), and
+**deletes** them — both the in-memory topic instance and the persistent data
+in BookKeeper, plus topic-level metadata.
+
+The detection logic lives in `PersistentTopic.checkGC()` /
+`NonPersistentTopic.checkGC()`. The policy object is
+`InactiveTopicPolicies`, which has two modes:
+
+- `delete_when_no_subscriptions`
+- `delete_when_subscriptions_caught_up`
+
+Both modes lead to permanent deletion of the topic and its data.
+
+Each loaded `PersistentTopic` pins a non-trivial amount of broker memory:
+a managed-ledger with its cache, subscription/cursor state, rate limiters,
+dispatchers, schema references, and — notably — per-topic metric series.
+Deployments with millions of low-traffic topics routinely hit memory pressure
+and metrics cardinality pressure even when most topics are idle most of the
+time.
+
+Pulsar also supports topic **unload** via the admin API
+(`pulsar-admin topics unload <topic>`) and namespace-bundle unload. Unload
+evicts the topic from the broker's topic map and releases all broker-side
+resources, while the ledgers in BookKeeper are preserved. The next
+produce/consume call transparently reloads the topic. There is currently no
+automatic mechanism that unloads idle topics based on inactivity.
+
+# Motivation
+
+Operators with very large topic counts (commonly tens of thousands to millions
+per cluster, with a long tail of low-traffic topics) face two problems today:
+
+1. **Broker memory pressure.** Every loaded topic holds a managed-ledger,
+   cursors, rate limiters, dispatchers, and metric series. An idle topic that
+   hasn't been produced/consumed for hours still occupies all of that memory.
+2. **Metrics cardinality.** Per-topic metrics grow linearly with the number
+   of loaded topics. Scrape endpoints get large and the monitoring system
+   gets expensive.
+
+The natural remedy today is `brokerDeleteInactiveTopicsEnabled`, but that
+**deletes the data**. Many users explicitly do not want deletion — they want
+to keep the data around for later consumers, replay, compliance retention, etc.
+Their only options today are:
+
+- Leave all idle topics loaded and accept the memory/metrics cost.
+- Build an external cron job that polls `topics stats` for zero
+  producers/consumers and calls `pulsar-admin topics unload` per topic.
+
+The external-cron approach works but is awkward: it reimplements the existing
+broker inactivity detection, has to run with admin credentials, adds a moving
+part to the deployment, and lacks integration with the existing
+`InactiveTopicPolicies` configuration surface.
+
+This PIP adds a first-class broker option that performs the same idle-topic
+detection the broker already does for delete-GC, but performs a **close**
+(unload) instead of a delete when the topic is determined to be inactive.
+
+# Goals
+
+## In Scope
+
+- A new broker configuration `brokerCloseInactiveTopicsEnabled` (default
+  `false`) that, when enabled, causes the inactivity monitor to **close**
+  (unload) topics determined to be inactive rather than delete them.
+- Mutual exclusion with `brokerDeleteInactiveTopicsEnabled`: only one of the
+  two may be enabled at a time. Broker start fails fast if both are set.
+- Reuse of the existing inactivity detection: same
+  `brokerDeleteInactiveTopicsMode`, `brokerDeleteInactiveTopicsFrequencySeconds`,
+  and `brokerDeleteInactiveTopicsMaxInactiveDurationSeconds` settings determine
+  when a topic is considered inactive. No new detection surface is introduced
+  for v1.
+- Data preservation guarantee: only the in-memory topic instance is released.
+  BookKeeper ledgers, topic metadata, subscriptions, cursors, and topic-level
+  policies are all retained. The next producer/consumer reconnect transparently
+  reloads the topic.
+- Coverage for both `PersistentTopic` and `NonPersistentTopic`.
+
+## Out of Scope
+
+- A per-topic or per-namespace override for close-on-inactive. The v1 knob is
+  broker-level only; a follow-up PIP can extend `InactiveTopicPolicies` with
+  an action field (`delete` vs `close`) if operators want per-namespace
+  control.
+- Changing the existing `InactiveTopicDeleteMode` enum or
+  `InactiveTopicPolicies` schema.
+- A new admin API. Manual `pulsar-admin topics unload` remains available for
+  ad-hoc use.
+- Partitioned-topic metadata handling changes. Because nothing is deleted,
+  `brokerDeleteInactivePartitionedTopicMetadataEnabled` has no interaction
+  with the new close mode.
+- Changing the detection cadence or detection semantics beyond reusing
+  what's there.
+
+# High Level Design
+
+The existing inactivity monitor in `BrokerService.startInactivityMonitor()`
+schedules `checkGC()` on all loaded topics when
+`brokerDeleteInactiveTopicsEnabled` is true. With this PIP, the monitor also
+runs when `brokerCloseInactiveTopicsEnabled` is true.
+
+`PersistentTopic.checkGC()` and `NonPersistentTopic.checkGC()` gain a branch:
+if close mode is enabled (and delete mode is not), after the same
+isActive / duration / replication-producer checks pass, the topic calls its
+own `close(disconnectClients=true, closeWithoutWaitingClientDisconnect=false)`
+path. That path already:
+
+- disconnects producers and consumers,

Review Comment:
   Since the inactivation only happens when the topic is inactive, which confirms no producer registered, the inactivation will not disconnect producers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org