You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2026/06/13 16:32:49 UTC
[PR] [fix][broker] Time out topic policies cache initialization when the __change_events reader is stuck [pulsar]
lhotari opened a new pull request, #26025:
URL: https://github.com/apache/pulsar/pull/26025
Main Issue: #25294
### Motivation
When a topic is loaded, the broker first waits for the namespace's topic policies cache to be
initialized. Initialization (`SystemTopicBasedTopicPoliciesService#initPolicesCache`) reads the
namespace's `__change_events` system topic to the end via a recursive `readNextAsync()` loop and
completes a **shared, per-namespace** future (`policyCacheInitMap`) that every topic load in the
namespace awaits.
That read loop has **no timeout**. If the system-topic reader reconnects but then stops making
progress — e.g. after `__change_events` is unloaded/moved and the reconnected reader gets stuck (see
the compacted-read stuck-reader bug fixed in #25998) — the shared init future stays pending forever.
There is a 60s `topicLoadTimeoutSeconds` on the individual topic-load future, but its handler only
logs; it does not clear the poisoned `policyCacheInitMap` entry or close the stuck reader. As a
result the per-namespace state stays poisoned and **every** topic in the namespace remains stuck and
unloadable until the broker is restarted, as reported in #25294.
This is a defense-in-depth fix that is complementary to #25998 (which fixes one concrete stuck-reader
trigger on the broker/cursor side): the timeout bounds the wait regardless of *why* a reader is stuck,
so a single stuck reader can no longer take a whole namespace's topics down until restart.
### Modifications
- Add `topicPoliciesCacheInitTimeoutSeconds` (default `60`, dynamic). It bounds topic policies cache
initialization for a namespace. Set to `0` or a negative value to disable (previous unbounded
behavior).
- In `SystemTopicBasedTopicPoliciesService#prepareInitPoliciesCacheAsync`, schedule a timeout on the
per-namespace init future. If the timeout wins the race to complete the future, the broker:
- fails the init future with a `TimeoutException` (so awaiting topic loads fail fast and can retry),
- clears the cached state and closes the stuck `__change_events` reader via
`cleanPoliciesCacheInitMap(namespace, true)`, so a subsequent load creates a fresh reader instead
of being pinned until restart,
- increments a new `pulsar.broker.topic.policies.cache.init.timeout.count` OpenTelemetry counter.
The timeout task is cancelled as soon as initialization completes (success or failure), so it adds
no overhead on the normal path.
### Verifying this change
This change added tests and can be verified as follows:
- Added `SystemTopicBasedTopicPoliciesServiceTest#testPrepareInitPoliciesCacheAsyncTimesOutWhenReaderStuck`,
which spies the `__change_events` reader so it reports more events but never delivers one (a stuck
reader), then asserts that `prepareInitPoliciesCacheAsync` fails with a `TimeoutException` (instead of
hanging), that the poisoned `policyCacheInitMap` entry is cleared, and that the stuck reader is closed.
Verified red without the fix (the future never completes) and green with it.
- The full `SystemTopicBasedTopicPoliciesServiceTest` suite passes, including the existing init/cleanup
tests, confirming the normal path (and cleanup call counts) is unaffected.
### Does this pull request potentially affect one of the following parts:
- [x] The default values of configurations *(new `topicPoliciesCacheInitTimeoutSeconds`, default 60s; topic policies cache initialization is now bounded by default instead of unbounded)*
- [x] The metrics *(new counter `pulsar.broker.topic.policies.cache.init.timeout.count`)*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't let a stuck or aborted topic policies cache init make a namespace's topics unloadable [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #26025:
URL: https://github.com/apache/pulsar/pull/26025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't let a stuck or aborted topic policies cache init make a namespace's topics unloadable [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #26025:
URL: https://github.com/apache/pulsar/pull/26025#issuecomment-4701398220
thanks @void-ptr974, pushed another commit to fix that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't let a stuck or aborted topic policies cache init make a namespace's topics unloadable [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #26025:
URL: https://github.com/apache/pulsar/pull/26025#issuecomment-4859293568
Follow up PR #26132 to address a remaining gap that came to the surface after this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org