You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "alexandrebrg (via GitHub)" <gi...@apache.org> on 2026/09/09 16:06:17 UTC
[PR] [fix][broker] Don't serve topic policies from a cache whose init future has not completed [pulsar]
alexandrebrg opened a new pull request, #26513:
URL: https://github.com/apache/pulsar/pull/26513
Related: #26137, #20763
### Motivation
`SystemTopicBasedTopicPoliciesService.getTopicPoliciesAsync` can return `Optional.empty()` for a topic that has topic-level policies, without any exception or log line, when a namespace-bundle bounce (unload followed by reload of the namespace's last bundle on the same broker) lands inside the thread hop the method performs between awaiting the namespace's policy-cache initialization and reading the cache. A topic load that hits this window builds its `ManagedLedgerConfig` from the namespace retention instead of the topic retention, and nothing repairs it afterwards. The consequence -- a topic silently enforcing its namespace retention instead of its own while the admin API keeps reporting the topic value -- is what surfaced in a production cluster; the interleaving itself is not claimed to have been captured there. The tests in this PR reproduce it without sleeps or timing tuning: the unpatched failure only requires the replacement generation not to finish loading within the thre
ad hop.
All line numbers below refer to master at the merge base of this PR, before the change.
#### The mechanism
Annotated excerpt of `getTopicPoliciesAsync` (`pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java`, lines 580-606, quoted verbatim; the annotations follow the excerpt, keyed by line number):
```
580 final CompletableFuture<Boolean> preparedFuture = prepareInitPoliciesCacheAsync(topicName.getNamespaceObject());
581 // switch thread to avoid potential metadata thread cost and recursive deadlock
582 return preparedFuture.thenComposeAsync(inserted -> {
583 // initialized : policies
584 final Mutable<Pair<Boolean, Optional<TopicPolicies>>> policiesFutureHolder = new MutableObject<>();
585 // NOTICE: avoid using any callback with lock scope to avoid deadlock
586 policyCacheInitMap.compute(namespace, (___, existingFuture) -> {
587 if (!inserted || existingFuture != null) {
588 final var partitionedTopicName = TopicName.get(topicName.getPartitionedTopicName());
589 final var policies = Optional.ofNullable(switch (type) {
590 case GLOBAL_ONLY -> globalPoliciesCache.get(partitionedTopicName);
591 case LOCAL_ONLY -> policiesCache.get(partitionedTopicName);
592 });
593 policiesFutureHolder.setValue(Pair.of(true, policies));
594 } else {
595 policiesFutureHolder.setValue(Pair.of(false, null));
596 }
597 return existingFuture;
598 });
599 final var p = policiesFutureHolder.get();
600 if (!p.getLeft()) {
601 log.info()
602 .attr("namespace", namespace)
603 .log("The future of has been removed from cache, retry getTopicPolicies again");
604 return getTopicPoliciesAsync(topicName, type);
605 }
606 return CompletableFuture.completedFuture(p.getRight());
```
- 580: `prepareInitPoliciesCacheAsync(namespace)` is a real await. `policyCacheInitMap` holds one `CompletableFuture<Void>` per namespace -- one "generation" of that namespace's policy cache -- installed with `putIfAbsent` (line 649) and completed once the `__change_events` reader has drained (line 667). The `inserted` it resolves to is `true` both when this call installed the future (line 697) and when it joined an existing one (line 699); it is `false` only when the service is closed (line 636) or the namespace is missing or deleted (line 644).
- 581-582: `thenComposeAsync` with no executor hands the continuation to `CompletableFuture`'s default async executor (the common `ForkJoinPool` when its parallelism is above 1, otherwise a new thread per task), so it runs on another thread after `preparedFuture` completes. Between the completion of `preparedFuture` and the execution of the lambda, `policyCacheInitMap` can change.
- 586-587: the guarantee that was just awaited is re-derived from the map, and the re-derivation is weaker than the await: `existingFuture != null` asks whether *some* init future is present, never whether it is complete. A future installed a moment ago that has not read a single event yet passes.
- 588-593: the caches are read and the result is returned as final (line 606).
- 594-596 and 599-604: the retry branch, taken only when the map has *no* entry for the namespace.
#### The interleaving
1. A topic load calls `getTopicPoliciesAsync(topic, LOCAL_ONLY)`. Generation N of the namespace is complete, so `preparedFuture` completes with `inserted == true` and the continuation of line 582 is queued on the common pool.
2. Before it runs, the namespace's last bundle is unloaded: `removeOwnedNamespaceBundleAsync` (line 724) calls `cleanPoliciesCacheInitMap(namespace)` (line 935), which, inside `policyCacheInitMap.compute`, removes every `policiesCache` and `globalPoliciesCache` entry of the namespace and returns `null`, i.e. removes the map entry (lines 944-950).
3. The bundle is loaded again: `addOwnedNamespaceBundleAsync` (line 610) calls `prepareInitPoliciesCacheAsync(namespace)`, which `putIfAbsent`s a new, incomplete future -- generation N+1 -- and starts a new reader (lines 647-658).
4. The queued continuation runs. `inserted` is `true` and `existingFuture` is generation N+1, non-null, so line 587 takes the read branch on a cache that was wiped in step 2 and not yet repopulated.
5. `Optional.empty()` is returned as a final answer for a topic that has policies. No exception, no log line.
Steps 2 and 3 are the ordinary bundle ownership callbacks registered in `start()` (lines 740-752); the defect is only their ordering against a queued continuation.
A wipe *without* a new generation is harmless: with the map entry gone, `existingFuture == null` selects the retry branch, `getTopicPoliciesAsync` calls `prepareInitPoliciesCacheAsync` again, installs and awaits a fresh generation and reads a populated cache. It is the *newly installed, still-loading* generation that turns a retry into a silent empty read. The injection seam used by the tests therefore replays exactly the two production calls of a bounce -- `cleanPoliciesCacheInitMap`, then `prepareInitPoliciesCacheAsync` -- in that window, and nothing else.
The other done-checks in this file all ask for "present AND complete": `existingFuture == null || existingFuture.isDone()` (line 250), `chainedFuture != null && chainedFuture.isDone()` (line 301), `initFuture != null && !initFuture.isDone()` (line 972). Line 587 is the outlier. Before PIP-376 (#23319) this read was guarded by `initialized == null || !initialized.isDone()` (in the lines that commit removed); #23319 replaced it with the current predicate, evaluated synchronously in a `thenAccept`, and #23786 moved the continuation behind the executor-less `thenComposeAsync` hop ("switch thread to avoid potential metadata thread cost and recursive deadlock"), which widened the gap between the generation's completion and the `compute` from a few instructions into a thread-pool hand-off.
#### Consequence: the retention is burned in and the repair is lost
`BrokerService.getManagedLedgerConfig` (line 2405; policy resolution at lines 2434-2475) reads the topic policies with `GetType.LOCAL_ONLY` (line 2418) and `GetType.GLOBAL_ONLY` (line 2422) through `getTopicPoliciesBypassSystemTopic`, which delegates to `getTopicPoliciesAsync` (line 1494). An `Optional.empty()` there is indistinguishable from "this topic has no retention policy": `retentionPolicies` stays `null` (lines 2446-2450), the namespace `retention_policies` -- or the broker defaults -- are taken instead (lines 2465-2475) and written into the `ManagedLedgerConfig` with `setRetentionTime` / `setRetentionSizeInMB` (lines 2557-2559) before the managed ledger is opened. A *failed* read, by contrast, fails the topic load loudly; the remedies discussed in #26137 key off that failure signal and cannot see this silent empty.
`AbstractTopic.initTopicPolicy()` (lines 629-670) reads the same policies again (`GLOBAL_ONLY` at lines 648-649, `LOCAL_ONLY` at lines 652-653) and hands them to `TopicPolicyListenerWrapper.completeInitialization(global.orElse(null), local.orElse(null))` (lines 657-658). `TopicPolicyListenerWrapper.emitInitialPolicies` (lines 145-152) emits nothing when the loaded value is `null` and nothing was received during initialization, so an empty read there repairs nothing. One topic load performs five policy reads in total. `BrokerService.getTopic` (lines 1379-1382) issues the first one, a `LOCAL_ONLY` read whose result is discarded, purely to await the namespace's policy-cache initialization before the load proceeds; `getManagedLedgerConfig` then reads twice and `initTopicPolicy()` twice more. The retention is wrong if the `getManagedLedgerConfig` pair is stale, and it stays wrong if the `initTopicPolicy()` pair is stale too.
With `topicPolicyListenerReplayEnabled=false` (the default since #26134, `ServiceConfiguration` line 2057) no later broadcast re-notifies the topic once generation N+1 has finished loading, so the wrong retention persists for the life of the topic instance while `admin.topicPolicies().getRetention(topic, true)` keeps returning the topic value. Data is trimmed to the namespace retention.
#### Related changes
- #20763 accepted "get topic policies as null during clean cache" as a bug and introduced the `compute` blocks; #24980 made `prepareInitPoliciesCacheAsync` genuinely await the initialization; #26132 identity-guards the *cleanup* side of a concurrent reload. None of them checks whether the future found after the hop is complete.
- #26137 (open) is the failure-path sibling ("topic policy loading errors are ignored which can result in data loss"); #26259 touches `PersistentTopic.initialize()`, which runs after `getManagedLedgerConfig` has already decided the retention, so it does not cover this path.
- The same predicate, behind the same executor-less hop, is present on `branch-4.0`, `branch-4.2` and `branch-5.0-M1`.
### Modifications
1. `SystemTopicBasedTopicPoliciesService.getTopicPoliciesAsync`: read the caches only when the init future currently owning the namespace has completed successfully.
```diff
- if (!inserted || existingFuture != null) {
+ if (!inserted || (existingFuture != null && existingFuture.isDone()
+ && !existingFuture.isCompletedExceptionally())) {
```
(Excerpt of the hunk: the six comment lines the patch adds above the predicate are omitted here.)
A future still loading, or one already dropped from the map, falls through to the existing retry branch, which re-awaits through `prepareInitPoliciesCacheAsync` before reading again. A comment above the predicate explains why `isDone()` is required (the bounce-inside-the-hop interleaving). `prepareInitPoliciesCacheAsync`, the `!inserted` short-circuit (service closed, or namespace being deleted -- unchanged behaviour: it still answers from the caches as they are) and the retry branch itself are untouched.
One narrow behaviour change is worth naming: a generation that completed *exceptionally* is now routed to the retry as well, and that retry re-enters `prepareInitPoliciesCacheAsync`, whose `existingFuture.thenApply(__ -> true)` (line 699) propagates the failure to the caller. The window is short -- between `initNamespacePolicyFuture.completeExceptionally(ex)` (lines 680/683, and the timeout path at line 778) and the identity-guarded `policyCacheInitMap.remove(namespace, initFuture)` inside `cleanupFailedPolicyCacheInit` (line 826) -- and a read landing in it previously returned the still-populated cache, since that cleanup deliberately leaves cached policies in place (lines 812-814). Failing the topic load is the safe direction here: it is the signal #26137 is about, and it is loud, where the previous behaviour was to serve a cache nobody was awaiting.
2. The retry log line. `"The future of has been removed from cache, retry getTopicPolicies again"` lost its subject when the `{}` placeholder was dropped in the slog migration (#25535), and "removed" describes only one of the three cases the branch now covers. It now reads `"Policy cache init future is missing, failed or not yet complete, retry getTopicPolicies again"`, same level (INFO, an existing line) and same structured `namespace` attribute. The level is kept deliberately: a bundle bounce is a low-frequency ownership event, and the line fires at most once per policy read in flight at that instant, since reads that start while the replacement generation is still loading await it instead of retrying.
3. Tests, all in `pulsar-broker/src/test/java/org/apache/pulsar/broker/service/`:
- `StaleCacheGenerationInjectingTopicPoliciesService` (new, package-private helper): a `SystemTopicBasedTopicPoliciesService` whose `prepareInitPoliciesCacheAsync` override, while armed for a namespace, lets the production method complete and then replays a bundle bounce -- `cleanPoliciesCacheInitMap(namespace)` followed by `super.prepareInitPoliciesCacheAsync(namespace)` fired and forgotten, as `addOwnedNamespaceBundleAsync` does -- waits off-thread (polling on the broker's scheduled executor, never blocking the caller's thread) until a generation owns the namespace again, and only then hands back the original `inserted`. The bounce therefore runs before the future the caller awaits completes, so the continuation of `getTopicPoliciesAsync` is guaranteed, not merely likely, to resume onto a replaced, still-loading generation; the reproduction needs no sleep and no timing tuning. It fires only when the awaited generation is complete and non-exceptional, which is the production p
recondition, one bounce at a time (two overlapping bounces would drop and fail each other's still-loading replacement, aborting an unrelated in-flight read), and at most `budget` times (`arm(namespace, budget)` / `disarm()`), so a broker that retries its way out of the stale read converges. `staleInjectionCount()` counts only the bounces actually handed back in the state under test -- replacement generation installed and still loading -- so neither test can pass or fail vacuously. No cache is edited by hand, nothing is stubbed and no failure is injected: the seam only chooses when two production methods run.
- `SystemTopicBasedTopicPoliciesServiceTest#testGetTopicPoliciesWhenBundleBounceReplacesPolicyCacheGeneration` (new method in the existing class): the direct regression test on `getTopicPoliciesAsync`.
- `TopicPoliciesStaleCacheGenerationRetentionTest` (new class): the end-to-end effect on the live `ManagedLedgerConfig`. It extends `MockedPulsarServiceBaseTest` rather than `SharedPulsarBaseTest` because it closes and replaces the broker's `TopicPoliciesService`, which would leak into every other class sharing a runtime.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- `SystemTopicBasedTopicPoliciesServiceTest#testGetTopicPoliciesWhenBundleBounceReplacesPolicyCacheGeneration` closes the current service and installs the injecting one with `FieldUtils.writeField(pulsar, "topicPoliciesService", ...)`, as the existing tests of that class do, and additionally calls `start(pulsar)` so the replacement receives bundle-ownership callbacks like the production service. It creates a topic with a local policy, waits until generation 1 of the namespace is complete and the policy is visible in the cache, arms four bounces, issues a single `getTopicPoliciesAsync(topic, LOCAL_ONLY)`, and asserts that at least one bounce was handed back a still-loading replacement generation (so the next assertions cannot hold vacuously), that the result is present, and that it carries the policy that was set.
- On unpatched master it fails with:
```
java.lang.AssertionError: [getTopicPoliciesAsync returned Optional.empty() for a topic that has a policy: a namespace-bundle bounce replaced the namespace's policy-cache generation while the read was hopping threads, and the read served the wiped cache of the new, still-loading generation instead of retrying]
Expecting Optional to contain a value but it was empty.
```
- With the fix it passes, and the test output contains the retry line `Policy cache init future is missing, failed or not yet complete, retry getTopicPolicies again` four times -- once per injected bounce, i.e. every bounce became a retry instead of an empty read. The whole class (20 tests) passes with the fix.
- `TopicPoliciesStaleCacheGenerationRetentionTest#testTopicRetentionSurvivesPolicyCacheGenerationReplacementDuringLoad` runs a broker with `topicLevelPoliciesEnabled=true`, `systemTopicEnabled=true`, `defaultNumberOfNamespaceBundles=1`, `defaultRetentionTimeInMinutes=0` and `defaultRetentionSizeInMB=0`, a namespace retention of 30 min and a topic-level retention of 300 min (synthetic values; only which of the two ends up live matters). A control load asserts that the live `topic.getManagedLedger().getConfig().getRetentionTimeMillis()` equals 300 min. The topic is then unloaded (waiting until `getTopicReference` is empty), the seam is armed with a budget of 8, the topic is loaded again and the seam is disarmed. The test asserts that at least one bounce was handed back a still-loading replacement generation, that both the `PersistentTopic` and the `ManagedLedger` instance differ from the control leg (`ManagedLedgerFactoryImpl` caches managed ledgers by name and ignores the config pa
ssed on a hit, so a reused instance would be asserting on the control leg's config), that the policy store still reports 300 min, and -- inside a bounded 10 s Awaitility, so that an out-of-band repair would still turn it green -- that the live retention equals 300 min.
- On unpatched master it fails with:
```
org.awaitility.core.ConditionTimeoutException: Assertion condition defined as a Lambda expression in org.apache.pulsar.broker.service.TopicPoliciesStaleCacheGenerationRetentionTest [the topic-policy reads made stale by the namespace-bundle bounces returned Optional.empty(), so the topic silently fell back to namespace retention: the live ManagedLedgerConfig enforces the namespace value of 30 minutes instead of the topic value of 300 minutes]
expected: 18000000L
but was: 1800000L within 10 seconds.
```
(1 800 000 ms is the 30 min namespace value; 18 000 000 ms is the 300 min topic value.)
- With the fix it passes, and the retry line appears eight times -- the bounce budget. All eight land on the preliminary `LOCAL_ONLY` read in `BrokerService.getTopic`, which is where the load spends the whole budget: the retries are sequential, since each one re-enters the seam, so the first read consumes it all. The `Loaded topic` latency breakdown of the armed load shows it -- its first `topic policies` stage takes `534 ms` against `902 us` on the control load, while its later `topic policies` stages stay at `2 ms`, `1 ms` and `13 ms` -- so the four retention-deciding reads run against a settled generation and the topic keeps its 300 min retention. The predicate itself is exercised directly by the unit test above.
Both tests fail on the unpatched code for the real reason (the empty read, the namespace retention), not because the seam forces internal state, and neither uses a sleep or a tuned delay: the unpatched failure only requires the replacement generation not to finish loading within the thread hop. In both, the vacuity assertion on `staleInjectionCount()` precedes the failing assertion and passes, so a run that never opened the window would fail there instead of reporting a false negative. `./gradlew :pulsar-broker:checkstyleMain :pulsar-broker:checkstyleTest` and `./gradlew quickCheck` pass on the change.
### Does this pull request potentially affect one of the following parts:
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
None of these apply: no configuration default changes, no public API or threading change; the fix only routes one more case into a retry path that already existed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't serve topic policies from a cache whose init future has not completed [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #26513:
URL: https://github.com/apache/pulsar/pull/26513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't serve topic policies from a cache whose init future has not completed [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #26513:
URL: https://github.com/apache/pulsar/pull/26513#discussion_r3972826187
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/StaleCacheGenerationInjectingTopicPoliciesService.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import java.time.Duration;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicName;
+import org.assertj.core.api.Assertions;
+import org.awaitility.Awaitility;
+
+/**
+ * A {@link SystemTopicBasedTopicPoliciesService} that, while armed, replays a namespace-bundle bounce before it
+ * completes the {@code prepareInitPoliciesCacheAsync(namespace)} future that
+ * {@link SystemTopicBasedTopicPoliciesService#getTopicPoliciesAsync} awaits. The continuation that re-derives that
+ * awaited guarantee from {@code policyCacheInitMap} after its thread hop is therefore guaranteed -- not merely
+ * likely -- to resume onto a replaced, still-loading generation, which is why the reproduction needs no sleep and no
+ * timing tuning. The bounce is two production calls and nothing else:
+ * {@code cleanPoliciesCacheInitMap(namespace)}, which drops the namespace's cached policies together with the future
+ * tracking their load, then {@code prepareInitPoliciesCacheAsync(namespace)}, which installs a new and still-loading
+ * one. No cache is edited by hand, nothing is stubbed and no failure is injected; only the moment at which those two
+ * production methods run is chosen.
+ *
+ * <p>A bounce fires only once the generation owning the namespace has completed successfully -- the production
+ * precondition being modelled -- one at a time, and at most as often as the budget passed to
+ * {@link #arm(NamespaceName, int)} allows, so a broker that retries its way out of the stale read converges instead
+ * of being perturbed for ever.
+ */
+class StaleCacheGenerationInjectingTopicPoliciesService extends SystemTopicBasedTopicPoliciesService {
+
+ private static final long GENERATION_INSTALL_TIMEOUT_MILLIS = 10_000;
+ private static final long GENERATION_INSTALL_POLL_MILLIS = 5;
+
+ private final PulsarService pulsar;
+ private final Object lock = new Object();
+ private final AtomicInteger staleInjections = new AtomicInteger();
+ private NamespaceName armedNamespace;
+ private int remainingBudget;
+ private boolean bounceInFlight;
+
+ StaleCacheGenerationInjectingTopicPoliciesService(PulsarService pulsar) {
+ super(pulsar);
+ this.pulsar = pulsar;
+ }
+
+ /**
+ * Starts interleaving a bundle bounce into the policy reads of {@code namespace}, at most {@code budget} times.
+ */
+ void arm(NamespaceName namespace, int budget) {
+ synchronized (lock) {
+ armedNamespace = namespace;
+ remainingBudget = budget;
+ }
+ }
+
+ /** Stops interleaving bounces; the service then behaves exactly like its superclass. */
+ void disarm() {
+ synchronized (lock) {
+ armedNamespace = null;
+ remainingBudget = 0;
+ }
+ }
+
+ /**
+ * How many bounces were handed back to the caller in the state under test: replacement generation installed and
+ * still loading. A bounce that misses that state leaves the resumed read correct even on unpatched code, so a
+ * test asserting on this counter cannot pass vacuously.
+ */
+ int staleInjectionCount() {
+ return staleInjections.get();
+ }
+
+ /**
+ * Waits until the namespace's first generation has finished loading {@code topicName}'s policy, which is the
+ * precondition every bounce needs. Until that policy is in the loaded cache, a read that resumes on a wiped cache
+ * is not distinguishable from a legitimate empty read, so arming earlier would race the policy write instead of
+ * testing the window.
+ */
+ void awaitGenerationLoaded(NamespaceName namespace, TopicName topicName) {
+ Awaitility.await().atMost(Duration.ofSeconds(60)).untilAsserted(() -> {
+ Assertions.assertThat(getPoliciesCacheInit(namespace))
+ .describedAs("the namespace's policy-cache generation has not loaded successfully yet")
+ .isCompleted();
+ Assertions.assertThat(TopicPolicyTestUtils.getLocalTopicPolicies(this, topicName))
+ .describedAs("the topic's policy is not in the loaded cache yet")
+ .isNotNull();
+ });
+ }
+
+ @Override
+ CompletableFuture<Boolean> prepareInitPoliciesCacheAsync(NamespaceName namespace) {
+ final CompletableFuture<Boolean> prepared = super.prepareInitPoliciesCacheAsync(namespace);
+ synchronized (lock) {
+ if (!namespace.equals(armedNamespace)) {
+ return prepared;
+ }
+ }
+ // Bounce only after the caller's guarantee has genuinely been satisfied, and hand back the value the
+ // production method computed: the caller still believes it awaited the generation it prepared, which is
+ // exactly the state this defect acts on.
+ return prepared.thenCompose(inserted -> bounceNamespaceBundle(namespace).thenApply(__ -> inserted));
+ }
+
+ /**
+ * Replays the two production calls of a bounce, then waits until a generation owns the namespace again so the
+ * caller never resumes into the (harmless) missing-generation case. One bounce at a time: a second bounce landing
+ * while a replacement is still loading would drop and fail that replacement, aborting an unrelated in-flight read.
+ * Both production calls run outside the monitor, since completing an init future runs the topic loads awaiting it.
+ */
+ private CompletableFuture<Void> bounceNamespaceBundle(NamespaceName namespace) {
+ final boolean bounce;
+ synchronized (lock) {
+ final CompletableFuture<Void> generation = getPoliciesCacheInit(namespace);
+ final boolean generationLoaded =
+ generation != null && generation.isDone() && !generation.isCompletedExceptionally();
+ bounce = !bounceInFlight && namespace.equals(armedNamespace) && remainingBudget > 0 && generationLoaded;
+ if (bounce) {
+ bounceInFlight = true;
+ remainingBudget--;
+ }
+ }
+ if (!bounce) {
+ return CompletableFuture.completedFuture(null);
+ }
+ cleanPoliciesCacheInitMap(namespace);
+ // Fire and forget, exactly like the bundle-load path: the caller is not awaiting this generation.
+ super.prepareInitPoliciesCacheAsync(namespace).exceptionally(ignored -> null);
+ return awaitGenerationInstalled(namespace,
+ System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(GENERATION_INSTALL_TIMEOUT_MILLIS))
Review Comment:
**[QUALITY] Hold the replacement reader until the stale-read decision is exercised**
This counter samples the replacement before the common-pool continuation in [`SystemTopicBasedTopicPoliciesService.java:582`](https://github.com/apache/pulsar/blob/5f687bd0b02f5172dddf148e6ea1debf2fdbc316/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java#L582) runs. The replacement reader is free to populate the cache and complete between this increment and that continuation, so the old implementation can return the correct policy with a positive counter. Conversely, if the replacement finishes before this check, the fixed implementation can fail the positive-counter assertion even though every read was correct. An incomplete future also does not mean the target policy has not already been read. Please gate the replacement reader before it reads that policy, observe the original read reach its decision or retry, and then release the gate. That would make the regression exercise the intended interleaving without depending on read
er speed or executor scheduling.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPoliciesStaleCacheGenerationRetentionTest.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import java.time.Duration;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.policies.data.ClusterData;
+import org.apache.pulsar.common.policies.data.RetentionPolicies;
+import org.apache.pulsar.common.policies.data.TenantInfoImpl;
+import org.assertj.core.api.Assertions;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+/**
+ * End-to-end effect of a stale topic-policy read on retention: a topic whose topic-level retention is longer than its
+ * namespace's must never end up enforcing the namespace value on its live {@code ManagedLedgerConfig}.
+ *
+ * <p>{@code BrokerService#getManagedLedgerConfig} reads the topic policies while building the config the managed
+ * ledger is opened with. An {@code Optional.empty()} there is indistinguishable from "this topic has no retention
+ * policy", so the namespace retention is used and burned into the config; a failed read, by contrast, fails the topic
+ * load loudly. {@code AbstractTopic#initTopicPolicy} reads the same policies again and would repair the config
+ * through the topic-policy listener, but an empty read there emits nothing, so nothing is repaired. Both reads happen
+ * within a single topic load, so a stale-read window that spans the whole load leaves the wrong retention in place
+ * for the life of the topic instance while the policy store keeps reporting the right one.
+ *
+ * <p>The window is opened by {@link StaleCacheGenerationInjectingTopicPoliciesService}, which interleaves
+ * namespace-bundle bounces (production calls only) into the thread hop inside
+ * {@link SystemTopicBasedTopicPoliciesService#getTopicPoliciesAsync}. The two retention values are synthetic
+ * (namespace 30 min, topic 300 min) and their magnitudes do not matter: what is asserted is which of the two is live
+ * on the managed ledger.
+ *
+ * <p>This test closes and replaces the broker's {@code topicPoliciesService}, which would leak into every other class
+ * sharing a runtime, so it runs its own broker instead of extending {@code SharedPulsarBaseTest}.
+ */
+@Test(groups = "broker")
+public class TopicPoliciesStaleCacheGenerationRetentionTest extends MockedPulsarServiceBaseTest {
+
+ private static final String TENANT = "stale-cache-generation";
+ private static final String NAMESPACE = TENANT + "/retention";
+ /** Synthetic namespace retention: the value the topic must not fall back to. */
+ private static final int NAMESPACE_RETENTION_MINUTES = 30;
+ /** Synthetic topic-level retention: the value the topic must keep. */
+ private static final int TOPIC_RETENTION_MINUTES = 300;
+ /** Capped so a broker that retries out of the stale read converges instead of being perturbed for ever. */
+ private static final int BOUNCE_BUDGET = 8;
+
+ private StaleCacheGenerationInjectingTopicPoliciesService injectingService;
+
+ @BeforeMethod(alwaysRun = true)
+ @Override
+ protected void setup() throws Exception {
+ // These are all defaults, made explicit because the test depends on them: system-topic-backed topic-level
+ // policies are the subject (with either flag off the topic-policies service is legitimately disabled and an
+ // empty read would not be anomalous), a single bundle keeps one policy-cache generation per namespace, and a
+ // broker-level fallback of "no retention" leaves only the namespace policy and the topic policy able to
+ // explain whatever ends up on the managed ledger.
+ conf.setTopicLevelPoliciesEnabled(true);
+ conf.setSystemTopicEnabled(true);
+ conf.setDefaultNumberOfNamespaceBundles(1);
+ conf.setDefaultRetentionTimeInMinutes(0);
+ conf.setDefaultRetentionSizeInMB(0);
+ super.internalSetup();
+
+ admin.clusters().createCluster("test",
+ ClusterData.builder().serviceUrl(pulsar.getWebServiceAddress()).build());
+ admin.tenants().createTenant(TENANT, new TenantInfoImpl(Set.of("role1"), Set.of("test")));
+ admin.namespaces().createNamespace(NAMESPACE, Set.of("test"));
+
+ // Substitute the topic-policies service the way PulsarService installs it, so the bundle bounce can be
+ // interleaved into the policy reads a topic load performs.
+ pulsar.getTopicPoliciesService().close();
+ injectingService = new StaleCacheGenerationInjectingTopicPoliciesService(pulsar);
+ FieldUtils.writeField(pulsar, "topicPoliciesService", injectingService, true);
Review Comment:
**[QUALITY] Replace the new private-field reflection with a typed test seam**
Nonblocking: this and [`SystemTopicBasedTopicPoliciesServiceTest.java:913`](https://github.com/apache/pulsar/blob/5f687bd0b02f5172dddf148e6ea1debf2fdbc316/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesServiceTest.java#L913) add reflective writes to PulsarService's private service field. CODING.md explicitly disallows new reflection into private state, even though older tests still use it. Please use a typed `@VisibleForTesting` injection seam so a field rename or type change is checked by the compiler instead of failing at runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't serve topic policies from a cache whose init future has not completed [pulsar]
Posted by "alexandrebrg (via GitHub)" <gi...@apache.org>.
alexandrebrg commented on code in PR #26513:
URL: https://github.com/apache/pulsar/pull/26513#discussion_r3981103908
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/StaleCacheGenerationInjectingTopicPoliciesService.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import java.time.Duration;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicName;
+import org.assertj.core.api.Assertions;
+import org.awaitility.Awaitility;
+
+/**
+ * A {@link SystemTopicBasedTopicPoliciesService} that, while armed, replays a namespace-bundle bounce before it
+ * completes the {@code prepareInitPoliciesCacheAsync(namespace)} future that
+ * {@link SystemTopicBasedTopicPoliciesService#getTopicPoliciesAsync} awaits. The continuation that re-derives that
+ * awaited guarantee from {@code policyCacheInitMap} after its thread hop is therefore guaranteed -- not merely
+ * likely -- to resume onto a replaced, still-loading generation, which is why the reproduction needs no sleep and no
+ * timing tuning. The bounce is two production calls and nothing else:
+ * {@code cleanPoliciesCacheInitMap(namespace)}, which drops the namespace's cached policies together with the future
+ * tracking their load, then {@code prepareInitPoliciesCacheAsync(namespace)}, which installs a new and still-loading
+ * one. No cache is edited by hand, nothing is stubbed and no failure is injected; only the moment at which those two
+ * production methods run is chosen.
+ *
+ * <p>A bounce fires only once the generation owning the namespace has completed successfully -- the production
+ * precondition being modelled -- one at a time, and at most as often as the budget passed to
+ * {@link #arm(NamespaceName, int)} allows, so a broker that retries its way out of the stale read converges instead
+ * of being perturbed for ever.
+ */
+class StaleCacheGenerationInjectingTopicPoliciesService extends SystemTopicBasedTopicPoliciesService {
+
+ private static final long GENERATION_INSTALL_TIMEOUT_MILLIS = 10_000;
+ private static final long GENERATION_INSTALL_POLL_MILLIS = 5;
+
+ private final PulsarService pulsar;
+ private final Object lock = new Object();
+ private final AtomicInteger staleInjections = new AtomicInteger();
+ private NamespaceName armedNamespace;
+ private int remainingBudget;
+ private boolean bounceInFlight;
+
+ StaleCacheGenerationInjectingTopicPoliciesService(PulsarService pulsar) {
+ super(pulsar);
+ this.pulsar = pulsar;
+ }
+
+ /**
+ * Starts interleaving a bundle bounce into the policy reads of {@code namespace}, at most {@code budget} times.
+ */
+ void arm(NamespaceName namespace, int budget) {
+ synchronized (lock) {
+ armedNamespace = namespace;
+ remainingBudget = budget;
+ }
+ }
+
+ /** Stops interleaving bounces; the service then behaves exactly like its superclass. */
+ void disarm() {
+ synchronized (lock) {
+ armedNamespace = null;
+ remainingBudget = 0;
+ }
+ }
+
+ /**
+ * How many bounces were handed back to the caller in the state under test: replacement generation installed and
+ * still loading. A bounce that misses that state leaves the resumed read correct even on unpatched code, so a
+ * test asserting on this counter cannot pass vacuously.
+ */
+ int staleInjectionCount() {
+ return staleInjections.get();
+ }
+
+ /**
+ * Waits until the namespace's first generation has finished loading {@code topicName}'s policy, which is the
+ * precondition every bounce needs. Until that policy is in the loaded cache, a read that resumes on a wiped cache
+ * is not distinguishable from a legitimate empty read, so arming earlier would race the policy write instead of
+ * testing the window.
+ */
+ void awaitGenerationLoaded(NamespaceName namespace, TopicName topicName) {
+ Awaitility.await().atMost(Duration.ofSeconds(60)).untilAsserted(() -> {
+ Assertions.assertThat(getPoliciesCacheInit(namespace))
+ .describedAs("the namespace's policy-cache generation has not loaded successfully yet")
+ .isCompleted();
+ Assertions.assertThat(TopicPolicyTestUtils.getLocalTopicPolicies(this, topicName))
+ .describedAs("the topic's policy is not in the loaded cache yet")
+ .isNotNull();
+ });
+ }
+
+ @Override
+ CompletableFuture<Boolean> prepareInitPoliciesCacheAsync(NamespaceName namespace) {
+ final CompletableFuture<Boolean> prepared = super.prepareInitPoliciesCacheAsync(namespace);
+ synchronized (lock) {
+ if (!namespace.equals(armedNamespace)) {
+ return prepared;
+ }
+ }
+ // Bounce only after the caller's guarantee has genuinely been satisfied, and hand back the value the
+ // production method computed: the caller still believes it awaited the generation it prepared, which is
+ // exactly the state this defect acts on.
+ return prepared.thenCompose(inserted -> bounceNamespaceBundle(namespace).thenApply(__ -> inserted));
+ }
+
+ /**
+ * Replays the two production calls of a bounce, then waits until a generation owns the namespace again so the
+ * caller never resumes into the (harmless) missing-generation case. One bounce at a time: a second bounce landing
+ * while a replacement is still loading would drop and fail that replacement, aborting an unrelated in-flight read.
+ * Both production calls run outside the monitor, since completing an init future runs the topic loads awaiting it.
+ */
+ private CompletableFuture<Void> bounceNamespaceBundle(NamespaceName namespace) {
+ final boolean bounce;
+ synchronized (lock) {
+ final CompletableFuture<Void> generation = getPoliciesCacheInit(namespace);
+ final boolean generationLoaded =
+ generation != null && generation.isDone() && !generation.isCompletedExceptionally();
+ bounce = !bounceInFlight && namespace.equals(armedNamespace) && remainingBudget > 0 && generationLoaded;
+ if (bounce) {
+ bounceInFlight = true;
+ remainingBudget--;
+ }
+ }
+ if (!bounce) {
+ return CompletableFuture.completedFuture(null);
+ }
+ cleanPoliciesCacheInitMap(namespace);
+ // Fire and forget, exactly like the bundle-load path: the caller is not awaiting this generation.
+ super.prepareInitPoliciesCacheAsync(namespace).exceptionally(ignored -> null);
+ return awaitGenerationInstalled(namespace,
+ System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(GENERATION_INSTALL_TIMEOUT_MILLIS))
Review Comment:
Done in 6f142a8e50: `createSystemTopicClient` wraps the replacement generation's reader so `hasMoreEventsAsync()` and `readNextAsync()` return nothing until a gate is released, and `prepareInitPoliciesCacheAsync` holds the read until that replacement is in `policyCacheInitMap` with its reader gated.
A sibling read resuming from the same generation completion joins that window, or re-awaits the replacement generation and opens its own, so none runs past a window unheld.
The gate is released only once every read the window holds has been observed to retry -- the re-entrant `getTopicPoliciesAsync` call with the same topic and `GetType` -- or to complete, so nothing depends on reader speed or scheduling and the counters count decisions, not samples.
Unpatched: 1 window / 0 retries / 1 read served in the unit test, and 4 windows over all 7 policy reads of the load / 7 served / 0 retries end-to-end, with the live retention at 30 min instead of 300; with the fix, 4/4/0 and 8/8/0, identical over two runs per leg.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPoliciesStaleCacheGenerationRetentionTest.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import java.time.Duration;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.policies.data.ClusterData;
+import org.apache.pulsar.common.policies.data.RetentionPolicies;
+import org.apache.pulsar.common.policies.data.TenantInfoImpl;
+import org.assertj.core.api.Assertions;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+/**
+ * End-to-end effect of a stale topic-policy read on retention: a topic whose topic-level retention is longer than its
+ * namespace's must never end up enforcing the namespace value on its live {@code ManagedLedgerConfig}.
+ *
+ * <p>{@code BrokerService#getManagedLedgerConfig} reads the topic policies while building the config the managed
+ * ledger is opened with. An {@code Optional.empty()} there is indistinguishable from "this topic has no retention
+ * policy", so the namespace retention is used and burned into the config; a failed read, by contrast, fails the topic
+ * load loudly. {@code AbstractTopic#initTopicPolicy} reads the same policies again and would repair the config
+ * through the topic-policy listener, but an empty read there emits nothing, so nothing is repaired. Both reads happen
+ * within a single topic load, so a stale-read window that spans the whole load leaves the wrong retention in place
+ * for the life of the topic instance while the policy store keeps reporting the right one.
+ *
+ * <p>The window is opened by {@link StaleCacheGenerationInjectingTopicPoliciesService}, which interleaves
+ * namespace-bundle bounces (production calls only) into the thread hop inside
+ * {@link SystemTopicBasedTopicPoliciesService#getTopicPoliciesAsync}. The two retention values are synthetic
+ * (namespace 30 min, topic 300 min) and their magnitudes do not matter: what is asserted is which of the two is live
+ * on the managed ledger.
+ *
+ * <p>This test closes and replaces the broker's {@code topicPoliciesService}, which would leak into every other class
+ * sharing a runtime, so it runs its own broker instead of extending {@code SharedPulsarBaseTest}.
+ */
+@Test(groups = "broker")
+public class TopicPoliciesStaleCacheGenerationRetentionTest extends MockedPulsarServiceBaseTest {
+
+ private static final String TENANT = "stale-cache-generation";
+ private static final String NAMESPACE = TENANT + "/retention";
+ /** Synthetic namespace retention: the value the topic must not fall back to. */
+ private static final int NAMESPACE_RETENTION_MINUTES = 30;
+ /** Synthetic topic-level retention: the value the topic must keep. */
+ private static final int TOPIC_RETENTION_MINUTES = 300;
+ /** Capped so a broker that retries out of the stale read converges instead of being perturbed for ever. */
+ private static final int BOUNCE_BUDGET = 8;
+
+ private StaleCacheGenerationInjectingTopicPoliciesService injectingService;
+
+ @BeforeMethod(alwaysRun = true)
+ @Override
+ protected void setup() throws Exception {
+ // These are all defaults, made explicit because the test depends on them: system-topic-backed topic-level
+ // policies are the subject (with either flag off the topic-policies service is legitimately disabled and an
+ // empty read would not be anomalous), a single bundle keeps one policy-cache generation per namespace, and a
+ // broker-level fallback of "no retention" leaves only the namespace policy and the topic policy able to
+ // explain whatever ends up on the managed ledger.
+ conf.setTopicLevelPoliciesEnabled(true);
+ conf.setSystemTopicEnabled(true);
+ conf.setDefaultNumberOfNamespaceBundles(1);
+ conf.setDefaultRetentionTimeInMinutes(0);
+ conf.setDefaultRetentionSizeInMB(0);
+ super.internalSetup();
+
+ admin.clusters().createCluster("test",
+ ClusterData.builder().serviceUrl(pulsar.getWebServiceAddress()).build());
+ admin.tenants().createTenant(TENANT, new TenantInfoImpl(Set.of("role1"), Set.of("test")));
+ admin.namespaces().createNamespace(NAMESPACE, Set.of("test"));
+
+ // Substitute the topic-policies service the way PulsarService installs it, so the bundle bounce can be
+ // interleaved into the policy reads a topic load performs.
+ pulsar.getTopicPoliciesService().close();
+ injectingService = new StaleCacheGenerationInjectingTopicPoliciesService(pulsar);
+ FieldUtils.writeField(pulsar, "topicPoliciesService", injectingService, true);
Review Comment:
Done in 6f142a8e50: `PulsarService` now has a typed `@VisibleForTesting public void setTopicPoliciesService(TopicPoliciesService)` next to `setTransactionBufferProvider`, and both tests call it instead of `FieldUtils.writeField`. The class-level `@Setter(AccessLevel.PROTECTED)` is why the generated setter was not usable from the tests; the explicit method makes Lombok skip its protected one and turns a rename or a type change into a compile error. This PR now adds no reflection at all; the pre-existing reflective writes elsewhere in `SystemTopicBasedTopicPoliciesServiceTest` are untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't serve topic policies from a cache whose init future has not completed [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #26513:
URL: https://github.com/apache/pulsar/pull/26513#discussion_r3982759998
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/StaleCacheGenerationInjectingTopicPoliciesService.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import java.time.Duration;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicName;
+import org.assertj.core.api.Assertions;
+import org.awaitility.Awaitility;
+
+/**
+ * A {@link SystemTopicBasedTopicPoliciesService} that, while armed, replays a namespace-bundle bounce before it
+ * completes the {@code prepareInitPoliciesCacheAsync(namespace)} future that
+ * {@link SystemTopicBasedTopicPoliciesService#getTopicPoliciesAsync} awaits. The continuation that re-derives that
+ * awaited guarantee from {@code policyCacheInitMap} after its thread hop is therefore guaranteed -- not merely
+ * likely -- to resume onto a replaced, still-loading generation, which is why the reproduction needs no sleep and no
+ * timing tuning. The bounce is two production calls and nothing else:
+ * {@code cleanPoliciesCacheInitMap(namespace)}, which drops the namespace's cached policies together with the future
+ * tracking their load, then {@code prepareInitPoliciesCacheAsync(namespace)}, which installs a new and still-loading
+ * one. No cache is edited by hand, nothing is stubbed and no failure is injected; only the moment at which those two
+ * production methods run is chosen.
+ *
+ * <p>A bounce fires only once the generation owning the namespace has completed successfully -- the production
+ * precondition being modelled -- one at a time, and at most as often as the budget passed to
+ * {@link #arm(NamespaceName, int)} allows, so a broker that retries its way out of the stale read converges instead
+ * of being perturbed for ever.
+ */
+class StaleCacheGenerationInjectingTopicPoliciesService extends SystemTopicBasedTopicPoliciesService {
+
+ private static final long GENERATION_INSTALL_TIMEOUT_MILLIS = 10_000;
+ private static final long GENERATION_INSTALL_POLL_MILLIS = 5;
+
+ private final PulsarService pulsar;
+ private final Object lock = new Object();
+ private final AtomicInteger staleInjections = new AtomicInteger();
+ private NamespaceName armedNamespace;
+ private int remainingBudget;
+ private boolean bounceInFlight;
+
+ StaleCacheGenerationInjectingTopicPoliciesService(PulsarService pulsar) {
+ super(pulsar);
+ this.pulsar = pulsar;
+ }
+
+ /**
+ * Starts interleaving a bundle bounce into the policy reads of {@code namespace}, at most {@code budget} times.
+ */
+ void arm(NamespaceName namespace, int budget) {
+ synchronized (lock) {
+ armedNamespace = namespace;
+ remainingBudget = budget;
+ }
+ }
+
+ /** Stops interleaving bounces; the service then behaves exactly like its superclass. */
+ void disarm() {
+ synchronized (lock) {
+ armedNamespace = null;
+ remainingBudget = 0;
+ }
+ }
+
+ /**
+ * How many bounces were handed back to the caller in the state under test: replacement generation installed and
+ * still loading. A bounce that misses that state leaves the resumed read correct even on unpatched code, so a
+ * test asserting on this counter cannot pass vacuously.
+ */
+ int staleInjectionCount() {
+ return staleInjections.get();
+ }
+
+ /**
+ * Waits until the namespace's first generation has finished loading {@code topicName}'s policy, which is the
+ * precondition every bounce needs. Until that policy is in the loaded cache, a read that resumes on a wiped cache
+ * is not distinguishable from a legitimate empty read, so arming earlier would race the policy write instead of
+ * testing the window.
+ */
+ void awaitGenerationLoaded(NamespaceName namespace, TopicName topicName) {
+ Awaitility.await().atMost(Duration.ofSeconds(60)).untilAsserted(() -> {
+ Assertions.assertThat(getPoliciesCacheInit(namespace))
+ .describedAs("the namespace's policy-cache generation has not loaded successfully yet")
+ .isCompleted();
+ Assertions.assertThat(TopicPolicyTestUtils.getLocalTopicPolicies(this, topicName))
+ .describedAs("the topic's policy is not in the loaded cache yet")
+ .isNotNull();
+ });
+ }
+
+ @Override
+ CompletableFuture<Boolean> prepareInitPoliciesCacheAsync(NamespaceName namespace) {
+ final CompletableFuture<Boolean> prepared = super.prepareInitPoliciesCacheAsync(namespace);
+ synchronized (lock) {
+ if (!namespace.equals(armedNamespace)) {
+ return prepared;
+ }
+ }
+ // Bounce only after the caller's guarantee has genuinely been satisfied, and hand back the value the
+ // production method computed: the caller still believes it awaited the generation it prepared, which is
+ // exactly the state this defect acts on.
+ return prepared.thenCompose(inserted -> bounceNamespaceBundle(namespace).thenApply(__ -> inserted));
+ }
+
+ /**
+ * Replays the two production calls of a bounce, then waits until a generation owns the namespace again so the
+ * caller never resumes into the (harmless) missing-generation case. One bounce at a time: a second bounce landing
+ * while a replacement is still loading would drop and fail that replacement, aborting an unrelated in-flight read.
+ * Both production calls run outside the monitor, since completing an init future runs the topic loads awaiting it.
+ */
+ private CompletableFuture<Void> bounceNamespaceBundle(NamespaceName namespace) {
+ final boolean bounce;
+ synchronized (lock) {
+ final CompletableFuture<Void> generation = getPoliciesCacheInit(namespace);
+ final boolean generationLoaded =
+ generation != null && generation.isDone() && !generation.isCompletedExceptionally();
+ bounce = !bounceInFlight && namespace.equals(armedNamespace) && remainingBudget > 0 && generationLoaded;
+ if (bounce) {
+ bounceInFlight = true;
+ remainingBudget--;
+ }
+ }
+ if (!bounce) {
+ return CompletableFuture.completedFuture(null);
+ }
+ cleanPoliciesCacheInitMap(namespace);
+ // Fire and forget, exactly like the bundle-load path: the caller is not awaiting this generation.
+ super.prepareInitPoliciesCacheAsync(namespace).exceptionally(ignored -> null);
+ return awaitGenerationInstalled(namespace,
+ System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(GENERATION_INSTALL_TIMEOUT_MILLIS))
Review Comment:
The reader gate now remains closed until every held read has retried or completed, and delayed sibling reads re-await the replacement. That addresses the scheduling gap.
https://github.com/apache/pulsar/blob/6f142a8e5064052a8868e116df18570ff703c618/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/StaleCacheGenerationInjectingTopicPoliciesService.java#L403-L408
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPoliciesStaleCacheGenerationRetentionTest.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import java.time.Duration;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.common.naming.NamespaceName;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.policies.data.ClusterData;
+import org.apache.pulsar.common.policies.data.RetentionPolicies;
+import org.apache.pulsar.common.policies.data.TenantInfoImpl;
+import org.assertj.core.api.Assertions;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+/**
+ * End-to-end effect of a stale topic-policy read on retention: a topic whose topic-level retention is longer than its
+ * namespace's must never end up enforcing the namespace value on its live {@code ManagedLedgerConfig}.
+ *
+ * <p>{@code BrokerService#getManagedLedgerConfig} reads the topic policies while building the config the managed
+ * ledger is opened with. An {@code Optional.empty()} there is indistinguishable from "this topic has no retention
+ * policy", so the namespace retention is used and burned into the config; a failed read, by contrast, fails the topic
+ * load loudly. {@code AbstractTopic#initTopicPolicy} reads the same policies again and would repair the config
+ * through the topic-policy listener, but an empty read there emits nothing, so nothing is repaired. Both reads happen
+ * within a single topic load, so a stale-read window that spans the whole load leaves the wrong retention in place
+ * for the life of the topic instance while the policy store keeps reporting the right one.
+ *
+ * <p>The window is opened by {@link StaleCacheGenerationInjectingTopicPoliciesService}, which interleaves
+ * namespace-bundle bounces (production calls only) into the thread hop inside
+ * {@link SystemTopicBasedTopicPoliciesService#getTopicPoliciesAsync}. The two retention values are synthetic
+ * (namespace 30 min, topic 300 min) and their magnitudes do not matter: what is asserted is which of the two is live
+ * on the managed ledger.
+ *
+ * <p>This test closes and replaces the broker's {@code topicPoliciesService}, which would leak into every other class
+ * sharing a runtime, so it runs its own broker instead of extending {@code SharedPulsarBaseTest}.
+ */
+@Test(groups = "broker")
+public class TopicPoliciesStaleCacheGenerationRetentionTest extends MockedPulsarServiceBaseTest {
+
+ private static final String TENANT = "stale-cache-generation";
+ private static final String NAMESPACE = TENANT + "/retention";
+ /** Synthetic namespace retention: the value the topic must not fall back to. */
+ private static final int NAMESPACE_RETENTION_MINUTES = 30;
+ /** Synthetic topic-level retention: the value the topic must keep. */
+ private static final int TOPIC_RETENTION_MINUTES = 300;
+ /** Capped so a broker that retries out of the stale read converges instead of being perturbed for ever. */
+ private static final int BOUNCE_BUDGET = 8;
+
+ private StaleCacheGenerationInjectingTopicPoliciesService injectingService;
+
+ @BeforeMethod(alwaysRun = true)
+ @Override
+ protected void setup() throws Exception {
+ // These are all defaults, made explicit because the test depends on them: system-topic-backed topic-level
+ // policies are the subject (with either flag off the topic-policies service is legitimately disabled and an
+ // empty read would not be anomalous), a single bundle keeps one policy-cache generation per namespace, and a
+ // broker-level fallback of "no retention" leaves only the namespace policy and the topic policy able to
+ // explain whatever ends up on the managed ledger.
+ conf.setTopicLevelPoliciesEnabled(true);
+ conf.setSystemTopicEnabled(true);
+ conf.setDefaultNumberOfNamespaceBundles(1);
+ conf.setDefaultRetentionTimeInMinutes(0);
+ conf.setDefaultRetentionSizeInMB(0);
+ super.internalSetup();
+
+ admin.clusters().createCluster("test",
+ ClusterData.builder().serviceUrl(pulsar.getWebServiceAddress()).build());
+ admin.tenants().createTenant(TENANT, new TenantInfoImpl(Set.of("role1"), Set.of("test")));
+ admin.namespaces().createNamespace(NAMESPACE, Set.of("test"));
+
+ // Substitute the topic-policies service the way PulsarService installs it, so the bundle bounce can be
+ // interleaved into the policy reads a topic load performs.
+ pulsar.getTopicPoliciesService().close();
+ injectingService = new StaleCacheGenerationInjectingTopicPoliciesService(pulsar);
+ FieldUtils.writeField(pulsar, "topicPoliciesService", injectingService, true);
Review Comment:
Both test call sites now use the typed setter, so the new reflective field writes are gone. This addresses the point.
https://github.com/apache/pulsar/blob/6f142a8e5064052a8868e116df18570ff703c618/pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java#L2386-L2389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't serve topic policies from a cache whose init future has not completed [pulsar]
Posted by "alexandrebrg (via GitHub)" <gi...@apache.org>.
alexandrebrg commented on PR #26513:
URL: https://github.com/apache/pulsar/pull/26513#issuecomment-5621751360
Addressed the review in 6f142a8e50. Both points addressed: the fixture now gates the replacement generation's reader and releases it only once every read the window holds has been observed to retry or to answer, and both tests install the service through a new typed `@VisibleForTesting PulsarService#setTopicPoliciesService` instead of reflection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org