You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2026/08/27 10:42:56 UTC

[PR] [fix][test] Fix flaky IsolatedBookieEnsemblePlacementPolicyTest by awaiting the initial rack config load [pulsar]

lhotari opened a new pull request, #26429:
URL: https://github.com/apache/pulsar/pull/26429

   ### Motivation
   
   `IsolatedBookieEnsemblePlacementPolicyTest` keeps flaking. It has already been patched five times, one test
   method at a time — #25473, #25474, #25496, #25640, #25900 — and it is still failing. The most recent failure
   is on `branch-4.2` CI, where `testNonRegionBookie` failed on both the initial run and the retry:
   
   ```
   [ERROR] IsolatedBookieEnsemblePlacementPolicyTest.testNonRegionBookie:129 expected [false] but found [true]
   ```
   
   i.e. `assertFalse(ensemble.contains(BOOKIE3))` — a bookie that is *not* in the isolation group ended up in the
   ensemble. `master` carries the identical code, so the same latent race is here.
   
   **Root cause.** `IsolatedBookieEnsemblePlacementPolicy.initialize()` kicks off a fire-and-forget load of the
   bookie rack configuration:
   
   ```java
   bookieMappingCache.get(BOOKIE_INFO_ROOT_PATH).thenAccept(opt -> opt.ifPresent(
           brc -> cachedRackConfiguration = brc))
   ```
   
   and `getExcludedBookiesWithIsolationGroups()` degrades silently while that load is in flight:
   
   ```java
   BookiesRackConfiguration allGroupsBookieMapping = cachedRackConfiguration;
   if (allGroupsBookieMapping == null) {
       log.debug("The bookies rack config is not available at now.");
       return excludedBookies;   // empty -> nothing is excluded -> no isolation at all
   }
   ```
   
   Every test in the class calls `newEnsemble()` straight after `initialize()`. If the load has not completed,
   no bookies are excluded and the rack-aware policy is free to pick any of the four writable bookies. Since
   PIP-453 (#25187) moved metadata-store callbacks onto a dedicated executor, that window is wide enough to lose
   on a loaded CI runner.
   
   **The existing guards do not work.** Three tests carry this guard, added by the previous flake fixes:
   
   ```java
   Awaitility.await().atMost(Duration.ofSeconds(5)).untilAsserted(() ->
           assertNotNull(isolationPolicy.getBookieMappingCache()
                   .getIfCached(BookieRackAffinityMapping.BOOKIE_INFO_ROOT_PATH)));
   ```
   
   `MetadataCache.getIfCached` returns `Optional<T>`, which is never `null`, so the condition is a tautology that
   passes on the first poll. The guard only ever worked as an accidental `Thread.sleep` — Awaitility's default
   100 ms poll delay. The remaining 11 test methods have no guard at all.
   
   ### Modifications
   
   - `IsolatedBookieEnsemblePlacementPolicy` keeps a reference to the load chain it already creates, exposed as a
     `@VisibleForTesting` `initialRackConfigurationLoadFuture`. No behaviour change: the same chain is built, the
     same `exceptionally` handler applies, nothing new blocks or is retained.
   - `IsolatedBookieEnsemblePlacementPolicyTest` routes all 14 policy creations through one `createIsolationPolicy()`
     helper that initializes the policy and then joins that future. Waiting on the *composed* future is what makes
     this airtight: completing the underlying cache future and running the `thenAccept` that assigns
     `cachedRackConfiguration` are two separate steps, so waiting on the cache entry alone still races.
   - The three tautological `assertNotNull(...getIfCached(...))` guards are removed.
   
   Future tests added to this class get the wait for free, which is the point — the per-test patching has not
   converged in five attempts.
   
   Not addressed here: the same window exists in production. Between BK client init and the first successful rack
   config load, `getExcludedBookiesWithIsolationGroups()` applies no isolation at all and only says so at DEBUG
   level, so ledgers created in that window can land on bookies outside the configured isolation group. That is a
   behaviour change and belongs in its own PR.
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   This change is already covered by existing tests, such as `IsolatedBookieEnsemblePlacementPolicyTest`.
   
   Evidence that the diagnosis and the fix are correct, since the race is timing-dependent and does not reproduce
   on a fast developer machine:
   
   - With the metadata cache gated so the rack config load completes 300 ms late — exactly the CI condition — the
     un-waited path produced isolation violations in **17 of 20** rounds (the ensemble contained BOOKIE3/BOOKIE4);
     the waited path in **0 of 20**.
   - Full class with `@Test(invocationCount = 10)`: 140 tests, 0 failures.
   - `checkstyleMain`, `checkstyleTest` and `spotlessCheck` pass for `pulsar-broker-common`.
   
   ### Does this pull request potentially affect one of the following parts:
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][test] Fix flaky IsolatedBookieEnsemblePlacementPolicyTest by awaiting the initial rack config load [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #26429:
URL: https://github.com/apache/pulsar/pull/26429#issuecomment-5450967341

   @david-streamlio Settled it your way — thanks for chasing that one down. You were right that it was the last remaining path where "no isolation applied" could still arrive silently, wearing the same symptom as the race this PR exists to close.
   
   **Main point.** `initialize()` now captures the load *before* the `exceptionally` stage, and hands that to tests:
   
   ```java
   CompletableFuture<Void> rackConfigurationLoad = bookieMappingCache
           .get(BookieRackAffinityMapping.BOOKIE_INFO_ROOT_PATH).thenAccept(opt -> opt.ifPresent(
                   bookiesRackConfiguration -> cachedRackConfiguration = bookiesRackConfiguration));
   // Initialization continues when the load fails; isolation is simply not applied until a later refresh.
   rackConfigurationLoad.exceptionally(e -> {
       log.warn().exception(e)
               .log("Failed to load bookies rack configuration while initialize the PlacementPolicy.");
       return null;
   });
   initialRackConfigurationLoadFuture = rackConfigurationLoad;
   ```
   
   Exactly as you put it: the `exceptionally` stage still attaches to the same upstream, so production keeps its log-and-continue behaviour and only the test's view changes. One thing that came along for free — the WARN now carries the exception. It previously reported "failed" with no cause at all, which is part of why a failed load was so hard to tell apart from a slow one.
   
   **Your alternative I deliberately did not take**, and it seems worth saying why rather than just picking one: asserting in the helper that `cachedRackConfiguration` actually landed would fail two tests where `null` is the *correct* outcome. `testNoBookieInfo` runs with no `/bookies` node, so `opt.ifPresent(...)` never fires; `testNoIsolationGroup` never sets `isolationBookieGroups`, so the cache and the load are never created in the first place and the field stays at its pre-completed initializer. What the helper can honestly guarantee is "the load finished, one way or the other" — which is what the future now expresses.
   
   **Minor point taken as well.** The helper waits with `get(30, TimeUnit.SECONDS)` instead of `join()`. That sits comfortably inside the 5-minute per-test timeout `buildtools`' `AnnotationListener` stamps on every `@Test`, so a load that hangs fails with the helper's own timeout naming the wait, rather than the suite timeout naming nothing.
   
   **Both halves are pinned by a test.** `testInitialRackConfigurationLoadFailureIsReportedWithoutFailingInitialize` asserts that `initialize()` still returns normally on a failed load (the production behaviour you did not want changed), and that the exposed future completes exceptionally carrying that failure's cause. Run against the previous commit it fails with `AssertionError: Expecting code to raise a throwable` — precisely the "completes normally when the load fails" behaviour you described, so the test is anchored to your finding rather than to my restatement of it.
   
   A second small commit closes the real `memory:local` store from `setUp()` before that test replaces the field with a mock; otherwise `teardown()` closes the mock and the store's eagerly-started `OrderedExecutor` worker leaks for the life of the test JVM.
   
   Local state: 15/15 in the class, 10 consecutive full-class runs, `quickCheck` clean. CI running.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][test] Fix flaky IsolatedBookieEnsemblePlacementPolicyTest by awaiting the initial rack config load [pulsar]

Posted by "david-streamlio (via GitHub)" <gi...@apache.org>.
david-streamlio commented on PR #26429:
URL: https://github.com/apache/pulsar/pull/26429#issuecomment-5446656642

   @lhotari friendly ping on the one open point from my review above — this now looks like the only thing outstanding, since @merlimat has approved and CI is green apart from one still-running job.
   
   To restate it in a sentence: the captured `initialRackConfigurationLoadFuture` is the tail of the chain *including* `.exceptionally(...)`, so it completes normally when the load fails — `join()` returns, `cachedRackConfiguration` stays null, and the test fails with the exact symptom this PR fixes, from a different cause. Capturing the future before `.exceptionally`, or asserting in the helper that `cachedRackConfiguration` actually landed, closes that path with no change to production behaviour.
   
   No strong feelings about which form it takes, and if you think that failure path isn't worth closing, just say so and I'll approve as-is rather than hold the PR on it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][test] Fix flaky IsolatedBookieEnsemblePlacementPolicyTest by awaiting the initial rack config load [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #26429:
URL: https://github.com/apache/pulsar/pull/26429


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org