You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/10/12 21:55:43 UTC
[PR] [fix][broker] Don't leave a failed scalable topic controller or its leader election behind [pulsar]
merlimat opened a new pull request, #26877:
URL: https://github.com/apache/pulsar/pull/26877
### Motivation
`ScalableTopicService.getOrCreateController` creates a `ScalableTopicController` and calls `initialize()`. When `initialize()` fails, the failed future is evicted from the `controllers` map so that the next call can retry, but the controller itself is never closed. What it leaves behind depends on where `initialize()` failed.
**After the election.** `initialize()` reads the topic record, wins the election on the controller lock, schedules the leader's periodic tasks (the sealed-segment GC, and auto split/merge, which is enabled by default), and only then runs `ensureActiveSegmentsExist()` and `restoreSessionsFromStore()`. When `restoreSessionsFromStore()` hits a metadata error, the evicted controller keeps holding the controller lock and keeps its GC and auto split/merge tasks scheduled. Its `leaderState` stays `Leading`, so those tasks keep doing the leader's work.
The next `getOrCreateController` builds a second controller on the same cached `LeaderElectionImpl`. That election is still `Leading`, so `elect()` returns `Leading` at once, and both controllers run the leader's tasks. Releasing the second controller (topic delete, service close) stops only its own tasks. The first controller's tasks keep running until the broker restarts.
**Before the election.** The `ScalableTopicController` constructor calls `coordinationService.getLeaderElection(...)` before anything checks that the topic exists. `CoordinationServiceImpl` caches each election per path for the broker's lifetime, even after `asyncClose()`. Each `LeaderElectionImpl` registers a metadata store listener and a session listener, which are never removed (`MetadataStore` has no `unregisterListener`). So every distinct topic name a controller is created for permanently adds one election and two listeners to the broker:
- every metadata notification is dispatched to all of them;
- on every metadata session reconnect, each of them runs an election attempt: it reads its lock path and logs a `NullPointerException` warning, since it never had a value to propose.
Every request that reaches `getOrCreateController` for a name that doesn't exist creates one:
- a `CommandScalableTopicSubscribe` for a topic that doesn't exist, which fails with `UnknownError`. For example, the topic was deleted after the client read its layout;
- an admin call served by the controller leader on a mistyped topic name, such as `getStats`, `createSubscription` or a split.
Example (from the new test):
```java
for (int i = 0; i < 3; i++) {
TopicName missing = TopicName.get("topic://tenant/ns/t-missing-" + i);
service.registerConsumer(missing, "sub", "c1", i, ScalableConsumerType.STREAM, cnx).get();
// fails: Failed to initialize controller for topic://tenant/ns/t-missing-<i>
}
// getLeaderElection was called for /topics/tenant/ns/t-missing-{0,1,2}/controller:
// 3 more elections and 6 more metadata store listeners, until the broker restarts
```
Current code path:
```
ServerCnx.handleCommandScalableTopicSubscribe, or a ScalableTopics REST call (onControllerLeader)
-> ScalableTopicService.getOrCreateController(topic)
-> new ScalableTopicController(...)
-> coordinationService.getLeaderElection(String.class, ".../controller", ...) // cached forever
-> controller.initialize() // fails: "Scalable topic not found", or a metadata error after the election
-> controllers.remove(key, stored) // never closed
```
### Modifications
- `ScalableTopicController` creates its leader election on the first election attempt, which `initialize()` makes only after reading the topic record. `close()` and `getLeaderBrokerId()` handle a controller that never created one.
- `ScalableTopicService.getOrCreateController` closes a controller whose `initialize()` failed before dropping it. The close cancels the GC and auto split/merge tasks, stops the coordinators of the subscriptions restored so far, and closes the election, which releases the controller lock. The failed future is evicted only once the close is done, so the next attempt finds the lock released. Callers still get the initialization failure.
```java
return controller.initialize()
// initialize() can fail after winning the election, with the controller lock
// held and the leader's periodic tasks scheduled: close the controller before
// dropping it, or it keeps acting as the leader next to the controller of the
// next attempt.
.exceptionallyCompose(ex -> controller.close()
.handle((__, closeEx) -> {
if (closeEx != null) {
log.warn().attr("topic", topic).exceptionMessage(closeEx)
.log("Failed to close a controller that failed to initialize");
}
throw FutureUtil.wrapToCompletionException(ex);
}))
.thenApply(__ -> controller);
```
Not changed: `CoordinationServiceImpl` still keeps a closed election for the broker's lifetime. A later controller of the same topic on the same broker reuses that instance, which still reports state changes to the listener of the controller that created it. Dropping closed elections needs a change in pulsar-metadata (central notification dispatch, since a `MetadataStore` listener can't be unregistered) that keeps `LeaderElectionService`'s re-election after `close()` working, so it is left for a separate change.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- `ScalableTopicServiceTest#testFailedInitializationStopsTheController` injects a one-shot metadata failure into `restoreSessionsFromStore()` with `FaultInjectionMetadataStore`. It checks that the failed call leaves the controller lock free and no periodic task scheduled, and that the retry's controller is then the only one: releasing it leaves nothing running. It fails on master because the lock is still held. Also on master: after the failure, the failed controller's GC task is still scheduled; after the retry, two GC tasks are scheduled; after the retry's controller is released, one is still scheduled.
- `ScalableTopicServiceTest#testSubscribeToMissingTopicsCreatesNoLeaderElection` subscribes to three topics that don't exist and checks that `getLeaderElection` is never called. On master it is called once per topic name.
- Each of the two tests fails with only the other half of the change.
- These existing tests also pass:
- all of `org.apache.pulsar.broker.service.scalable.*`;
- the V5 client tests in `org.apache.pulsar.client.api.v5.*` and `org.apache.pulsar.client.impl.v5.*`, except the `V5MultiBroker*` tests, which need Docker;
- the scalable topic tests in `org.apache.pulsar.broker.admin`, `org.apache.pulsar.broker.resources` and `org.apache.pulsar.broker.service`;
- `./gradlew quickCheck`.
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't leave a failed scalable topic controller or its leader election behind [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26877:
URL: https://github.com/apache/pulsar/pull/26877#discussion_r4254720951
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable/ScalableTopicController.java:
##########
@@ -1652,7 +1668,9 @@ public CompletableFuture<Void> close() {
// task keeps running after the controller goes away.
subscriptions.values().forEach(SubscriptionCoordinator::close);
Review Comment:
Merged master into the branch: #26878 does exactly this for every close (`SubscriptionCoordinator.close()` cancels the sessions' grace timers, and `evictExpiredConsumer()` returns once closed), and the test now runs your partial-restore scenario (8df8c176f92): no task is left after the failed attempt and both registrations survive, and it fails if `close()` keeps the grace timers.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable/ScalableTopicController.java:
##########
@@ -751,14 +750,30 @@ private CompletableFuture<Void> electLeader() {
// NamespaceService.createLookupResult(brokerId, ...), matching the pattern used by
// the cluster-leader redirection in NamespacesBase.
String brokerId = brokerService.getPulsar().getBrokerId();
- return leaderElection.elect(brokerId)
+ return getOrCreateLeaderElection().elect(brokerId)
Review Comment:
Handled on master, which this branch now merges (the branch no longer changes the controller): `electLeader()` fails fast when closed (#26875), the takeover and the session restore check `closed` and the election term (#26878), and since #26879 a closed election is never handed to the next controller (it gets a new one, and the closed one can't be re-opened once that exists), so the late `asyncClose()` in `electLeader()` only closes this controller's own election.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable/ScalableTopicController.java:
##########
@@ -751,14 +750,30 @@ private CompletableFuture<Void> electLeader() {
// NamespaceService.createLookupResult(brokerId, ...), matching the pattern used by
// the cluster-leader redirection in NamespacesBase.
String brokerId = brokerService.getPulsar().getBrokerId();
- return leaderElection.elect(brokerId)
+ return getOrCreateLeaderElection().elect(brokerId)
.thenAccept(state -> {
this.leaderState = state;
log.info().attr("state", state)
.log("Leader election for scalable topic");
});
}
+ /**
+ * The leader election on the controller lock, created by the first election attempt.
+ * {@link #initialize()} makes it only after reading the topic record: the coordination
+ * service keeps an election, with its metadata store listeners, for the broker's lifetime,
+ * so a controller created for a topic that doesn't exist must not create one.
+ */
+ private synchronized LeaderElection<String> getOrCreateLeaderElection() {
+ if (leaderElection == null) {
+ leaderElection = coordinationService.getLeaderElection(
+ String.class,
+ resources.controllerLockPath(topicName),
+ this::onLeaderStateChange);
Review Comment:
Fixed on master, which this branch now merges: since #26879 a closed election is dropped from `CoordinationServiceImpl`, so the next controller of the topic gets an election of its own that reports to its own listener, and since #26878 the controller follows its election's state changes (a lost lock makes it stop serving). `testControllerCreatedAfterFailedInitializationLeads` and `testClosedControllerIsReplaced` cover it, so I don't think a follow-up issue is needed.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -233,6 +237,60 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationStopsTheController() throws Exception {
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ // Fail restoreSessionsFromStore(), which runs once the controller won the election and
+ // scheduled the leader's periodic tasks. The failure is injected once.
+ String subscriptionsPath = resources.topicPath(tn) + "/subscriptions";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(subscriptionsPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ assertFalse(store.exists(lockPath).get(), "the failed controller must release the controller lock");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must stop its periodic tasks");
Review Comment:
Done in 8df8c176f92: the test stubs `pulsar.getConfig()`, so the leader also schedules its auto split/merge tick and the release must cancel it, and it fails the restore on sub-b's consumers after sub-a/c1 was restored with a grace timer, checking that no task is left and both registrations survive. It fails if `SubscriptionCoordinator.close()` keeps the grace timers, or if `close()` keeps the auto split/merge tick.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable/ScalableTopicController.java:
##########
@@ -770,7 +785,8 @@ public boolean isLeader() {
* boolean, String)}.
*/
public CompletableFuture<Optional<String>> getLeaderBrokerId() {
- return leaderElection.getLeaderValue();
+ LeaderElection<String> election = leaderElection;
+ return election != null ? election.getLeaderValue() : CompletableFuture.completedFuture(Optional.empty());
Review Comment:
No longer applies: after merging master the branch doesn't change the controller anymore, so it creates its election in the constructor as before, and `getLeaderBrokerId()` reads the lock from the store until the first election.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't leave a failed scalable topic controller or its leader election behind [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #26877:
URL: https://github.com/apache/pulsar/pull/26877#discussion_r4253179677
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable/ScalableTopicController.java:
##########
@@ -751,14 +750,30 @@ private CompletableFuture<Void> electLeader() {
// NamespaceService.createLookupResult(brokerId, ...), matching the pattern used by
// the cluster-leader redirection in NamespacesBase.
String brokerId = brokerService.getPulsar().getBrokerId();
- return leaderElection.elect(brokerId)
+ return getOrCreateLeaderElection().elect(brokerId)
Review Comment:
`close()` doesn't stop an `initialize()` already in flight: `onLeaderStateChange(NoLeader)` checks `!closed` and then starts an async `initialize()`, and nothing checks `closed` again before this `elect()`. `LeaderElectionImpl.elect()` reopens a closed election, so if the lock is lost while the first `initialize()` is failing (likely when the metadata store is flapping), the closed controller can take the lock back and restore sessions. Scheduling the periodic tasks is guarded by `closed`; the election and the restore aren't.
Suggest failing fast here when `closed`, and skipping the leader branch of `initialize()` when `closed`. Not a late `asyncClose()`, though: the coordination service caches the election instance per path, so it may already be shared with the next controller.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable/ScalableTopicController.java:
##########
@@ -751,14 +750,30 @@ private CompletableFuture<Void> electLeader() {
// NamespaceService.createLookupResult(brokerId, ...), matching the pattern used by
// the cluster-leader redirection in NamespacesBase.
String brokerId = brokerService.getPulsar().getBrokerId();
- return leaderElection.elect(brokerId)
+ return getOrCreateLeaderElection().elect(brokerId)
.thenAccept(state -> {
this.leaderState = state;
log.info().attr("state", state)
.log("Leader election for scalable topic");
});
}
+ /**
+ * The leader election on the controller lock, created by the first election attempt.
+ * {@link #initialize()} makes it only after reading the topic record: the coordination
+ * service keeps an election, with its metadata store listeners, for the broker's lifetime,
+ * so a controller created for a topic that doesn't exist must not create one.
+ */
+ private synchronized LeaderElection<String> getOrCreateLeaderElection() {
+ if (leaderElection == null) {
+ leaderElection = coordinationService.getLeaderElection(
+ String.class,
+ resources.controllerLockPath(topicName),
+ this::onLeaderStateChange);
Review Comment:
Not new, and the description mentions it, but worth tracking: `CoordinationServiceImpl` caches elections per path and keeps the first listener, so the retry's controller gets an election whose listener is the closed controller's `onLeaderStateChange`. After its own `elect()` it never sees a state change: if the lock is later lost (e.g. session expiry, another broker wins), its `leaderState` stays `Leading`, and it keeps running GC and auto split/merge and passing `checkLeader()` next to the new leader.
A broker-side fix is possible without pulsar-metadata changes, e.g. derive `isLeader()` from `leaderElection.getState()`, or have `ScalableTopicService` own one listener per lock path that forwards to the current controller. Could you open a follow-up issue for it?
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -233,6 +237,60 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationStopsTheController() throws Exception {
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ // Fail restoreSessionsFromStore(), which runs once the controller won the election and
+ // scheduled the leader's periodic tasks. The failure is injected once.
+ String subscriptionsPath = resources.topicPath(tn) + "/subscriptions";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(subscriptionsPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ assertFalse(store.exists(lockPath).get(), "the failed controller must release the controller lock");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must stop its periodic tasks");
Review Comment:
`pulsar.getConfig()` isn't stubbed in this class, so `scheduleAutoScaleTask()` returns early and only the GC task is ever queued here; the auto split/merge cancellation isn't exercised. Stubbing `when(pulsar.getConfig()).thenReturn(new ServiceConfiguration())` in this test would cover both tasks (the default 60s tick won't fire during the test).
A partial-restore variant would also help: two subscriptions with persisted consumers, fail the second one's `.../consumers` GET_CHILDREN, then assert the queue is empty. That variant fails today because of the restored sessions' grace timers (see the comment on `ScalableTopicController#close`).
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable/ScalableTopicController.java:
##########
@@ -1652,7 +1668,9 @@ public CompletableFuture<Void> close() {
// task keeps running after the controller goes away.
subscriptions.values().forEach(SubscriptionCoordinator::close);
Review Comment:
`SubscriptionCoordinator.close()` only cancels the drain poller. The sessions created by `restoreConsumers()` keep their grace timers (`ConsumerSession.restored()` arms one), and on expiry `evictExpiredConsumer()` deletes the persisted registration via `unregisterConsumerAsync`, without checking `closed`.
So when `restoreSessionsFromStore()` fails partway (one subscription restored, the next one's consumer listing fails), this closed controller still deletes registrations after the grace period, including ones the retry's controller is serving. I reproduced it locally with `FaultInjectionMetadataStore` (`sub-a/c1` and `sub-b/c2` persisted, one-shot failure on `sub-b`'s `consumers` GET_CHILDREN):
- after the failed `getOrCreateController`, the lock is released but one task stays queued (`c1`'s grace timer);
- with a 1s grace period, `c1` re-registered on the retry's controller, and ~1s later the closed controller's timer evicted it and deleted `.../subscriptions/sub-a/consumers/c1`.
Could `SubscriptionCoordinator.close()` also cancel each session's grace timer (`ConsumerSession.cancelGraceTimer()` is package-private), and `evictExpiredConsumer()` return early when `closed`? E.g.:
```java
public synchronized void close() {
closed = true;
...
// Eviction deletes the persisted registration the next controller relies on.
sessions.values().forEach(ConsumerSession::cancelGraceTimer);
}
```
(The same gap applies to every other controller close, e.g. topic delete.)
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/scalable/ScalableTopicController.java:
##########
@@ -770,7 +785,8 @@ public boolean isLeader() {
* boolean, String)}.
*/
public CompletableFuture<Optional<String>> getLeaderBrokerId() {
- return leaderElection.getLeaderValue();
+ LeaderElection<String> election = leaderElection;
+ return election != null ? election.getLeaderValue() : CompletableFuture.completedFuture(Optional.empty());
Review Comment:
Nit: before the first election this now returns `Optional.empty()`, where it used to read the lock from the store (an election that hasn't run yet reads it from there in `getLeaderValue()`). Only tests call it, after `initialize()`, so there's no impact; a Javadoc line would document the new behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][test] Cover the cleanup of a scalable topic controller that fails to initialize [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #26877:
URL: https://github.com/apache/pulsar/pull/26877
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't leave a failed scalable topic controller or its leader election behind [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #26877:
URL: https://github.com/apache/pulsar/pull/26877#discussion_r4260450949
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
Review Comment:
Folded it in cc4e93609f1: `testControllerFailingAfterElectionGivesTheLockBack` is gone, since `testFailedInitializationLeavesNothingRunning` checks the lock too; its failure point (before any subscription is restored) is now the one of the new `testRetryWhileTheFailedControllerDeletesTheLock`, and each test has a topic name of its own.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
Review Comment:
Done in cc4e93609f1: the test now asserts that the cause is the injected `MetadataStoreException` instance (`FutureUtil.unwrapCompletionException(e.getCause().getCause())`), so an attempt failing before the restore fails right there.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ // The failed controller is closed: its election gives the lock back, and none of its tasks
+ // (takeover retries, grace timers of restored consumers) stays scheduled.
+ Awaitility.await().untilAsserted(() -> {
+ assertFalse(store.exists(lockPath).get(), "the failed controller must give the lock back");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must leave no task scheduled");
+ });
+ // An expiring grace timer would delete the registration that the next controller restores.
Review Comment:
Done in cc4e93609f1: the comment now reads "The close leaves the persisted registrations to the next controller."
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ // The failed controller is closed: its election gives the lock back, and none of its tasks
+ // (takeover retries, grace timers of restored consumers) stays scheduled.
+ Awaitility.await().untilAsserted(() -> {
+ assertFalse(store.exists(lockPath).get(), "the failed controller must give the lock back");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must leave no task scheduled");
+ });
+ // An expiring grace timer would delete the registration that the next controller restores.
+ assertEquals(resources.listConsumersAsync(tn, "sub-a").get(), List.of("c1"));
+ assertEquals(resources.listConsumersAsync(tn, "sub-b").get(), List.of("c2"));
+
+ // The retry runs the only controller of the topic: releasing it leaves nothing running.
+ assertTrue(service.getOrCreateController(tn).get().isLeader(), "the retry must lead the topic");
+ assertEquals(scheduler.getQueue().size(), 4,
+ "the GC and auto split/merge ticks, and the grace timers of c1 and c2: " + scheduler.getQueue());
+ service.releaseController(tn).get();
+ assertFalse(store.exists(lockPath).get());
+ assertTrue(scheduler.getQueue().isEmpty(), "no task may outlive the released controller");
+ }
+
+ @Test
+ public void testSubscribeToMissingTopicsLeavesNoLeaderElectionOpen() throws Exception {
+ CoordinationService observedCoordinationService = spy(coordinationService);
+ Map<String, LeaderElection<?>> elections = new ConcurrentHashMap<>();
+ doAnswer(inv -> {
+ LeaderElection<?> election = (LeaderElection<?>) inv.callRealMethod();
+ elections.put(inv.getArgument(1), election);
+ return election;
+ }).when(observedCoordinationService).getLeaderElection(any(), anyString(), any());
+ ScalableTopicService observedService =
+ new ScalableTopicService(brokerService, resources, observedCoordinationService);
+ try {
+ for (int i = 0; i < 3; i++) {
+ TopicName missing = scalableTopic("t-missing-" + i);
+ try {
+ observedService.registerConsumer(missing, "sub", "c1", i, ScalableConsumerType.STREAM,
+ mock(TransportCnx.class)).get();
+ fail("expected the subscribe to a missing topic to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ }
+ assertEquals(elections.size(), 3);
Review Comment:
Done in cc4e93609f1: the test walks the lock paths of the missing topics and compares each probe with a snapshot of the elections taken before probing, so a controller that creates no election for a missing topic passes too.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ // The failed controller is closed: its election gives the lock back, and none of its tasks
+ // (takeover retries, grace timers of restored consumers) stays scheduled.
+ Awaitility.await().untilAsserted(() -> {
+ assertFalse(store.exists(lockPath).get(), "the failed controller must give the lock back");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must leave no task scheduled");
+ });
Review Comment:
Added `testRetryWhileTheFailedControllerDeletesTheLock` in cc4e93609f1: a small `FaultInjectionMetadataStore` subclass holds back the failed controller's delete of the lock, the retry leads with the lock it finds (same session), and once the delete goes through its election takes the lock again, while the controller keeps leading and stays the only one; releasing it leaves nothing behind.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ // The failed controller is closed: its election gives the lock back, and none of its tasks
+ // (takeover retries, grace timers of restored consumers) stays scheduled.
+ Awaitility.await().untilAsserted(() -> {
+ assertFalse(store.exists(lockPath).get(), "the failed controller must give the lock back");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must leave no task scheduled");
+ });
+ // An expiring grace timer would delete the registration that the next controller restores.
+ assertEquals(resources.listConsumersAsync(tn, "sub-a").get(), List.of("c1"));
+ assertEquals(resources.listConsumersAsync(tn, "sub-b").get(), List.of("c2"));
Review Comment:
Reworded in cc4e93609f1: the failed attempt is described as leaving c1's grace timer, which the close cancels, and a takeover retry, which the close only makes a no-op (the queue check waits for it); the registration checks as catching a close that deletes them; and the exact task count is replaced by "non-empty while serving, empty after the release". Cancelling the retry in `close()` is left for a follow-up, since this PR only changes tests.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't leave a failed scalable topic controller or its leader election behind [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #26877:
URL: https://github.com/apache/pulsar/pull/26877#discussion_r4255369064
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
Review Comment:
This checks what `testControllerFailingAfterElectionGivesTheLockBack` (#26875) checks (the lock is given back after `initialize()` fails past the election), and more. It even uses the same topic name, `t-init-fails`. Could the older test go, or be folded into this one? If both failure points are worth keeping (no subscription restored yet vs. a partial restore), a distinct topic name would keep their logs apart.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ // The failed controller is closed: its election gives the lock back, and none of its tasks
+ // (takeover retries, grace timers of restored consumers) stays scheduled.
+ Awaitility.await().untilAsserted(() -> {
+ assertFalse(store.exists(lockPath).get(), "the failed controller must give the lock back");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must leave no task scheduled");
+ });
+ // An expiring grace timer would delete the registration that the next controller restores.
+ assertEquals(resources.listConsumersAsync(tn, "sub-a").get(), List.of("c1"));
+ assertEquals(resources.listConsumersAsync(tn, "sub-b").get(), List.of("c2"));
+
+ // The retry runs the only controller of the topic: releasing it leaves nothing running.
+ assertTrue(service.getOrCreateController(tn).get().isLeader(), "the retry must lead the topic");
+ assertEquals(scheduler.getQueue().size(), 4,
+ "the GC and auto split/merge ticks, and the grace timers of c1 and c2: " + scheduler.getQueue());
+ service.releaseController(tn).get();
+ assertFalse(store.exists(lockPath).get());
+ assertTrue(scheduler.getQueue().isEmpty(), "no task may outlive the released controller");
+ }
+
+ @Test
+ public void testSubscribeToMissingTopicsLeavesNoLeaderElectionOpen() throws Exception {
+ CoordinationService observedCoordinationService = spy(coordinationService);
+ Map<String, LeaderElection<?>> elections = new ConcurrentHashMap<>();
+ doAnswer(inv -> {
+ LeaderElection<?> election = (LeaderElection<?>) inv.callRealMethod();
+ elections.put(inv.getArgument(1), election);
+ return election;
+ }).when(observedCoordinationService).getLeaderElection(any(), anyString(), any());
+ ScalableTopicService observedService =
+ new ScalableTopicService(brokerService, resources, observedCoordinationService);
+ try {
+ for (int i = 0; i < 3; i++) {
+ TopicName missing = scalableTopic("t-missing-" + i);
+ try {
+ observedService.registerConsumer(missing, "sub", "c1", i, ScalableConsumerType.STREAM,
+ mock(TransportCnx.class)).get();
+ fail("expected the subscribe to a missing topic to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ }
+ assertEquals(elections.size(), 3);
Review Comment:
This also pins that the controller creates an election for a topic that doesn't exist, which isn't the property under test. A later change that skips the election for a missing topic (this PR's first approach) would fail here while leaving fewer elections. Walking the expected lock paths covers both cases:
```java
// in the loop: lockPaths.add(resources.controllerLockPath(missing));
Map<String, LeaderElection<?>> handedOut = Map.copyOf(elections);
for (String path : lockPaths) {
assertNotSame(observedCoordinationService.getLeaderElection(String.class, path, state -> { }),
handedOut.get(path), "the election of a controller that failed to initialize must be closed: " + path);
}
```
A path with no election gives `null`, which passes, and walking the expected paths keeps it from passing vacuously. Take the snapshot before probing: the `doAnswer` records the probe's own election.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
Review Comment:
Nit: this only checks the wrapper message, so nothing confirms that the attempt reached the restore, which the lock and queue assertions below rely on. If `initialize()` ever fails earlier, `listConsumersAsync(tn, "sub-b")` at L279 consumes the one-shot fault, and the test fails there with an unrelated-looking `ExecutionException`. Asserting the injected `MetadataStoreException` in the cause chain (e.g. `FutureUtil.unwrapCompletionException(e.getCause().getCause()) instanceof MetadataStoreException`) would state the precondition.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ // The failed controller is closed: its election gives the lock back, and none of its tasks
+ // (takeover retries, grace timers of restored consumers) stays scheduled.
+ Awaitility.await().untilAsserted(() -> {
+ assertFalse(store.exists(lockPath).get(), "the failed controller must give the lock back");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must leave no task scheduled");
+ });
+ // An expiring grace timer would delete the registration that the next controller restores.
Review Comment:
Nit: with the default 60 s grace period, no timer can expire before these assertions. What they check is that the close leaves the persisted registrations alone; the queue assertion above covers the timers. Maybe `// The close leaves the persisted registrations to the next controller.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Don't leave a failed scalable topic controller or its leader election behind [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #26877:
URL: https://github.com/apache/pulsar/pull/26877#discussion_r4259316367
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ // The failed controller is closed: its election gives the lock back, and none of its tasks
+ // (takeover retries, grace timers of restored consumers) stays scheduled.
+ Awaitility.await().untilAsserted(() -> {
+ assertFalse(store.exists(lockPath).get(), "the failed controller must give the lock back");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must leave no task scheduled");
+ });
Review Comment:
**[QUALITY] the test never retries while the failed controller's cleanup is still pending**
`newController` completes the future exceptionally without waiting for `controller.close()`, and `LeaderElectionImpl.asyncClose()` drops the election from the cache at once but deletes the lock later. The wait here, and the inline `LocalMemoryMetadataStore`, mean the retry below always starts after the lock is gone. So a retry that starts while the delete is still in flight, and the repeated close, aren't covered.
From reading the code that overlap should heal itself: the retry's election adopts its own lock, then the delete notification forces a re-election. It does cost a leadership flap on a real store. A test that holds back the `DELETE` of `lockPath` (a small `FaultInjectionMetadataStore` wrapper or subclass; `failConditional` can only fail an operation, not delay it), retries, then releases it, and asserts a single leading controller and a final clean state would pin that down.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/scalable/ScalableTopicServiceTest.java:
##########
@@ -236,6 +242,85 @@ public void testGetOrCreateControllerFailsIfTopicMissing() throws Exception {
}
}
+ @Test
+ public void testFailedInitializationLeavesNothingRunning() throws Exception {
+ // With a broker configuration, the leader schedules its auto split/merge tick next to the GC
+ // tick. Neither fires during the test (60s).
+ when(pulsar.getConfig()).thenReturn(new ServiceConfiguration());
+ TopicName tn = scalableTopic("t-init-fails");
+ service.createScalableTopic(tn, 1).get();
+ String lockPath = resources.controllerLockPath(tn);
+ resources.createSubscriptionAsync(tn, "sub-a", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-a", "c1").get();
+ resources.createSubscriptionAsync(tn, "sub-b", SubscriptionType.STREAM).get();
+ resources.registerConsumerAsync(tn, "sub-b", "c2").get();
+ // Restoring the consumer sessions runs once the controller won the election. Fail it on
+ // sub-b, while sub-a's consumer is restored with a grace timer. The failure is injected
+ // once, so the next attempt succeeds.
+ String consumersPath = resources.subscriptionPath(tn, "sub-b") + "/consumers";
+ store.failConditional(new MetadataStoreException("injected"), (op, path) ->
+ op == FaultInjectionMetadataStore.OperationType.GET_CHILDREN && path.equals(consumersPath));
+
+ try {
+ service.getOrCreateController(tn).get();
+ fail("expected the controller initialization to fail");
+ } catch (ExecutionException e) {
+ assertTrue(e.getCause().getMessage().contains("Failed to initialize controller"),
+ "got: " + e.getCause().getMessage());
+ }
+ // The failed controller is closed: its election gives the lock back, and none of its tasks
+ // (takeover retries, grace timers of restored consumers) stays scheduled.
+ Awaitility.await().untilAsserted(() -> {
+ assertFalse(store.exists(lockPath).get(), "the failed controller must give the lock back");
+ assertTrue(scheduler.getQueue().isEmpty(), "the failed controller must leave no task scheduled");
+ });
+ // An expiring grace timer would delete the registration that the next controller restores.
+ assertEquals(resources.listConsumersAsync(tn, "sub-a").get(), List.of("c1"));
+ assertEquals(resources.listConsumersAsync(tn, "sub-b").get(), List.of("c2"));
Review Comment:
**[QUALITY] the comments and assertions around the cleanup check overstate what it proves**
A few details here claim more than they check:
- The failed start never reaches `serveAsLeader()`, so the GC and auto split/merge ticks aren't scheduled in this phase. What it leaves behind is c1's grace timer and the takeover retry. Periodic-task cancellation is only covered by the release at the end.
- `close()` doesn't cancel the takeover retry scheduled in `startTakeOver`; it only makes it a no-op once it fires. The `isEmpty()` wait passes after that retry fires, so "none of its tasks (takeover retries …) stays scheduled" isn't quite accurate.
- The grace period is 60 s and `evictExpiredConsumer()` returns once the coordinator is closed, so these `listConsumersAsync` checks can't detect a leaked timer. They only catch `close()` deleting registrations. The queue check is what guards the timers.
- `assertEquals(scheduler.getQueue().size(), 4, …)` breaks if any other task is added to the broker executor.
Rewording the comments would be enough. Cancelling the retry in `close()` would be a nice follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org