[I] [Bug] Graceful shutdown leaks cluster subscriptions (/em/subs) and leaves ghost subscribers on every peer (eventmesh)

["qqeasonchen (via GitHub)" <gi...@apache.org>]

qqeasonchen opened a new issue, #5415:
URL: https://github.com/apache/eventmesh/issues/5415

   ### Describe the bug
   
   A gracefully stopped instance leaves its cluster subscriptions in the shared MetaStore (`/em/subs/<topic>/<clientId>`), so every peer keeps a **ghost subscriber** pointing at a dead URL.
   
   Consequences:
   
   - peers keep forwarding events to the dead instance's `/internal/forward` (delivery errors, events dropped after the retry budget);
   - `activeTopics()` on every peer keeps the topic alive, so the survivor keeps polling and assigning Kafka partitions for a topic whose only subscriber no longer exists (combined with the shared-consumer bug this starves live topics);
   - nothing can clean the entry up afterwards: with `nacos-client` 2.1.1 a **foreign** client's `deregisterInstance` for an ephemeral instance is silently ignored by the server (only the registering client can remove it).
   
   ### Reproduction
   
   1. Start A and B, `POST /events/subscribe` with `clientId=c1` on A.
   2. `A.shutdown()`.
   3. Read `/em/subs/<topic>/c1` from Nacos — it is still there (forever).
   4. Publish through B: B tries to forward events to A's dead URL and keeps pulling A's partitions.
   
   Seen in e2e as cross-test pollution: while testing graceful restart, the live instances logged `assigned partitions for <topic-of-the-previous-crash-test>: [0, 2]` — they were pulling a dead instance's topic.
   
   ### Root cause
   
   Nothing deregisters the departing instance's subscriptions. `UniIngressService.unsubscribeByClient` exists but is only reached from the HTTP unsubscribe endpoint, never from shutdown, and the 2.1.1 server-side rule means a peer cannot compensate later.
   
   ### Fix
   
   - `UniIngressService.dropAllClusterSubscriptions()` — snapshot the locally subscribed client ids, then run each through `unsubscribeByClient` (local removal + `cluster.unsubscribe` Meta delete + push-buffer cleanup);
   - `EventMeshApplication.shutdown()` calls it after the HTTP/WS/gRPC servers and the session router are stopped (no new subscription can arrive) and **before** the heartbeat scheduler is stopped / `leave()` runs, so this instance's own Meta client is still alive to perform the delete.
   
   ### Verification
   
   - `UniIngressServiceTest` +2 cases: clustered (both the local manager and the cluster store are cleared) and standalone (no cluster wired).
   - E2E `LoadBalancingAndFailoverE2EIntegrationTest`: after the fix the cross-topic assignment pollution disappears (the only `assigned partitions for ...` lines name the test's own topic) and all events of the graceful phase are delivered.
   
   ### Notes
   
   With nacos-client 2.1.1 a test-side sweep of `/em/subs/*` cannot remove entries owned by another process, so the runtime-side cleanup on shutdown is the only reliable mechanism; a SIGKILLed instance's ephemeral naming entries disappear with its connection, but its ConfigService keys (`/em/subs/*`) still need the stale-client reaper (see the stale-client eviction issue).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org

Re: [I] [Bug] Graceful shutdown leaks cluster subscriptions (/em/subs) and leaves ghost subscribers on every peer (eventmesh)

["qqeasonchen (via GitHub)" <gi...@apache.org>]

qqeasonchen closed issue #5415: [Bug] Graceful shutdown leaks cluster subscriptions (/em/subs) and leaves ghost subscribers on every peer
URL: https://github.com/apache/eventmesh/issues/5415


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org