You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@eventmesh.apache.org by "qqeasonchen (via GitHub)" <gi...@apache.org> on 2026/08/18 03:11:26 UTC
[I] [Bug] Unify cluster delivery topology and fence stale partition owners (eventmesh)
qqeasonchen opened a new issue, #5293:
URL: https://github.com/apache/eventmesh/issues/5293
## Problem
Cluster-related code currently retains three partially overlapping models:
1. sticky local pull/local delivery configured by `EventMeshApplication.enableCluster()`;
2. partition-owned pull through `PartitionOwnership`;
3. cross-instance forwarding through `ClusterCoordinator` and `HttpForwarder`.
Their startup wiring, offset ownership, ACK semantics, failure behavior, and production maturity are not represented by one explicit topology choice. `PartitionOwnership` also relies on soft generation fencing and read/write Meta updates that are not a compare-and-set lease protocol.
Relevant code:
- `eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/boot/EventMeshApplication.java`
- `eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/ingress/UniIngressService.java`
- `eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/cluster/PartitionOwnership.java`
- `eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/cluster/ClusterMembership.java`
## Reproduction
1. Configure two instances against a shared MetaStore and Kafka topic with multiple partitions.
2. Introduce a network partition or MetaStore outage for one instance.
3. Continue publishing and consuming during the split.
4. Restore connectivity and compare per-partition delivery sets, duplicates, offsets, and ownership records.
5. Repeat for sticky mode and partition-owned mode.
## Expected behavior
The deployment must select one explicit delivery topology with documented guarantees. Ownership handoff must prevent stale owners from continuing to create valid deliveries after their epoch/lease is lost.
## Proposed direction
- Introduce an explicit `DeliveryTopology` configuration:
- `LOCAL_STICKY_PULL`
- `PARTITION_OWNED_PULL`
- `FORWARDED_DISPATCH`
- Define subscription persistence, physical cursor ownership, ACK target, duplicate bound, and Meta outage behavior for each topology.
- For partition ownership, use lease revision/CAS plus owner epoch fencing; carry the epoch into polling/delivery validation.
- Keep unfinished topologies experimental until their failover contract is tested.
## Acceptance criteria
- [ ] Runtime startup selects one topology explicitly; inactive topology components are not silently half-wired.
- [ ] Documentation defines ACK, offset, duplicate, and recovery semantics per topology.
- [ ] Partition handoff uses an atomic lease/revision or equivalent fencing mechanism.
- [ ] A stale owner cannot continue valid delivery after losing ownership.
- [ ] Fault-injection tests cover Meta outage, network split, owner crash, graceful handoff, and recovery.
- [ ] Kafka multi-instance tests verify no unintended simultaneous partition ownership.
## Suggested labels
`bug`, `need research`, `architecture`, `cluster`, `priority:p1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Bug] Unify cluster delivery topology and fence stale partition owners (eventmesh)
Posted by "qqeasonchen (via GitHub)" <gi...@apache.org>.
qqeasonchen closed issue #5293: [Bug] Unify cluster delivery topology and fence stale partition owners
URL: https://github.com/apache/eventmesh/issues/5293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org
Re: [I] [Bug] Unify cluster delivery topology and fence stale partition owners (eventmesh)
Posted by "qqeasonchen (via GitHub)" <gi...@apache.org>.
qqeasonchen commented on issue #5293:
URL: https://github.com/apache/eventmesh/issues/5293#issuecomment-5350249343
The proposed direction has been implemented and submitted as #5308 (single PR, based on current develop):
- **Sticky-only topology**: `HttpForwarder` and the `/internal/forward` + `/internal/reply-forward` endpoints are deleted; `reply()` on a non-owner instance now 404s (the SDK pins to one instance via `instanceUrl`, so cross-instance forwarding is no longer needed). `DistributionMode.LOAD_BALANCE_STICKY` is removed; `LOAD_BALANCE` absorbs the sticky semantics (partition-key hash routes to one subscriber).
- **Meta CAS fencing**: `MetaStore.tryAcquire(key, expectedOldValue, newValue)` atomic CAS (InMemory `ConcurrentHashMap` / Nacos 2.x `publishConfigCas` with `casMd5`); `FencingToken "<bootEpoch>:<counter>"` per JVM, monotonic and restart-safe; assignment record `/em/assignments/<topic#partition>` = `"<token>|<ownerInstanceId>"`; `acquireOrFence` 3-branch protocol with TTL-eviction takeover; `releaseStale` CAS'es a `""` tombstone for partitions that left our assigner share so the new rightful owner can claim them (prevents stranding after membership churn).
- **Fault-injection tests** (`ClusterDeliveryFaultTest`): in-process 3-4 instance scenarios (steady-state split, crash takeover, scale-out churn, Meta partition split-brain guard, healed partition reclaim) driven by a mutable clock — fully deterministic, no sleeps.
Design details: `docs/eventmesh-uni-architecture-redesign.md` §13.2.10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org