[I] [P1] chaos (MetaStore / broker / watch / RocksDB) + rolling upgrade + admin security (eventmesh)

["qqeasonchen (via GitHub)" <gi...@apache.org>]

qqeasonchen opened a new issue, #5364:
URL: https://github.com/apache/eventmesh/issues/5364

   Tracking sub-issue of #5354. Part of the production-HA acceptance plan (Phase 3, P1).
   
   ## What this PR adds
   
   The "production operations" half of the review: chaos engineering, rolling upgrade, and admin security. These are P1 not P0 because the wiring + tests in #5358 / #5359 / #5360 / #5362 close the data-correctness gaps; this issue adds the operational evidence that production deployments will need.
   
   ### Chaos test cases (Testcontainers-based, building on the #5362 harness)
   
   1. `MetaStoreOutageTest`: stop the Nacos container mid-test for 30s; the runtime degrades to "fail-closed for new assignments, continue for owned partitions"; on Nacos restart, partition reassignment resumes without offset regression.
   2. `BrokerOutageTest`: stop the RocketMQ 5 broker mid-test for 30s; the runtime backs off and resumes polling on broker restart; no message loss; at-least-once bound is respected.
   3. `WatchLossTest`: drop the Nacos watch mid-test by toggling the network namespace; the runtime rebuilds its view from a full `getWithPrefix(...)` and does not double-dispatch.
   4. `RocksDBCorruptionTest`: write 100 offsets, then overwrite the RocksDB SST file with garbage; the runtime refuses to start with the documented error (`OffsetStore is corrupt, refusing to start; restore from backup or wipe the data dir`).
   5. `BackupRestoreTest`: snapshot the RocksDB + OffsetStore + MetaStore records, run a 5-minute test, restore from the snapshot to a fresh instance, assert the restored instance does not overwrite newer offsets / ownership / DLQ entries.
   
   ### Rolling upgrade test
   
   Build a `v(N)` and `v(N+1)` Docker image; roll one of the 2 Runtime instances to `v(N+1)` with a 5-minute mixed-version window; assert no offset regression, no duplicate delivery beyond the at-least-once bound, no stale-owner writes.
   
   ### Admin security
   
   - `/admin/*` and `/metrics` are protected by a config-gated token (`eventmesh.admin.token`); missing token returns 401, wrong token returns 403, fail-closed by default.
   - Document the security posture in `docs/control-plane.md` section 4 (new section).
   - Add a security matrix in `docs/production-readiness.md` listing each endpoint, its auth requirement, and the documented behavior on missing/wrong token.
   
   ### Metrics
   
   - New metrics: `fencedPartitions`, `metaStoreErrorClass{class}`, `leaseLoss`, `staleOwnerRejections`, `partitionTakeover`, `deferredAckBacklog`, `deferredAckRetries`, `deferredAckExpired`, `recoveryReplay`, `duplicateDelivery`, `pollDuration`, `pollTimeout`.
   - Each metric has a documented alert rule in `docs/production-readiness.md` section 7 (new).
   
   ## Acceptance criteria
   
   - All 5 chaos test cases are green in CI.
   - Rolling upgrade test runs in CI with 2 instances for 5 minutes, asserts the documented invariants.
   - Admin endpoints return 401 / 403 on missing / wrong token; default config is fail-closed.
   - All 11 new metrics are emitted by the runtime; the test asserts their existence on the `/metrics` endpoint.
   - `docs/control-plane.md` and `docs/production-readiness.md` are updated.
   
   ## Verification
   
   ```bash
   ./gradlew :eventmesh-testcontainers:test --tests "*ChaosTest*"
   ./gradlew :eventmesh-testcontainers:test --tests "*RollingUpgradeTest*"
   ./gradlew :eventmesh-runtime:test --tests "*AdminSecurityTest*"
   ```
   
   ## Depends on / blocks
   
   - Depends on: #5362 (the cross-instance Testcontainers harness is the substrate for the chaos tests).
   - Blocks: #5364 (docs PR cannot close until the chaos + admin work has links to concrete commits + test runs).
   
   ## References
   
   - `eventmesh-testcontainers/` (from #5362)
   - `eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/admin/UniAdminServer.java`
   - `eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/offset/OffsetStore.java`
   - `eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/metrics/`
   - `docs/control-plane.md`
   - `docs/production-readiness.md`
   
   Part of the production-HA topology in #5354. See also #5352 and #5353.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org

Re: [I] [P1] chaos (MetaStore / broker / watch / RocksDB) + rolling upgrade + admin security (eventmesh)

["qqeasonchen (via GitHub)" <gi...@apache.org>]

qqeasonchen closed issue #5364: [P1] chaos (MetaStore / broker / watch / RocksDB) + rolling upgrade + admin security
URL: https://github.com/apache/eventmesh/issues/5364


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@eventmesh.apache.org
For additional commands, e-mail: issues-help@eventmesh.apache.org