You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/04/26 10:50:32 UTC
(skywalking) branch feature/runtime-rule-hot-update updated: docs: BanyanDB schema-watch fence (runtime-rule + CLAUDE.md + changes)
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch feature/runtime-rule-hot-update
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/feature/runtime-rule-hot-update by this push:
new 306f2934bc docs: BanyanDB schema-watch fence (runtime-rule + CLAUDE.md + changes)
306f2934bc is described below
commit 306f2934bc6f76981bd65518ba450c9a9951a9cb
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Sun Apr 26 18:50:19 2026 +0800
docs: BanyanDB schema-watch fence (runtime-rule + CLAUDE.md + changes)
Documents the replacement of the metadata-poll verify path with
SchemaBarrierService.AwaitRevisionApplied / AwaitSchemaDeleted.
- runtime-rule-hot-update.md gets a "Schema-visibility fence (BanyanDB)"
subsection under Consistency bounds, covering the three RPCs, when each
applies, and the t0..tn data boundary the fence enforces. The
step-by-step structural workflow's "Verify" step now describes the
single isExists() (no retry) instead of the gone polling loop.
- CLAUDE.md gains tip #16: fence on mod_revision, do not poll metadata.
Names the SchemaWatcher accessor and the AwaitSchemaDeleted fallback
for deletes that returned mod_revision == 0. Calls out that JDBC and
ES don't need a fence.
- changes/changes.md (Project, 10.5.0) describes the user-visible
effect: clean t0..tn data boundary, "samples written at or after
200 OK use the new shape", removal of the 5s retry budget.
No code changes in this commit.
---
CLAUDE.md | 1 +
docs/en/changes/changes.md | 8 +++
.../runtime-rule-hot-update.md | 65 ++++++++++++++++++++--
3 files changed, 69 insertions(+), 5 deletions(-)
diff --git a/CLAUDE.md b/CLAUDE.md
index 21c00be545..0ffb3fef76 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -267,6 +267,7 @@ Actions owned by `actions/*` (GitHub), `github/*`, and `apache/*` are always all
13. **`moduleManager.find(X.NAME)` requires `X.NAME` in `requiredModules()`**: every call to `moduleManager.find(SomeModule.NAME)` (direct or through a helper) must have `SomeModule.NAME` in the provider's `requiredModules()` array. Missing declarations cause runtime exceptions the first time the code path fires — not at module boot. Wrapping the call in `try { ... } catch (Throwable)` is NOT a substitute; declare the module and keep the try/catch only for defensive handling of transient [...]
14. **Don't look up `ClusterModule` services directly**: the `ClusterModule` (ZooKeeper / K8s / Nacos coordination) exposes `ClusterRegister` / `ClusterNodesQuery` / `ClusterCoordinator`. Most receiver / analyzer modules don't declare `ClusterModule` in `requiredModules()`, so calling `moduleManager.find(ClusterModule.NAME)` will throw at runtime. Instead, go through `CoreModule`'s `RemoteClientManager` service — it's already populated by the cluster module and exposes the peer list ever [...]
15. **No `ThreadLocal` side-channels to hijack downstream behaviour**: routing a caller's intent through a `ThreadLocal` that downstream code reads (e.g., `if (PeerMode.isActive()) skipSomething()`) is almost always the wrong answer — it creates invisible coupling between far-apart code paths, leaks across async hand-offs (executors, gRPC threads, Armeria event loops), and makes the behaviour impossible to understand from a method signature. The correct fix is almost always to **extend t [...]
+16. **BanyanDB schema-visibility: fence on `mod_revision`, do NOT poll metadata**: every BanyanDB Create / Update / Delete returns an etcd `mod_revision` (0 on a delete that didn't record a tombstone). After firing DDL, fence on `BanyanDBClient.getSchemaWatcher().awaitRevisionApplied(maxRev, timeout)` before unparking dispatch / firing data writes — this blocks until every data node has caught up, which the registry's read-after-write does not guarantee. For deletes that returned `mod_re [...]
## Analysis and Design Principles
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index fc5a28b09a..8ccc3d4b74 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -35,6 +35,14 @@
field so operators can audit them without grepping logs.
* Bump infra-e2e to testcontainers-go v0.42.0 (apache/skywalking-infra-e2e#146), which uses Docker Compose v2 plugin natively and removes docker-compose v1 dependency.
* Remove deprecated `version` field from all docker-compose files for Compose v2 compatibility.
+* **BanyanDB schema-visibility fence.** Replaces the metadata-poll verify path with the new `SchemaBarrierService`
+ RPCs from `banyandb-client-proto` (`AwaitRevisionApplied` after Create / Update; `AwaitSchemaDeleted` after a
+ Delete that returned `mod_revision == 0`). Every install / drop now blocks until every BanyanDB data node has
+ caught up to the etcd `mod_revision` of the write, then returns. Combined with the runtime-rule
+ `localSuspend` / `localResume` bracket around an apply, this enforces a clean **t0..tn data boundary**:
+ samples written at or after the operator's `200 OK` use the new shape; samples written before use the old
+ shape. The previous "poll `findMeasure` until you can read your own write" loop is removed (registry-side
+ visibility never implied data-node visibility), as is `PostApplyVerifier`'s 5 s retry budget.
#### OAP Server
* Add Zipkin Virtual GenAI e2e test. Use `zipkin_json` exporter to avoid protobuf dependency conflict
diff --git a/docs/en/concepts-and-designs/runtime-rule-hot-update.md b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
index 67f5c572e9..773582b80e 100644
--- a/docs/en/concepts-and-designs/runtime-rule-hot-update.md
+++ b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
@@ -207,11 +207,16 @@ starts; the node that actually runs the workflow below is always the current mai
metric system creates classes for new metrics, removes stale ones for retired
metrics, and the backend installer runs whatever DDL is implied (BanyanDB
`deleteMeasure` / `define`, ES `createIndex`, JDBC `CREATE TABLE IF NOT EXISTS`).
-7. **Verify.** Re-query the backend via each model's existence check and confirm the
- live shape matches what the rule declares. Polls briefly to cover metadata-propagation
- latency. If verify fails the workflow aborts, reinstalls the old bundle locally, and
- returns HTTP 500; peers self-heal within a minute because the runtime-rule entry
- was never advanced (see workflow 3).
+7. **Verify.** A single existence check per model — no retry. The BanyanDB installer
+ internally fences on `SchemaBarrierService.AwaitRevisionApplied(maxModRevision)`
+ before returning, so by the time DDL hands control back to the apply pipeline
+ every BanyanDB data node has observed the new schema. ES and JDBC are
+ synchronous-DDL on the coordinator and likewise visible immediately. The previous
+ metadata-poll retry loop existed because the install path returned before
+ propagation completed; the schema-watch fence makes it redundant. If verify fails
+ the workflow aborts, reinstalls the old bundle locally, and returns HTTP 500;
+ peers self-heal within a minute because the runtime-rule entry was never advanced
+ (see workflow 3).
8. **Persist.** Upsert the runtime-rule entry with the new content and
`status=ACTIVE`. **This is the cluster-wide commit point.** Until this write
succeeds, peers that received the suspend broadcast stay suspended on the old
@@ -317,6 +322,56 @@ Formally: every node's in-memory bundle for a given `(catalog, name)` either equ
the current runtime-rule entry's content, or it is suspended and in the process of
transitioning toward it. No CAS, no quorum, no leader election.
+### Schema-visibility fence (BanyanDB)
+
+BanyanDB's distributed mode propagates registry writes from the meta-server to every
+data node asynchronously. A naive flow — `client.define(measure)` returns → caller
+proceeds to `verifyPostApply` → caller resumes dispatch — has a race: the registry
+holds the new measure, but a data node may not yet have caught up, so the first
+sample dispatched after the apply lands on an unprepared node and the write returns
+`NOT_FOUND`.
+
+The `SchemaBarrierService` proto (introduced in BanyanDB main, schema-consistency
+Phase 1+) closes the gap. Every Create / Update / Delete response now carries an
+etcd `mod_revision`, and three blocking RPCs let the client wait for cluster-wide
+visibility:
+
+| RPC | Use |
+|-----|-----|
+| `AwaitRevisionApplied(min_revision, timeout)` | Block until every data node has observed `mod_revision >= min_revision`. The etcd revision is global, so a single fence covers every write performed under it. |
+| `AwaitSchemaApplied(keys, min_revisions, timeout)` | Per-key fence. Useful when only specific resources need confirmation. |
+| `AwaitSchemaDeleted(keys, timeout)` | Tombstone-fanout fence. Use after a Delete that returned `mod_revision == 0` (server didn't record a tombstone), where the revision-based fence would never observe the deletion. |
+
+The runtime-rule install path captures `mod_revision` from every BanyanDB registry
+write into `StorageManipulationOpt.recordModRevision(...)` and accumulates the
+maximum. After the last DDL fires for a model, `BanyanDBIndexInstaller` calls
+`SchemaWatcher.awaitRevisionApplied(maxRev, 10s)` before returning. The previous
+`PostApplyVerifier` polling helper is gone — reading registry metadata to confirm
+"my own write" was a half-measure (registry-side visibility doesn't imply data-node
+visibility) and it cost up to 5 s of artificial delay even on a healthy cluster.
+
+The fence enforces a clean **t0..tn data boundary** across a structural apply:
+
+1. `t0` — REST main calls `localSuspend(catalog, name)` and broadcasts `Suspend`.
+ Every node's dispatch for this bundle is parked. Samples in flight are buffered
+ or dropped according to backend policy.
+2. `t0..tn` — The apply pipeline fires DDL on the registry (define, update, drop),
+ each call returning a `mod_revision`. The installer then calls
+ `AwaitRevisionApplied(maxRev)` and blocks until every data node has caught up,
+ or `AwaitSchemaDeleted(key)` for deletes that didn't get a tombstone.
+3. `tn` — Fence returns. The apply completes. REST handler calls `localResume`
+ and broadcasts `Resume`. Every node's dispatch re-opens.
+
+Data dispatched between t0 and tn cannot reach the backend (dispatch is suspended).
+Data dispatched after tn lands on a backend whose schema is fully visible across
+every data node — operators can rely on the timeline that "samples written at or
+after the operator's `200 OK` use the new shape, samples written before use the
+old shape."
+
+ES and JDBC don't have multi-node schema fan-out — DDL is synchronous on the
+coordinator and visible immediately on return. Their installers don't need a
+fence; the no-retry verify is correct.
+
## Distributed cluster model
The feature runs on any OAP cluster coordinator (Zookeeper / Kubernetes / Standalone /