You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/04/27 08:39:05 UTC
(skywalking) branch feature/runtime-rule-hot-update updated: docs(changes): rewrite Project section in user-perspective voice
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch feature/runtime-rule-hot-update
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/feature/runtime-rule-hot-update by this push:
new d75da8dff0 docs(changes): rewrite Project section in user-perspective voice
d75da8dff0 is described below
commit d75da8dff0651ceea1319f0bb9f251e1d8821d7d
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Mon Apr 27 16:38:57 2026 +0800
docs(changes): rewrite Project section in user-perspective voice
Reframe the runtime-rule, schema-drift, and BanyanDB schema-fence entries to
say what an operator can do now and what they get, rather than narrating the
internal mechanism. The technical reference for endpoints, lifecycle states,
self-heal, and split-brain handling already lives in
docs/en/setup/backend/backend-runtime-rule-api.md and the design doc — the
changelog now points at them instead of duplicating them.
---
docs/en/changes/changes.md | 78 +++++++++++++++++++++++-----------------------
1 file changed, 39 insertions(+), 39 deletions(-)
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 8ccc3d4b74..7a3816d9ca 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -2,47 +2,47 @@
#### Project
-* Runtime rule hot-update: change MAL and LAL rule files without restarting OAP. A new admin HTTP endpoint
- (port `17128`, **disabled by default**) accepts `POST /runtime/rule/addOrUpdate?catalog=&name=` with the
- rule file as the raw request body and persists it into the configured storage backend. Every node in the cluster
- converges to the new content within one reconciler tick (default 30 s). Lifecycle is strictly
- `ACTIVE → /inactivate → INACTIVE → /delete → absent`: `/inactivate` is the soft-pause path (broadcasts
- `Suspend`, tears down OAP-internal state — handlers, prototypes, applied bookkeeping — but preserves the
- BanyanDB measure and its data) and `/delete` is the destructive endpoint (requires the rule to already be
- `INACTIVE`, returns `HTTP 409 requires_inactivate_first` otherwise; drops the backend measure, then removes
- the tombstone row). Reactivation via `/addOrUpdate` against an `INACTIVE` row reuses the preserved measure
- and is therefore lossless. `GET /runtime/rule?catalog=&name=` fetches a single rule's YAML (raw or JSON
- envelope on `Accept: application/json`) with `ETag`/`If-None-Match` support; `GET /runtime/rule/bundled?catalog=`
- lists every static (bundled) rule for the catalog with an `overridden` flag so UIs can render the runtime
- + static merge. `GET /runtime/rule/list` reports the current state of every known rule (running,
- suspended, skipped, overridden) in NDJSON; `GET /runtime/rule/dump` returns a tar.gz archive for backup / DR.
- A bundled operator CLI at `tools/runtime-rule.sh` wraps the API. Cluster coordination uses a single-direction
- `Suspend` RPC plus a paired `Resume` and a 60 s self-heal timer so aborted or crashed updates roll back
- automatically; concurrent writes to different nodes during a brief topology flip are detected as
- split-brain and return `HTTP 409 split_brain_detected`. DB-backed overrides take effect at the next OAP boot
- via a classpath-discovered extension point consulted by the MAL and LAL engines, so restarts never regress to
- the static file content. **Security notice:** the admin port has no authentication — operators must
- gateway-protect it with IP allow-lists and never expose it to the public internet. See
- `docs/en/setup/backend/backend-runtime-rule-api.md` for the full operator guide.
-* **Boot-time schema-drift handling changed.** When OAP boots and finds that the storage backend holds a metric
- under the same name but a different shape than the current static / override rule declares, the metric is now
- **skipped with an ERROR log line** (backend shape vs. declared shape, with remediation hint) and OAP continues to
- boot. Previously this situation was silently tolerated on BanyanDB and JDBC (`ALREADY_EXISTS` was swallowed,
- leading to runtime drift and lost samples) and caused a hard boot failure on Elasticsearch field-type conflicts.
- All three backends now report the drift explicitly and leave the affected metric unregistered until the operator
- reshapes it via `POST /runtime/rule/addOrUpdate` (the on-demand workflow, which is the only path that changes
- backend schema). Skipped metrics are also surfaced in `GET /runtime/rule/list` under a new `bootSkippedReason`
- field so operators can audit them without grepping logs.
+* **Runtime rule hot-update for MAL and LAL.** Operators can now ship metric (MAL) and log
+ (LAL) rule changes without restarting OAP. A push to a new admin endpoint persists the rule
+ to the configured storage backend, and every node in the cluster converges to the new
+ content within ~30 seconds. Common workflows:
+ * `addOrUpdate` — create or replace a rule. Body is the raw YAML you would normally ship in
+ `application.yml`'s rule directories. Returns 200 once the rule is applied locally and
+ persisted; peers pick it up on their next reconciler tick.
+ * `inactivate` — soft-pause a rule. The OAP stops emitting metrics for that rule but the
+ backend measure (and its history) is preserved, so a later `addOrUpdate` to the same
+ `(catalog, name)` is lossless. This is the safe way to take a rule offline.
+ * `delete` — destructive removal. Requires the rule to already be `INACTIVE` (otherwise
+ returns 409); drops the backend measure and removes the row.
+ * `get` / `bundled` / `list` / `dump` — read-side endpoints for fetching a single rule's
+ YAML (with `ETag` support), listing the static-vs-runtime overlay per catalog,
+ inspecting cluster-wide rule state in NDJSON, and exporting all rules as a tar.gz for
+ backup / DR.
+ Hot-updates survive OAP restart: at boot the MAL and LAL engines merge the disk baseline
+ with persisted runtime rules, so the cluster never silently regresses to the bundled
+ defaults. A bundled operator CLI at `tools/runtime-rule.sh` wraps the API for shell use.
+ **The endpoint is disabled by default and listens on port `17128` when enabled. It has
+ no built-in authentication — operators must gateway-protect it with IP allow-lists and
+ never expose it to the public internet.** Full setup, security guidance, and the
+ endpoint reference are in `docs/en/setup/backend/backend-runtime-rule-api.md`.
+* **Schema-drift is now visible at boot, not silent.** If the storage backend already holds
+ a metric whose shape doesn't match what the current rule declares (e.g., a rule was edited
+ on disk while OAP was offline), OAP now skips the affected metric, logs an ERROR with the
+ declared-vs-backend diff, and continues booting. Previously BanyanDB and JDBC silently
+ accepted the divergence (samples were quietly dropped) and Elasticsearch failed the boot
+ outright. Skipped metrics surface in `GET /runtime/rule/list` under a new
+ `bootSkippedReason` field so they're easy to audit. To re-shape a drifted metric, push the
+ desired YAML through `POST /runtime/rule/addOrUpdate` — that is the only path that
+ changes backend schema, and it runs through the verified hot-update flow described above.
* Bump infra-e2e to testcontainers-go v0.42.0 (apache/skywalking-infra-e2e#146), which uses Docker Compose v2 plugin natively and removes docker-compose v1 dependency.
* Remove deprecated `version` field from all docker-compose files for Compose v2 compatibility.
-* **BanyanDB schema-visibility fence.** Replaces the metadata-poll verify path with the new `SchemaBarrierService`
- RPCs from `banyandb-client-proto` (`AwaitRevisionApplied` after Create / Update; `AwaitSchemaDeleted` after a
- Delete that returned `mod_revision == 0`). Every install / drop now blocks until every BanyanDB data node has
- caught up to the etcd `mod_revision` of the write, then returns. Combined with the runtime-rule
- `localSuspend` / `localResume` bracket around an apply, this enforces a clean **t0..tn data boundary**:
- samples written at or after the operator's `200 OK` use the new shape; samples written before use the old
- shape. The previous "poll `findMeasure` until you can read your own write" loop is removed (registry-side
- visibility never implied data-node visibility), as is `PostApplyVerifier`'s 5 s retry budget.
+* **Stronger consistency guarantee for BanyanDB schema changes.** Adopting the new
+ `SchemaBarrierService` API in BanyanDB, every schema install or drop driven by OAP now
+ waits until every BanyanDB data node has actually caught up to the change before
+ returning. For runtime hot-updates this means the operator's `200 OK` is the moment the
+ cluster's data boundary moves: samples written from that point on use the new shape, and
+ samples written before still go to the old shape — no overlap, no rejected writes during
+ the cutover.
#### OAP Server
* Add Zipkin Virtual GenAI e2e test. Use `zipkin_json` exporter to avoid protobuf dependency conflict