You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/04/26 09:29:52 UTC
(skywalking) branch feature/runtime-rule-hot-update updated: runtime-rule: LOCAL_CACHE_VERIFY mode + flag-based StorageManipulationOpt
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch feature/runtime-rule-hot-update
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/feature/runtime-rule-hot-update by this push:
new afedfa70b5 runtime-rule: LOCAL_CACHE_VERIFY mode + flag-based StorageManipulationOpt
afedfa70b5 is described below
commit afedfa70b5b49d4f4bac210b058013203ac2d61c
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Sun Apr 26 17:29:40 2026 +0800
runtime-rule: LOCAL_CACHE_VERIFY mode + flag-based StorageManipulationOpt
Add a fourth StorageManipulationOpt mode for the boot pass on a non-init
OAP and refactor every install / remove call site onto a per-mode flag
profile so adding modes no longer requires auditing scattered if/else
chains. Wire the runtime-rule reconciler's boot pass to localCacheVerify
so missing or shape-mismatched backend schema fails the bootstrap (k8s
pod backloop) instead of silently registering local workers against a
backend the init OAP hasn't prepared yet.
The Mode enum carries an immutable Flags record (inspectBackend,
createMissing, updateOnMismatch, dropOnRemoval, failOnAbsence,
failOnShapeMismatch, escalateToCaller). ModelInstaller, BanyanDBIndexInstaller
checks, and MeterSystem.removeMetric all branch on individual flags
instead of mode predicates; predicates remain as thin wrappers so
existing tests compile unchanged.
Reconciler.tick gains an atBoot variant; the synchronous boot pass from
RuntimeRuleModuleProvider.notifyAfterCompleted opts into
localCacheVerify only in no-init mode and re-throws as ModuleStartException
on failure. Periodic ticks stay on the lenient localCacheOnly retry path.
Docs updated: design doc adds the inactive-rule-still-holds-names
section; backend-runtime-rule-api fixes the duplicate header row.
---
.../runtime-rule-hot-update.md | 20 +++
docs/en/setup/backend/backend-runtime-rule-api.md | 6 +-
.../server/core/analysis/meter/MeterSystem.java | 9 +-
.../server/core/storage/model/ModelInstaller.java | 75 +++++++--
.../core/storage/model/StorageManipulationOpt.java | 182 ++++++++++++++++++++-
.../module/RuntimeRuleModuleProvider.java | 18 +-
.../receiver/runtimerule/reconcile/Reconciler.java | 66 +++++---
.../plugin/banyandb/BanyanDBIndexInstaller.java | 102 +++++++++---
8 files changed, 396 insertions(+), 82 deletions(-)
diff --git a/docs/en/concepts-and-designs/runtime-rule-hot-update.md b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
index 552457dfca..67f5c572e9 100644
--- a/docs/en/concepts-and-designs/runtime-rule-hot-update.md
+++ b/docs/en/concepts-and-designs/runtime-rule-hot-update.md
@@ -547,6 +547,26 @@ a transient DAO read failure surfaces a 503 to the operator, never as "no prior
treat as first-time create" — the latter would let a STRUCTURAL change bypass the
`allowStorageChange` guardrail because the prior content read as null.
+### Inactive rules still hold their names
+
+The cross-file ownership guards (MAL metric names; LAL `(layer, ruleName)` keys) check
+two sources before accepting a new file:
+
+1. **In-memory `appliedMal` / `appliedLal`** — covers active runtime + boot-seeded
+ static rules.
+2. **INACTIVE rows in storage** — `/inactivate` clears the in-memory entry but
+ preserves the row's content and status. The inactive rule still HOLDS its names per
+ the soft-pause contract; a new file claiming any of those names is rejected with
+ "held by inactive `<key>` (update or /delete that rule first)".
+
+The operator's recourse is to update the inactive rule (a re-`/addOrUpdate` of the same
+file) or `/delete` it before reusing its names elsewhere. Without this, a sequence
+like *(A active with `foo`) → /inactivate A → /addOrUpdate B with `foo` → /delete A*
+would tear down the BanyanDB measure that B is now using, because `/delete` re-registers
+A's metrics locally and drives the destructive cascade by name. `/delete` adds a
+defense-in-depth check (refuses to drop if any of A's metric names are owned by another
+active bundle) so a race or DAO blip can't bypass the primary check.
+
### Inactivate is soft-pause; delete is destructive
`/inactivate` and `/delete` are split so that deactivating a rule never destroys data
diff --git a/docs/en/setup/backend/backend-runtime-rule-api.md b/docs/en/setup/backend/backend-runtime-rule-api.md
index 5af511fdb0..6f94967a06 100644
--- a/docs/en/setup/backend/backend-runtime-rule-api.md
+++ b/docs/en/setup/backend/backend-runtime-rule-api.md
@@ -78,10 +78,8 @@ delta-classifier compile step and the response is `400 compile_failed`.
### Canonical routes
-| Method | Path | Body | Effect |
-|---|---|---|---|
-| Method | Path | Body | Effect |
-|--------|---------------------------------------------------------------------|---------------|-----------------------------------------------------------------------------|
+| Method | Path | Body | Effect |
+|--------|----------------------------------------------------------------------|---------------|-----------------------------------------------------------------------------|
| POST | `/runtime/rule/addOrUpdate?catalog=&name=[&allowStorageChange=true]` | raw rule YAML | compile, classify, apply locally with Suspend + DDL + verify, persist row |
| POST | `/runtime/rule/fix?catalog=&name=` | raw rule YAML | recovery — same flow as addOrUpdate with `allowStorageChange=true` forced |
| POST | `/runtime/rule/inactivate?catalog=&name=` | empty | soft-pause — broadcasts Suspend, tears down OAP-internal state, preserves backend measure, persists INACTIVE |
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/MeterSystem.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/MeterSystem.java
index 4fe03742be..c3ba7b9247 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/MeterSystem.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/MeterSystem.java
@@ -465,15 +465,16 @@ public class MeterSystem implements Service {
modelCreator.remove(prototypeClass, opt);
} catch (final Throwable t) {
log.error("Failed to cascade storage-model removal for metric {}", metricsName, t);
- if (opt.isFullInstall()) {
+ if (opt.getFlags().isEscalateToCaller()) {
throw new RuntimeException(
"Storage-model cascade failed for metric " + metricsName
+ "; backend drop did not complete. Local state preserved for retry.",
t);
}
- // localCacheOnly: peer-side caller — backend drop is suppressed by policy
- // anyway, so a listener throw here is the listener's local bookkeeping
- // misbehaving, not real backend debt. Fall through to clear local state.
+ // Non-escalating opt (peer-side localCacheOnly, etc.) — backend drop is
+ // suppressed by policy anyway, so a listener throw here is the listener's
+ // local bookkeeping misbehaving, not real backend debt. Fall through to
+ // clear local state.
}
// Backend cascade succeeded (or local-cache-only, where it doesn't matter). Drop
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelInstaller.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelInstaller.java
index 30b381ba55..233acf91ee 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelInstaller.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelInstaller.java
@@ -48,6 +48,50 @@ public abstract class ModelInstaller implements ModelRegistry.CreatingListener,
@Override
public void whenCreating(Model model, StorageManipulationOpt opt) throws StorageException {
+ final StorageManipulationOpt.Flags flags = opt.getFlags();
+
+ // Zero server RPCs — peer-side ticks. The earlier order called isExists first
+ // (which on ES/JDBC fires a backend read) and only then checked the policy,
+ // which made the contract a half-truth. Gate ahead of isExists so a peer apply
+ // is genuinely zero-RPC.
+ if (!flags.isInspectBackend()) {
+ opt.recordOutcome("table", model.getName(),
+ StorageManipulationOpt.Outcome.SKIPPED_NOT_ALLOWED,
+ "local-cache-only mode; main-node is expected to have installed this resource");
+ log.debug(
+ "install: model [{}] not installed; local-cache-only mode — skipping (no isExists probe)",
+ model.getName()
+ );
+ return;
+ }
+
+ // Strict verify path — run the read-only existence/shape inspection and surface
+ // missing or mismatched resources as fatal so module bootstrap exits (k8s pod
+ // backloop). Operator must align with the init OAP first. Distinct from the
+ // legacy non-init poll loop further down: that loop waits forever; this path
+ // fails fast.
+ if (flags.isFailOnAbsence() || flags.isFailOnShapeMismatch()) {
+ InstallInfo info = isExists(model, opt);
+ if (flags.isFailOnShapeMismatch() && opt.hasShapeMismatch()) {
+ final StorageManipulationOpt.ResourceOutcome o = opt.firstShapeMismatch();
+ throw new StorageException(
+ "local-cache-verify boot: backend resource '" + (o == null ? model.getName() : o.getResourceName())
+ + "' shape diverges from declared model — refusing to start. "
+ + "Reconcile via the init OAP's /runtime/rule/addOrUpdate first. diff: "
+ + (o == null ? "n/a" : o.getDiff()));
+ }
+ if (flags.isFailOnAbsence() && !info.isAllExist()) {
+ throw new StorageException(
+ "local-cache-verify boot: backend resources for model '" + model.getName()
+ + "' are not all present — refusing to start. Wait for the init OAP to "
+ + "create them or push the runtime rule. " + info.buildInstallInfoMsg());
+ }
+ return;
+ }
+
+ // Legacy poll loop for non-init OAPs that did not opt into the strict verify
+ // mode. Static models (boot-time) still take this path; runtime-rule reconciler
+ // explicitly chooses verify so this loop is bypassed.
if (RunningMode.isNoInitMode()) {
while (true) {
InstallInfo info = isExists(model, opt);
@@ -67,26 +111,21 @@ public abstract class ModelInstaller implements ModelRegistry.CreatingListener,
}
return;
}
- // Honour the localCacheOnly contract — "no server RPCs". The earlier order called
- // isExists first (which on ES/JDBC fires a backend read) and only then checked the
- // policy, which made the contract a half-truth: peer-side ticks did issue read
- // RPCs to the storage even though they could never act on the result. Move the
- // policy gate ahead of isExists so a peer apply under localCacheOnly is genuinely
- // zero-RPC.
- if (opt.isLocalCacheOnly()) {
- opt.recordOutcome("table", model.getName(),
- StorageManipulationOpt.Outcome.SKIPPED_NOT_ALLOWED,
- "local-cache-only mode; main-node is expected to have installed this resource");
- log.debug(
- "install: model [{}] not installed; local-cache-only mode — skipping (no isExists probe)",
- model.getName()
- );
- return;
- }
+
InstallInfo info = isExists(model, opt);
if (info.isAllExist()) {
return;
}
+ if (!flags.isCreateMissing()) {
+ // Inspect-but-don't-create: caller wants existence reported as outcome but
+ // explicitly forbids DDL. Today no canonical mode hits this branch, but the
+ // flag combination is valid (e.g. dry-run reporting) and falling through to
+ // createTable would silently violate the contract.
+ opt.recordOutcome("table", model.getName(),
+ StorageManipulationOpt.Outcome.MISSING,
+ "missing on backend; createMissing flag is off — skipping DDL");
+ return;
+ }
log.info(
"install info: {}. table for model: [{}] not all required resources exist, creating or updating...",
info.buildInstallInfoMsg(), model.getName()
@@ -98,10 +137,10 @@ public abstract class ModelInstaller implements ModelRegistry.CreatingListener,
@Override
public void whenRemoving(Model model, StorageManipulationOpt opt) throws StorageException {
- if (opt.isLocalCacheOnly()) {
+ if (!opt.getFlags().isDropOnRemoval()) {
opt.recordOutcome("table", model.getName(),
StorageManipulationOpt.Outcome.SKIPPED_NOT_ALLOWED,
- "local-cache-only mode; peer-side local teardown, server drop is main-node responsibility");
+ "dropOnRemoval flag is off; server drop is main-node responsibility (or boot path that never drops)");
return;
}
dropTable(model);
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageManipulationOpt.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageManipulationOpt.java
index 9acd12990d..dbf03bc264 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageManipulationOpt.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageManipulationOpt.java
@@ -21,6 +21,7 @@ package org.apache.skywalking.oap.server.core.storage.model;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
+import lombok.Builder;
import lombok.Getter;
/**
@@ -31,8 +32,8 @@ import lombok.Getter;
* rule, binding, template, etc.).
*
* <h3>Canonical profiles — always use a named factory</h3>
- * Three modes, each matching one distinct caller scenario. Use the factories; the
- * constructor is private. If a future scenario genuinely needs a fourth mode, add it to
+ * Four modes, each matching one distinct caller scenario. Use the factories; the
+ * constructor is private. If a future scenario genuinely needs a fifth mode, add it to
* {@link Mode} here so every caller keeps picking from a known set.
*
* <h4>{@link #fullInstall()} — {@link Mode#FULL_INSTALL} (predicate: {@link #isFullInstall()})</h4>
@@ -62,6 +63,24 @@ import lombok.Getter;
* skip surfaces the drift to the operator, who must reshape via the on-demand
* runtime-rule REST endpoint (the only workflow that may change backend schema).
*
+ * <h4>{@link #localCacheVerify()} — {@link Mode#LOCAL_CACHE_VERIFY} (predicate: {@link #isLocalCacheVerify()})</h4>
+ * <p>Callers:
+ * <ul>
+ * <li>Boot-time reconciler pass on a non-init OAP — the operator declared
+ * {@code init=false}, so this OAP must not perform DDL but must refuse to start if
+ * the backend isn't already in the shape the persisted runtime-rule catalog
+ * declares.</li>
+ * </ul>
+ * <p>Backend behaviour: read-only inspection. The installer issues the same metadata
+ * read RPCs as {@link Mode#CREATE_IF_ABSENT} but never invokes create / update / drop. On
+ * resource missing OR shape mismatch the installer throws — the exception propagates up
+ * through the module bootstrap and causes the OAP process to exit, which under k8s
+ * results in a pod backloop until either the init OAP has caught up or the operator has
+ * fixed the rule files. This matches general OAP boot semantics for static models in
+ * non-init mode: the OAP will not silently start with a backend that disagrees with
+ * what's declared. Local {@code MetadataRegistry} is populated only when the live shape
+ * matches the declared shape.
+ *
* <h4>{@link #localCacheOnly()} — {@link Mode#LOCAL_CACHE_ONLY} (predicate: {@link #isLocalCacheOnly()})</h4>
* <p>Callers:
* <ul>
@@ -74,7 +93,10 @@ import lombok.Getter;
* {@link Outcome#SKIPPED_NOT_ALLOWED SKIPPED_NOT_ALLOWED} outcomes instead of firing
* {@code createTable} / {@code dropTable}. Peer's local MeterSystem still compiles
* Metrics classes and populates {@code meterPrototypes} — that's pure in-JVM work the
- * opt doesn't (and shouldn't) gate.
+ * opt doesn't (and shouldn't) gate. Differs from {@link Mode#LOCAL_CACHE_VERIFY} in two
+ * ways: no server RPCs (cache populates from local model), and missing / mismatched
+ * resources are <strong>not</strong> a fatal error (the next tick will retry, or the
+ * main will catch up).
*
* <h3>Why it's a single mutable object instead of separate policy/result</h3>
* Installers can nest many resource operations per Model (BanyanDB Measure + N index rules
@@ -90,12 +112,131 @@ public final class StorageManipulationOpt {
* scenario each mode covers.
*/
public enum Mode {
- /** Main-node on-demand path: installer performs full DDL including reshape. */
- FULL_INSTALL,
- /** Static boot path: create absent resources; skip + warn on shape mismatch. */
- CREATE_IF_ABSENT,
- /** Peer-node reconcile path: no server RPCs; populate local caches only. */
- LOCAL_CACHE_ONLY
+ /**
+ * Main-node on-demand path. Installer performs full DDL: create absent resources,
+ * detect shape mismatch and apply the additive subset each backend supports
+ * online ({@code client.update} for BanyanDB, add-column for JDBC, mapping append
+ * for ES). Reshape is treated as intended because the caller explicitly asked
+ * for it via the operator REST endpoint.
+ */
+ FULL_INSTALL(Flags.builder()
+ .inspectBackend(true)
+ .createMissing(true)
+ .updateOnMismatch(true)
+ .dropOnRemoval(true)
+ .escalateToCaller(true)
+ .build()),
+ /**
+ * Static boot path on an init-mode OAP. Installer creates absent resources, but
+ * if a resource already exists with a shape that diverges from the declared
+ * model it records {@link Outcome#SKIPPED_SHAPE_MISMATCH} and does <strong>not</strong>
+ * call update / reshape. Operator must reconcile via the runtime-rule REST
+ * endpoint — boot is not allowed to silently mutate backend shape.
+ */
+ CREATE_IF_ABSENT(Flags.builder()
+ .inspectBackend(true)
+ .createMissing(true)
+ .build()),
+ /**
+ * Boot path on a non-init OAP. Installer issues the same read-only inspection
+ * RPCs as {@link #CREATE_IF_ABSENT} but never creates / updates / drops. On
+ * resource missing or shape mismatch the installer <strong>throws</strong>; the
+ * exception propagates up through module bootstrap and exits the process.
+ * Under k8s this causes a pod backloop until the init OAP has caught up or the
+ * operator has aligned rule files with the backend. Local {@code MetadataRegistry}
+ * is populated only when the live shape matches the declared shape.
+ */
+ LOCAL_CACHE_VERIFY(Flags.builder()
+ .inspectBackend(true)
+ .failOnAbsence(true)
+ .failOnShapeMismatch(true)
+ .build()),
+ /**
+ * Peer-node reconciler tick path. Zero server RPCs — local caches populate from
+ * the declared model and the main is trusted to own backend DDL. Missing or
+ * mismatched resources are not an error: the next tick will retry, and the main
+ * will eventually converge. Distinct from {@link #LOCAL_CACHE_VERIFY} in that
+ * verification is skipped entirely, not run-and-fail.
+ */
+ LOCAL_CACHE_ONLY(Flags.builder().build());
+
+ @Getter
+ private final Flags flags;
+
+ Mode(final Flags flags) {
+ this.flags = flags;
+ }
+ }
+
+ /**
+ * Per-mode behavioural flags. Each control point in the install / remove pipeline
+ * checks one flag instead of branching on the {@link Mode} value, so adding a new
+ * mode is a matter of choosing flag values rather than auditing every {@code if
+ * (opt.isXxx())} site. Flags are immutable and shared across all opts of the same
+ * mode.
+ *
+ * <p>Each flag describes a distinct privilege the installer is granted by the
+ * caller. They are independently composable on paper, but the canonical
+ * combinations live on {@link Mode} — call sites should never construct a
+ * {@code Flags} directly.</p>
+ */
+ @Builder
+ @Getter
+ public static final class Flags {
+ /**
+ * Issue read RPCs to the backend (existence + shape compare). False on
+ * {@link Mode#LOCAL_CACHE_ONLY} where the contract is "zero server RPCs". When
+ * false the installer must populate local caches from the declared model and
+ * return early without inspecting the backend.
+ */
+ private final boolean inspectBackend;
+ /**
+ * Call backend create primitives ({@code client.define}, JDBC {@code CREATE
+ * TABLE}, ES {@code createIndex}, BanyanDB {@code defineIndexRule} /
+ * {@code defineIndexRuleBinding}) when a resource is absent.
+ */
+ private final boolean createMissing;
+ /**
+ * Call backend update primitives ({@code client.update}, JDBC {@code ALTER
+ * TABLE}, ES mapping append) when a present resource's live shape diverges from
+ * the declared shape. Only {@link Mode#FULL_INSTALL} (the operator-driven path)
+ * permits this — boot must never silently reshape backend storage.
+ *
+ * <p>Note: BanyanDB's index-rule / index-rule-binding update path is gated by
+ * {@link #failOnShapeMismatch} instead of this flag, preserving the long-standing
+ * behaviour that init-mode OAPs reconcile index rules even under
+ * {@link Mode#CREATE_IF_ABSENT}.</p>
+ */
+ private final boolean updateOnMismatch;
+ /**
+ * Call backend drop primitives ({@code client.dropMeasure} / {@code dropStream}
+ * / etc.) from {@link ModelRegistry.CreatingListener#whenRemoving}. Only
+ * {@link Mode#FULL_INSTALL} (operator-driven runtime-rule deletion) permits
+ * this; peers under {@link Mode#LOCAL_CACHE_ONLY} short-circuit with
+ * {@link Outcome#SKIPPED_NOT_ALLOWED}.
+ */
+ private final boolean dropOnRemoval;
+ /**
+ * Throw a {@link org.apache.skywalking.oap.server.core.storage.StorageException}
+ * when a resource is absent on the backend after inspection. Used by
+ * {@link Mode#LOCAL_CACHE_VERIFY} to fail boot rather than silently start
+ * against an unprepared backend.
+ */
+ private final boolean failOnAbsence;
+ /**
+ * Throw a {@link org.apache.skywalking.oap.server.core.storage.StorageException}
+ * when a present resource's live shape diverges from the declared shape. Used
+ * by {@link Mode#LOCAL_CACHE_VERIFY} so boot does not silently start against a
+ * backend whose schema disagrees with the rule file.
+ */
+ private final boolean failOnShapeMismatch;
+ /**
+ * Re-throw cascaded backend errors to the caller (REST handler, operator
+ * tooling) instead of swallowing them. Set on {@link Mode#FULL_INSTALL}; other
+ * modes log and continue so a peer-side bookkeeping glitch doesn't take down
+ * the node.
+ */
+ private final boolean escalateToCaller;
}
@Getter
@@ -106,6 +247,16 @@ public final class StorageManipulationOpt {
* never see a torn list. */
private final List<ResourceOutcome> outcomes = new CopyOnWriteArrayList<>();
+ /**
+ * Behavioural flags for this opt. Convenience accessor — equivalent to
+ * {@code getMode().getFlags()}. Call sites read individual flags (e.g.
+ * {@code opt.getFlags().isCreateMissing()}) instead of pattern-matching on the
+ * {@link Mode}.
+ */
+ public Flags getFlags() {
+ return mode.getFlags();
+ }
+
public static StorageManipulationOpt fullInstall() {
return new StorageManipulationOpt(Mode.FULL_INSTALL);
}
@@ -114,6 +265,10 @@ public final class StorageManipulationOpt {
return new StorageManipulationOpt(Mode.CREATE_IF_ABSENT);
}
+ public static StorageManipulationOpt localCacheVerify() {
+ return new StorageManipulationOpt(Mode.LOCAL_CACHE_VERIFY);
+ }
+
public static StorageManipulationOpt localCacheOnly() {
return new StorageManipulationOpt(Mode.LOCAL_CACHE_ONLY);
}
@@ -135,6 +290,15 @@ public final class StorageManipulationOpt {
return mode == Mode.CREATE_IF_ABSENT;
}
+ /**
+ * True for {@link Mode#LOCAL_CACHE_VERIFY}. Boot-time strict verification on a
+ * non-init OAP — installer issues read-only inspection RPCs and throws on missing or
+ * shape-mismatched resources. No DDL.
+ */
+ public boolean isLocalCacheVerify() {
+ return mode == Mode.LOCAL_CACHE_VERIFY;
+ }
+
/**
* True for {@link Mode#LOCAL_CACHE_ONLY}. The {@code BanyanDBIndexInstaller.isExists}
* short-circuit reads this to skip every server RPC and populate
diff --git a/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/module/RuntimeRuleModuleProvider.java b/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/module/RuntimeRuleModuleProvider.java
index eebcae0ff5..9db4570394 100644
--- a/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/module/RuntimeRuleModuleProvider.java
+++ b/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/module/RuntimeRuleModuleProvider.java
@@ -229,9 +229,25 @@ public class RuntimeRuleModuleProvider extends ModuleProvider {
// rows once it does — better a short delay than a wrong "applied" log line.
if (isPeerListReadyNow()) {
try {
- reconciler.tick();
+ // atBoot=true so a no-init OAP picks localCacheVerify and refuses to
+ // start with a missing or shape-mismatched backend (k8s pod backloop)
+ // instead of silently registering local workers against schema that
+ // doesn't exist. Init / default-mode OAPs are unaffected — their boot
+ // opt mirrors the standard tick choice for those modes.
+ reconciler.tick(true);
log.info("Runtime rule reconciler: synchronous first tick completed "
+ "(runtime-only DB rows are now applied locally).");
+ } catch (final RuntimeException re) {
+ // Boot pass under localCacheVerify re-throws missing/mismatch as a
+ // RuntimeException so module bootstrap aborts. Translate to
+ // ModuleStartException so the OAP exit message points the operator at
+ // the right place.
+ throw new ModuleStartException(
+ "Runtime rule reconciler boot pass failed under localCacheVerify; "
+ + "the backend schema is missing or diverges from the declared rule. "
+ + "Bring up the init OAP first or align rule files with the backend, "
+ + "then restart this node.",
+ re);
} catch (final Throwable t) {
log.warn("Runtime rule reconciler: synchronous first tick failed — "
+ "runtime-only DB rows will be picked up on the scheduled tick.", t);
diff --git a/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/Reconciler.java b/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/Reconciler.java
index e70c46901a..e1a059a1d1 100644
--- a/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/Reconciler.java
+++ b/oap-server/server-receiver-plugin/skywalking-runtime-rule-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/runtimerule/reconcile/Reconciler.java
@@ -116,7 +116,7 @@ import org.apache.skywalking.oap.server.receiver.runtimerule.util.ContentHash;
* classes + registers local prototypes so samples dispatch correctly — that's a
* pure in-JVM operation.</li>
* </ul>
- * The tick path picks the opt via {@link #tickStorageOpt()} (looks up
+ * The tick path picks the opt via {@link #tickStorageOpt(boolean)} (looks up
* {@code RemoteClientManager} via {@code CoreModule}, asks {@code MainRouter.isSelfMain}).
*
* <h3>REST path — /addOrUpdate STRUCTURAL on the main node</h3>
@@ -414,11 +414,32 @@ public final class Reconciler {
* next tick proceeds from whatever state the last one left.
*/
public void tick() {
+ tick(false);
+ }
+
+ /**
+ * Variant invoked once at boot from {@code RuntimeRuleModuleProvider.notifyAfterCompleted}
+ * with {@code atBoot=true}. The boot pass on a no-init OAP picks
+ * {@link StorageManipulationOpt#localCacheVerify()} so missing or shape-mismatched
+ * backend schema fails the bootstrap (k8s pod backloop) instead of silently
+ * proceeding. The scheduled executor calls the no-arg overload so subsequent ticks
+ * stay on the lenient {@code localCacheOnly} retry path.
+ *
+ * <p>Boot semantics are scoped to no-init mode only — init-mode OAPs continue to
+ * pick {@link StorageManipulationOpt#createIfAbsent()} (boot creates), and
+ * default-mode OAPs continue to pick by cluster main-ness.
+ */
+ public void tick(final boolean atBoot) {
try {
sweepLoaderGraveyard();
sweepSuspendedForSelfHeal();
- applyDeltasFromDatabase();
+ applyDeltasFromDatabase(atBoot);
} catch (final Throwable t) {
+ if (atBoot) {
+ // Re-throw so the bootstrap aborts; pod backloops on k8s, operator sees
+ // the failure instead of silently starting against an unprepared backend.
+ throw new RuntimeException("runtime-rule reconciler boot pass failed", t);
+ }
log.error("runtime-rule reconciler tick failed; will retry on next interval", t);
}
}
@@ -516,7 +537,7 @@ public final class Reconciler {
* discard path is the recovery, not self-heal).
*
* <p>Bundles whose DB content HAS advanced since the suspend are left for
- * {@link #applyDeltasFromDatabase()} to pick up via the normal content-hash diff — those
+ * {@link #applyDeltasFromDatabase(boolean)} to pick up via the normal content-hash diff — those
* are the "main node succeeded, we're catching up" path. We deliberately do not flip
* those back to RUNNING here: the correct handlers for the new content haven't been
* installed yet, so a premature flip would resume dispatch against a bundle whose schema
@@ -655,7 +676,7 @@ public final class Reconciler {
* <p>Diff set is sorted by {@code (catalog, name)} before apply — the design's load-bearing
* invariant for deterministic LAL cross-file rule-name conflict resolution across nodes.
*/
- private void applyDeltasFromDatabase() {
+ private void applyDeltasFromDatabase(final boolean atBoot) {
final RuntimeRuleManagementDAO dao;
try {
dao = moduleManager.find(StorageModule.NAME).provider().getService(RuntimeRuleManagementDAO.class);
@@ -680,7 +701,7 @@ public final class Reconciler {
// mid-tick is no more authoritative than the first read, just a bit later. One
// snapshot per tick keeps the third-gate skip decision and the marker-update branch
// consistent with each other; the next tick gets a fresh observation.
- final StorageManipulationOpt tickOpt = tickStorageOpt();
+ final StorageManipulationOpt tickOpt = tickStorageOpt(atBoot);
final boolean tickRunsFullInstall = tickOpt.isFullInstall();
final Set<String> seenKeys = new HashSet<>();
final long nowMs = System.currentTimeMillis();
@@ -2398,7 +2419,7 @@ public final class Reconciler {
/**
* Post-apply DDL verify. Resolves the active backend's {@link ModelInstaller} and calls
- * {@link ModelInstaller#isExists(Model)} for every Model that matches one of the
+ * {@link ModelInstaller#isExists(Model, StorageManipulationOpt)} for every Model that matches one of the
* just-registered metric names (across all three downsampling variants — Hour/Day/Minute
* — that {@code MetricsStreamProcessor.create} registers per metric).
*
@@ -2531,9 +2552,19 @@ public final class Reconciler {
* rest of the static-rule install path does in init mode (idempotent against
* backends that already hold the table).
* <li>{@code no-init} mode — this OAP must NOT touch the backend; the init OAP
- * owns schema. Use {@link StorageManipulationOpt#localCacheOnly()} so the
- * reconciler's first tick still populates MeterSystem / MetadataRegistry from
- * the seeded static rules without firing a single server RPC.
+ * owns schema. The opt depends on whether this is the synchronous boot pass
+ * or a scheduled tick:
+ * <ul>
+ * <li><b>Boot pass</b> ({@code atBoot=true}) →
+ * {@link StorageManipulationOpt#localCacheVerify()}. Strict: backend
+ * resources must already exist with the declared shape. A missing or
+ * drifted schema fails the bootstrap (k8s pod backloop) — operator must
+ * bring up the init OAP first, or align rule files with the backend.
+ * <li><b>Scheduled tick</b> ({@code atBoot=false}) →
+ * {@link StorageManipulationOpt#localCacheOnly()}. Lenient: the timer
+ * retries forever without raising errors so transient absence (init OAP
+ * still catching up between ticks) self-heals.
+ * </ul>
* <li>default mode (regular running OAP) — branch on cluster main-ness, see below.
* </ul>
*
@@ -2551,21 +2582,18 @@ public final class Reconciler {
* MainRouter#isSelfMain} returns {@code true} and the default-mode branch falls
* through to {@code fullInstall} — single-process deployments are always main.
*
- * <p>Note: {@link StorageManipulationOpt#localCacheOnly()} does NOT verify the
- * target table exists. {@link
- * org.apache.skywalking.oap.server.core.storage.model.ModelInstaller#whenCreating
- * ModelInstaller.whenCreating} skips {@code isExists} under {@code localCacheOnly}
- * to honour the zero-RPC contract — a no-init OAP that boots before the init OAP
- * has installed the table will silently have no schema, and writes fail later. Same
- * trade-off the rest of the storage layer accepts; flagged in the design doc as a
- * follow-up if it bites.
+ * @param atBoot true for the synchronous one-shot pass invoked from
+ * {@code RuntimeRuleModuleProvider.notifyAfterCompleted}; false for
+ * scheduled-executor ticks.
*/
- private StorageManipulationOpt tickStorageOpt() {
+ private StorageManipulationOpt tickStorageOpt(final boolean atBoot) {
if (RunningMode.isInitMode()) {
return StorageManipulationOpt.createIfAbsent();
}
if (RunningMode.isNoInitMode()) {
- return StorageManipulationOpt.localCacheOnly();
+ return atBoot
+ ? StorageManipulationOpt.localCacheVerify()
+ : StorageManipulationOpt.localCacheOnly();
}
try {
final RemoteClientManager rcm = moduleManager.find(CoreModule.NAME).provider()
diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIndexInstaller.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIndexInstaller.java
index a01ffe1d00..32afa8e8ab 100644
--- a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIndexInstaller.java
+++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBIndexInstaller.java
@@ -81,13 +81,12 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
installInfo.setKind(metadata.getKind());
installInfo.setGroup(metadata.getGroup());
- // Peer-mode shortcut: when the caller is the runtime-rule peer-side tick
- // (localCacheOnly — no create, no drop, no shape update, no index define), the
- // cluster main has already installed the resource on BanyanDB. We just need to
- // populate the local MetadataRegistry so this peer's DAOs can translate this
- // Model for sample read/write. Zero server RPCs, no checkMeasure auto-update,
+ // Peer-mode shortcut: when the caller has inspectBackend=false the contract is
+ // "zero server RPCs". The cluster main has already installed the resource on
+ // BanyanDB; we just populate the local MetadataRegistry so this peer's DAOs
+ // can translate this Model for sample read/write. No checkMeasure auto-update,
// no race with main's recent work.
- if (opt.isLocalCacheOnly()) {
+ if (!opt.getFlags().isInspectBackend()) {
registerLocallyByKind(model, downSamplingConfigService);
installInfo.setGroupExist(true);
installInfo.setTableExist(true);
@@ -108,6 +107,11 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
installInfo.setAllExist(false);
return installInfo;
} else {
+ // Run shape-compat checks unless we're in the legacy no-init poll loop
+ // path. failOnAbsence implies the caller wants strict verification even
+ // in non-init mode (LOCAL_CACHE_VERIFY), so honour that instead of just
+ // gating on RunningMode.
+ final boolean runShapeChecks = !RunningMode.isNoInitMode() || opt.getFlags().isFailOnAbsence();
if (model.isTimeSeries()) {
// register models only locally(Schema cache) but not remotely
if (model.isRecord()) {
@@ -118,43 +122,43 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
installInfo.setAllExist(true);
return installInfo;
}
- if (!RunningMode.isNoInitMode()) {
+ if (runShapeChecks) {
checkTrace(traceModel.getTrace(), c, opt);
- checkIndexRules(model.getName(), traceModel.getIndexRules(), c);
+ checkIndexRules(model.getName(), traceModel.getIndexRules(), c, opt);
checkIndexRuleBinding(
traceModel.getIndexRules(), metadata.getGroup(), metadata.name(),
- BanyandbCommon.Catalog.CATALOG_TRACE, c
+ BanyandbCommon.Catalog.CATALOG_TRACE, c, opt
);
}
} else {
// stream
StreamModel streamModel = MetadataRegistry.INSTANCE.registerStreamModel(
model, config);
- if (!RunningMode.isNoInitMode()) {
+ if (runShapeChecks) {
checkStream(streamModel.getStream(), c, opt);
- checkIndexRules(model.getName(), streamModel.getIndexRules(), c);
+ checkIndexRules(model.getName(), streamModel.getIndexRules(), c, opt);
checkIndexRuleBinding(
streamModel.getIndexRules(), metadata.getGroup(), metadata.name(),
- BanyandbCommon.Catalog.CATALOG_STREAM, c
+ BanyandbCommon.Catalog.CATALOG_STREAM, c, opt
);
// Stream not support server side TopN pre-aggregation
}
}
} else { // measure
MeasureModel measureModel = MetadataRegistry.INSTANCE.registerMeasureModel(model, config, downSamplingConfigService);
- if (!RunningMode.isNoInitMode()) {
+ if (runShapeChecks) {
checkMeasure(measureModel.getMeasure(), c, opt);
- checkIndexRules(model.getName(), measureModel.getIndexRules(), c);
+ checkIndexRules(model.getName(), measureModel.getIndexRules(), c, opt);
checkIndexRuleBinding(
measureModel.getIndexRules(), metadata.getGroup(), metadata.name(),
- BanyandbCommon.Catalog.CATALOG_MEASURE, c
+ BanyandbCommon.Catalog.CATALOG_MEASURE, c, opt
);
- checkTopNAggregation(model, c);
+ checkTopNAggregation(model, c, opt);
}
}
} else {
PropertyModel propertyModel = MetadataRegistry.INSTANCE.registerPropertyModel(model, config);
- if (!RunningMode.isNoInitMode()) {
+ if (runShapeChecks) {
checkProperty(propertyModel.getProperty(), c, opt);
}
}
@@ -622,7 +626,7 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.build()
.equals(measure.toBuilder().clearMetadata().build());
if (!equals) {
- if (opt.isCreateIfAbsent()) {
+ if (!opt.getFlags().isUpdateOnMismatch()) {
log.error("BanyanDB measure {} shape mismatch at boot — backend holds a "
+ "different shape than the declared rule. SKIPPING metric; operator "
+ "must reshape via POST /runtime/rule/addOrUpdate or align the rule "
@@ -655,7 +659,7 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.build()
.equals(stream.toBuilder().clearUpdatedAt().clearMetadata().build());
if (!equals) {
- if (opt.isCreateIfAbsent()) {
+ if (!opt.getFlags().isUpdateOnMismatch()) {
log.error("BanyanDB stream {} shape mismatch at boot — backend holds a "
+ "different shape than the declared rule. SKIPPING; operator must "
+ "reshape via POST /runtime/rule/addOrUpdate. backend={}, declared={}",
@@ -682,7 +686,7 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.build()
.equals(trace.toBuilder().clearUpdatedAt().clearMetadata().build());
if (!equals) {
- if (opt.isCreateIfAbsent()) {
+ if (!opt.getFlags().isUpdateOnMismatch()) {
log.error("BanyanDB trace {} shape mismatch at boot — backend holds a "
+ "different shape than the declared rule. SKIPPING; operator must "
+ "reshape via POST /runtime/rule/addOrUpdate. backend={}, declared={}",
@@ -713,7 +717,7 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.build()
.equals(property.toBuilder().clearUpdatedAt().clearMetadata().build());
if (!equals) {
- if (opt.isCreateIfAbsent()) {
+ if (!opt.getFlags().isUpdateOnMismatch()) {
log.error("BanyanDB property {} shape mismatch at boot — backend holds a "
+ "different shape than the declared rule. SKIPPING; operator must "
+ "reshape via POST /runtime/rule/addOrUpdate. backend={}, declared={}",
@@ -730,9 +734,12 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
}
/**
- * Check if the index rules exist and update them if necessary
+ * Check if the index rules exist and update them if necessary. In
+ * {@link StorageManipulationOpt#isLocalCacheVerify() verify} mode the writes are
+ * skipped and a {@link StorageManipulationOpt.Outcome#SKIPPED_SHAPE_MISMATCH} is
+ * recorded instead — the orchestrator promotes that to a fatal boot error.
*/
- private void checkIndexRules(String modelName, List<IndexRule> indexRules, BanyanDBClient client) throws BanyanDBException {
+ private void checkIndexRules(String modelName, List<IndexRule> indexRules, BanyanDBClient client, StorageManipulationOpt opt) throws BanyanDBException {
for (IndexRule indexRule : indexRules) {
if (checkIndexRuleProcessed(modelName, indexRule)) {
return;
@@ -740,6 +747,12 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
IndexRule hisIndexRule = client.findIndexRule(
indexRule.getMetadata().getGroup(), indexRule.getMetadata().getName());
if (hisIndexRule == null) {
+ if (!opt.getFlags().isCreateMissing()) {
+ opt.recordOutcome("indexRule", indexRule.getMetadata().getName(),
+ StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
+ "IndexRule absent on backend; createMissing flag is off — refusing to define");
+ continue;
+ }
try {
client.define(indexRule);
log.info("new IndexRule created: {}", indexRule);
@@ -757,6 +770,12 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.build()
.equals(indexRule.toBuilder().clearUpdatedAt().clearMetadata().build());
if (!equals) {
+ if (opt.getFlags().isFailOnShapeMismatch()) {
+ opt.recordOutcome("indexRule", indexRule.getMetadata().getName(),
+ StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
+ "IndexRule shape mismatch on backend; failOnShapeMismatch flag is on — refusing to update");
+ continue;
+ }
client.update(indexRule);
log.info(
"update IndexRule: {} from: {} to: {}", hisIndexRule.getMetadata().getName(), hisIndexRule,
@@ -768,13 +787,16 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
}
/**
- * Check if the index rule binding exists and update it if necessary.
+ * Check if the index rule binding exists and update it if necessary. In
+ * {@link StorageManipulationOpt#isLocalCacheVerify() verify} mode skip the write and
+ * record {@link StorageManipulationOpt.Outcome#SKIPPED_SHAPE_MISMATCH}.
*/
private void checkIndexRuleBinding(List<IndexRule> indexRules,
String group,
String name,
BanyandbCommon.Catalog catalog,
- BanyanDBClient client) throws BanyanDBException {
+ BanyanDBClient client,
+ StorageManipulationOpt opt) throws BanyanDBException {
if (indexRules.isEmpty()) {
return;
}
@@ -793,6 +815,12 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.addAllRules(indexRuleNames).build();
IndexRuleBinding hisIndexRuleBinding = client.findIndexRuleBinding(group, name);
if (hisIndexRuleBinding == null) {
+ if (!opt.getFlags().isCreateMissing()) {
+ opt.recordOutcome("indexRuleBinding", name,
+ StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
+ "IndexRuleBinding absent on backend; createMissing flag is off — refusing to define");
+ return;
+ }
try {
client.define(indexRuleBinding);
log.info("new IndexRuleBinding created: {}", indexRuleBinding);
@@ -812,6 +840,12 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.build()
.equals(indexRuleBinding.toBuilder().clearMetadata().build());
if (!equals) {
+ if (opt.getFlags().isFailOnShapeMismatch()) {
+ opt.recordOutcome("indexRuleBinding", name,
+ StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
+ "IndexRuleBinding shape mismatch on backend; failOnShapeMismatch flag is on — refusing to update");
+ return;
+ }
// update binding and use the same begin expire time
client.update(indexRuleBinding.toBuilder()
.setBeginAt(hisIndexRuleBinding.getBeginAt())
@@ -827,9 +861,11 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
/**
* Check if the TopN aggregation exists and update it if necessary.
- * If the TopN rules are not used, will be checked and deleted after install, in the `BanyanDBStorageProvider.notifyAfterCompleted()`
+ * If the TopN rules are not used, will be checked and deleted after install, in the `BanyanDBStorageProvider.notifyAfterCompleted()`.
+ * In {@link StorageManipulationOpt#isLocalCacheVerify() verify} mode skip the write
+ * and record {@link StorageManipulationOpt.Outcome#SKIPPED_SHAPE_MISMATCH}.
*/
- private void checkTopNAggregation(Model model, BanyanDBClient client) throws BanyanDBException {
+ private void checkTopNAggregation(Model model, BanyanDBClient client, StorageManipulationOpt opt) throws BanyanDBException {
MetadataRegistry.Schema schema = MetadataRegistry.INSTANCE.findMetadata(model);
if (schema.getTopNSpecs() == null) {
return;
@@ -838,6 +874,12 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
String topNName = topNAggregation.getMetadata().getName();
TopNAggregation hisTopNAggregation = client.findTopNAggregation(schema.getMetadata().getGroup(), topNName);
if (hisTopNAggregation == null) {
+ if (!opt.getFlags().isCreateMissing()) {
+ opt.recordOutcome("topN", topNName,
+ StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
+ "TopNAggregation absent on backend; createMissing flag is off — refusing to define");
+ continue;
+ }
try {
client.define(topNAggregation);
log.info("new TopNAggregation created: {}", topNAggregation);
@@ -855,6 +897,12 @@ public class BanyanDBIndexInstaller extends ModelInstaller {
.build()
.equals(topNAggregation.toBuilder().clearMetadata().build());
if (!equals) {
+ if (opt.getFlags().isFailOnShapeMismatch()) {
+ opt.recordOutcome("topN", topNName,
+ StorageManipulationOpt.Outcome.SKIPPED_SHAPE_MISMATCH,
+ "TopNAggregation shape mismatch on backend; failOnShapeMismatch flag is on — refusing to update");
+ continue;
+ }
client.update(topNAggregation);
log.info(
"update TopNAggregation: {} from: {} to: {}", hisTopNAggregation.getMetadata().getName(),